path: root/layerindex/update.py
Commit message (Collapse)AuthorAgeFilesLines
* update.py: check whether branch existed when nocheckoutrbt/li_recsRobert Yang2018-07-091-2/+2
| | | | | | | | | | | | | | | | | | | Fixed: Assume there is no master branch in hello layer: $ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't exist This is correct since hello layer doesn't have master branch, but when --nocheckout: $ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip> This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout. Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
* update.py: add layers when RECOMMENDS isn't satisfiedRobert Yang2018-07-091-21/+46
| | | | | | | | | | | | | | | When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if layer_b is not found, still add layer_a. And print summary error mesage: $ update.py -b master ERROR: Issues found on branch master: openembedded-core: Added without LAYERRECOMMENDS meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
* Move run_command_interruptible() to utilsPaul Eggleton2018-05-041-33/+2
| | | | | | Make this function more easily reusable from the RRS code. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update_layer.py: move layer validation to update.py (Performance improve)Robert Yang2018-04-241-1/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The utils.setup_django() costs a lot of time, but both update.py and update_layer.py call it, so move layer validation from update_layer.py to update.py to avoid calling update_layer.py when possible can save a lot of time. Now we don't have to call update_layer.py in the following cases: * The branch doesn't exist * The layer is already update to date on specified branch (when no reload) * The layer dir or conf/layer.layer doesn't exist We can save up to 98% time in my testing: $ update.py -b master --nofetch [--fullreload] Before Now Reduced No update: 276s 3.6s 98% Partial update: 312s 87s 72% Full reload: 1016s 980s 3% Note: * All of the testing are based on --nofetch * "No update" means all layers on the branch is up-to-date, for example, when we run it twice, there is no update in the second run, so we only need about 3s now, which is the most common case when we use cron to run it per half an hour. * "Partial update" means part of the layers have been updated. * "Full reload" means all of the layers have been updated. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: print failed layers summary in the endRobert Yang2018-04-241-0/+9
| | | | | | | | | This makes it easy to see which layers failed. For example: ERROR: Failed layers on branch master: openembedded-core meta-python Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: add an option --timeout for lockfileRobert Yang2018-04-241-1/+4
| | | | | | | | | | | | | We have an update.py running periodically in background, but we also need to run it manually, for example, run it to update actual_branch, the manually run usually failed because can't get lockfile, we have to run it again and again. A timeout option helps a lot in such a case. Now the following command can make sure we can run the command successfully: $ update.py -b master -a actual_branch -t 2000 Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update: fix dependency processingRobert Yang2018-04-241-2/+2
| | | | | | | | | | | | | | | | | | | | | | The previous commit broke the layer order, e.g.: A -> B -> C -> D The algorithm is checking the dependencies one by one, and until we find D, add D to layerquery_sorted, and add it "collections", the one in "collections" means it's dependencies are OK, then C, B and A will check against collections, so that update_layer.py will update them one by one. The previous commit added A/B/C/D to collections directly, so that when check against it, all the dependencies are met, thus broke the layer sorting, and then there would be failures if we pass layer A to update_layer.py before B, C and D (suppose they are newly to database). This commit fix the problem. BTW., why I use collections to record the one whose dependencies are matched, but not directly use layerquery_sorted, it is because collections contains both the ones to be added/updated and the ones in database, but layerquery_sorted only contains the ones to be updated/added. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix Ctrl+C behaviourPaul Eggleton2018-02-201-2/+10
| | | | | | | | | | | | | If the user hit Ctrl+C during the initial info gathering then it didn't break out of the loop in update.py, so you had to hit Ctrl+C for as many layers as were involved in the update. Look for exit code 254 from update_layer.py and stop if it is returned since that indicates Ctrl+C has been used. Additionally, ensure we return exit code 254 and print a message from the main update script when it is interrupted in this way. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fetch repos in parallelRobert Yang2018-02-051-19/+34
| | | | | | | | | | | | This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10, this is the fetch time only, I hacked it to stop when the fetch is done to get the data (124 layers): $ update.py -b <branch> Before: 2m30 Now: 16s Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: make sure oe-core is fetchedRobert Yang2018-02-051-1/+7
| | | | | | | | | | | | | | | | | Fixed: $ ./update.py -l foo -b new_branch INFO: Fetching remote repository foo DEBUG: run cmd 'git fetch -p' in 'foo' [snip] DEBUG: run cmd 'git checkout origin/new_branch' in oe-core ERROR: error: pathspec 'origin/new_branch' did not match any file(s) known to git. The "new_branch" is newly created, it doesn't exist in local repo since it isn't fetched, it only fetches the layer specified by -l, so only the foo layer is fetched. This patch fixes problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update: don't stop on unsatisfied layer dependenciesPaul Eggleton2018-01-091-10/+10
| | | | | | | Unsatisfied layer dependencies shouldn't error out of the script - broken metadata isn't supposed to terminate the index update process. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update: allow preserving temp directoryPaul Eggleton2017-10-031-0/+5
| | | | | | | | | | | If you're diagnosing problems with the bitbake server when running the update script, then you need to be able to look at bitbake-cookerdaemon.log, but you couldn't do that after the fact because the temporary directory it gets written out to was being unconditionally deleted. Add a --keep-temp option which preserves it and some debug messages to tell you where it is. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: add -p to git fetchRobert Yang2017-07-211-2/+2
| | | | | | | | | | | | | | | -p, --prune Before fetching, remove any remote-tracking references that no longer exist on the remote. Fixed: $ git push origin :test_branch $ ./update.py The test_branch was still in fetched local repo which was incorrect, it should be gone since it has been removed by upstream. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: update layers in dependency orderRobert Yang2017-07-211-36/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Problems The update.py couldn't handle new (not only new branch, in fact, but also existing branches, see below for more info) branch well, for example, there are 3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for them, and they have depends: layer_A -> layer_B -> layer_C The "->" means depends on. Then run "update.py -b branch_1", there would be errors like: ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1 Though update.py runs "update_layer.py" twice, but it didn't help since layerbranch was None when it was failed to create in the first run. The reason is if update.py updates layer_A firstly, it would fail since it can't find layer_B:branch_1 in database (not added to database yet), similarly, if add layer_B before layer_C, it would also fail. Only layer_C can be added (assume it has no dependencies). So we have to re-run update.py again and again to make it work, here we may have to run update.py 3 times, and more runs are needed if the dependency chain is longer. * Solutions: Make update.py pass layers orderly to update_layer.py according to dependencies can fix the problem, we can get intial dependencies from tinfoil, add an option "-i, --initial" to update_layer.py to get them. Not only new branch, but also existing branches may have the problem, because collections and dependencies maybe changed in the coming update, so we can't trust database when the layer is going to be updated, for example, if there are 10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3), then we can not use the 3 layers' collections data from database, we need get them from tinfoil. * Performance improvement: It should be the same as before in theory, I have tested it with 97 layers: - Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I have to re-run update.py again and again (maybe 4 times to make all of them added). So: (11 * 60 + 6)/75*97/60 = 14m35s - Now 12m10.350s, all the layers are added in the first run. So about 2 minutes are saved. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: update actual branch for layer and bitbakeRobert Yang2017-07-211-5/+66
| | | | | | | | | | | | | | | | | | | | | | | | Add an option "-a" to update actual branch for layer and bitbake, it is useful when there are many layers and need update actual branches frequently. We only can update them via website without this patch, which is not fun and easy to make mistakes. * It works with "-l", and "-l bitbake" means update bitbake branch. * It requires "-b" to work, and only one branch is supported in a run. For example: $ update.py -b master -a branch_20170526 All the layers which have branch master and actual_branch branch_20170526 will be updated to branch_20170526. $ update.py -b master -l meta-oe -a branch_20170526 Only meta-oe layer will be updated. $ update.py -b master -l bitbake -a branch_20170526 The bitbake's bitbake_branch will be updated. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* layerindex: add "No update" status to LayerItemPaul Eggleton2017-03-141-0/+1
| | | | | | | | | | | | Add a status for a layer indicating it should not be updated. I don't expect this to be widely used (and is only settable from the admin interface) but would be useful if you have a legacy sub-layer that you want to prevent from being visible on certain branches - it will prevent the update script from doing anything with the layer and thus avoid branch records from being auto-created on branches where you've deleted it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: use reader to decode subprocess output correctlyPaul Eggleton2017-03-071-2/+3
| | | | | | | | | | | | | | We can't decode UTF-8 characters byte-by-byte, as soon as we hit a character that's more than one byte long then we'll fail. Use a reader object to do it properly. This fixes parsing current meta-angstrom on master. At the same time, specify errors="surrogateescape" to avoid the update process dying at this point in case of characters that aren't valid UTF-8. Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's very similar code. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update: fix logic for updating layer dependencies at the endPaul Eggleton2017-01-161-44/+41
| | | | | | | | | | | | | | | * We were passing the incorrect path (to the top of the layer repo) if the layer had a subdirectory, so this doesn't seem to have been able to work for such layers previously. * Doing this update in the main update.py script meant that this could never work across branches requiring a python version change (using PythonEnvironment records) since the code was running within the same environment in which update.py was launched - the entire point of the separation of the two scripts. Move the checking to update_layer.py and call it separately to perform these updates, splitting out some common code in order to do so. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update: only get branch object once per branchPaul Eggleton2017-01-161-2/+1
| | | | | | | A minor optimisation - it's not going to change between layers so get it in the branch loop. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Fix layerconfparser lifetime handlingPaul Eggleton2017-01-161-1/+1
| | | | | | | | | | | * Setting of the object has to be before the try: or otherwise the finally block can get called if that doesn't succeed with the result that the layerconfparser object won't have a value, which will trigger an exception * We shouldn't be using the config_data object after shutting down tinfoil, so avoid doing that Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Record and display update logsPaul Eggleton2016-11-161-115/+171
| | | | | | | | | | | | | | | | | At the moment it's a bit difficult to get update logs out of the environment in which the update script is being run. In order to make the logs more accessible, create a LayerUpdate model to record the output of update_layer.py separately for each layerbranch and tie the created LayerUpdates together with a single Update model per session. We provide two ways to look at this - a Tools->Updates page for logged-in users, and there's also an "Updates" tab on each layer that is accessible to anyone; which one is useful depends on whether you are looking at the index as a whole or an individual layer. Update records older than 30 days are deleted automatically by default. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix last_rev handlingPaul Eggleton2016-11-071-4/+12
| | | | | | | | | update_layer may create the layerbranch record (e.g. for a new branch) so we should be looking for it after that has been run. Also, we cannot assume that last_rev will get set because a layer might fail to fetch, so take that into account as well. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update_layer: fix regressions in missing layer detectionPaul Eggleton2016-11-071-3/+1
| | | | | | | | | | | | | | If a layer is removed by its subdirectory being deleted then we want to pick up on that and produce an appropriate error message - so let the layer_update code do the checking out and verifying things are correct before trying to parse layer.conf. This also fixes --nocheckout still checking out a branch. Additionally, drop some code that gets the layerbranch which we just retrieved a few lines above. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix regression handling layers that failed to fetchPaul Eggleton2016-11-071-0/+1
| | | | | | | | If the layer failed to fetch, we shouldn't be attempting to update its contents. (This got broken when we split the update script into two in c64e4c57a9158c8ae5e49f526c3ff87950d3a94e). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* layerindexer: Add layer recommends supportLiam R. Howlett2016-10-181-0/+1
| | | | | | | | | | | | Parse layer.conf and add dependencies that are not required from LAYERRECOMMENDS_<name>. Update the layerindex/template to support recommends. Uses bitbake parsing code & checks versions. Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com> Added associated migration. Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
* layerindex: Detect dependencies from layer.conf filesLiam R. Howlett2016-10-181-1/+40
| | | | | | | | | | | | Read dependencies from layer.conf and try to create the LayerDependency entry by looking up the correct database object. Dependencies are found by layer name only - no collection support. layer.conf parsing is handled by the bitbake code. Once all layers are added, the dependencies have to be rechecked in case the layers are not added in order. Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
* update.py: default to python3 without a defined environmentPaul Eggleton2016-09-201-1/+1
| | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Handle Python 2 and Python 3 branches in the same indexPaul Eggleton2016-06-121-1/+6
| | | | | | | | Add a model to support setting a python command and virtualenv per branch, which allows you to parse master with python3 and krogoth with python2 for example. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Support (and require) Python 3Paul Eggleton2016-06-121-1/+1
| | | | | | | | | | We need to be able to support Python 3 so that we can parse master of OE-Core with bitbake (which now requires it). This now means the interface itself and the update script require Python 3.4+. Part of the implementation for [YOCTO #9704]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: allow updating all branches with one commandPaul Eggleton2016-06-101-32/+38
| | | | | | | | | | | | | | | Allow updating multiple branches, and if no branches are specified, update all branches that have a new "updates_enabled" flag field set to True. This avoids the need to have a separate shell script which runs update.py for each branch (and thus has hardcoded knowledge of each active branch in the index, i.e. it needs to be kept up-to-date in addition to the database.) The migration will default updates_enabled to True for all branches so if you wish to take advantage of this functionality, the flag will need to be set to False for any branches that shouldn't be updated. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: refactor into two separate scriptsPaul Eggleton2016-05-301-467/+44
| | | | | | | | | | | | | In order to try to avoid problems with leaking memory, context bleeding from one layer to another, and lay the groundwork for supporting updating all branches in one operation, split the updating of a single layer out to its own internal script. This means that the tinfoil instantiation and metadata parsing is in a completely separate process per layer. Implements [YOCTO #9647]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Display recipe blacklist informationAlex Franco2016-05-241-0/+1
| | | | | | | | | | | | | | | | | Display blacklist information for recipes in the recipe details, as well as the recipe search page, and layer page tables. This information is pulled from the PNBLACKLIST variable. Includes a hover text containing the reason for blacklist labels. Changes to Django and Layerindex files: - models.py, update.py - migration file - templates and static content (stylesheet) [YOCTO #7855] Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com>
* update.py: fix splitting on _ when getting class inheritancePaul Eggleton2015-11-021-1/+1
| | | | | | | | | | | | The code to populate the inherits information was using a function designed to get the recipe name from a file path, but unlike recipe filenames, the underscore isn't treated as special in class filenames and in fact it's quite common to use underscores there; we were ending up with items in the inherit list such as "populate" and "rootfs". Use the standard python os.path functions to do the splitting instead and avoid this. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Support for recipe inherits fieldAlex Franco2015-10-071-0/+4
| | | | | | | | | | | | | | | To identify image recipes and provide inheritance data for recipes, an inherits field was added to the recipe model, and then populated using refactored data from __inherit_cache. Finally the field was also added to page templates, along with style changes proposed in attachment, and an additional style change to display the inherits field as a list in detailed view. The field skips globally inherited data as proposed. [YOCTO #7575] Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: minor fixup for last commitPaul Eggleton2015-06-151-1/+2
| | | | | | | Fix up memory leak fix for bitbake in daisy and earlier which did not have bb.codeparser.codecache. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix memory leaksPaul Eggleton2015-06-151-0/+13
| | | | | | | | | | | | We're kind of abusing some bitbake internals here and the result is that there are a bunch of global lists and caches that simply grow as we parse more layers, until available memory is exhausted. We don't care about the contents of any of these within the layer index update script so just clear them out between layers. Should fix [YOCTO #7663]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix not clearing recipe files list on reloadPaul Eggleton2014-06-121-1/+1
| | | | | | | | | | | | When we did a full reload this list wasn't getting cleared for the next layer, and we look at the contents of the list unconditionally later to determine which files need to be added. Given that the list items are full paths, this is unlikely to have caused any problems; however it could possibly have been an issue for nested layers with the right update order, or lead to issues if the usage of the list changes in future. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: ignore layers within layersPaul Eggleton2014-06-121-0/+41
| | | | | | | | | Some layers, such as meta-intel, also contain other layers. We don't want recipes/classes/appends in those child layers to appear in the parent layer so ignore any that appear within subdirectories that contain a conf/layer.conf file. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix layer exists-on-branch checkPaul Eggleton2014-05-241-13/+21
| | | | | | | | | | There was a logic error here - we were checking to see if the layer directory existed before actually checking out the branch, which means we were looking at the state of whatever branch was previously checked out. Replace this with some validation of the last revision check and an additional check for the directory after checking out the branch. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: use actual branch when getting last revisionPaul Eggleton2014-05-241-1/+1
| | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Allow layers to have no master branchPaul Eggleton2014-04-061-11/+15
| | | | | | | | With BSPs being "retired" e.g. in meta-intel, it is possible for layers to not exist on the master branch; since this is legitimate we need the layer index to handle it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix subdirectory does not exist error messagePaul Eggleton2013-12-271-1/+1
| | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: add --fullreload optionPaul Eggleton2013-09-201-16/+25
| | | | | | | | | | Recently the -r/--reload option was changed to preserve existing recipe data, so that it could be used to load values into newly added fields; however we still need an option to load recipe data from scratch for testing purposes (e.g. in combination with --nocheckout) so add a --fullreload option to do this. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix ability to test updating with --nocheckoutPaul Eggleton2013-09-201-3/+5
| | | | | | | | | In order to test updating we need to be able to check out a previous revision of the repository and have the update script load from that without checking out the latest revision, so fix the --nocheckout option to work properly. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: fix Ctrl+C handling during layer updatePaul Eggleton2013-09-201-0/+4
| | | | | | We do not want to continue on to the next layer on Ctrl+C. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: hide DeprecationWarningsPaul Eggleton2013-09-051-0/+3
| | | | | | These are just annoying if the script is called from a cron job. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* update.py: handle if deleted recipe can't be foundPaul Eggleton2013-09-051-3/+6
| | | | | | | Show a warning instead of a traceback if a deleted recipe can't be found in the database. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Ensure logger is passed into runcmd function or use sys.stderr.writePaul Eggleton2013-09-051-7/+7
| | | | | | Otherwise it might not be defined when an error.needs to be printed. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Implement locking for update/bulkchange processPaul Eggleton2013-09-051-310/+319
| | | | | | | Avoid the possibility of these two clashing especially when multiple branches are being used. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Add support for importing OE-Classic recipesPaul Eggleton2013-09-051-38/+6
| | | | | | | | | | Add a script for doing a one-time import of OE-Classic recipe information, so comparisons against OE-Core can be performed; this is stored using a new ClassicRecipe model supporting additional fields for tracking migration status. The migration status fields can be updated as well as viewed and summarised in graph format. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>