aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
* fetch2: Handle missing donestamp file when content is validNathan Rossi2018-02-092-11/+114
| | | | | | | | | | | | | | | | | | | | | | In order to allow users to manually populate the download directory with valid content change the assumption that missing the donestamp file means unfetched content. This allows users to populate the download dir without needing to create dummy .done files such that a user does not need a PREMIRROR when using BB_NO_NETWORK to provide valid content files in the download directory. To ensure the correct result this change also fails first if the localpath does not exist. This prevents further parts of the function attempting to calculating the checksum on non-existent files. This also fixes some edge conditions around where if the donestamp exists but the localpath does not it returns, and did not remove the donestamp. Also added test cases to cover this use case and additional use cases where for example the fetcher does not support checksums. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/npm.py: Fix inverted condition to prevent infinite loopBöszörményi Zoltán2018-02-091-3/+5
| | | | | | | | | | | | | At least the cli-color node module has dependencies that have cyclic dependency among themselves. npm.py is prepared to deal with such a case but the condition is handled only for downloading or not a dependency again, but then it goes checking the its dependency which causes an infinite loop in _getdependencies(). Make this function simply return when a dependency is already downloaded and only download and check its dependencies when not. Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* contrib/dump_cache.py: make it can dump everythingRobert Yang2018-02-091-34/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Have a simple tool to dump bb_cache.dat is useful for investigating and studying bitbake cache. The old contrib/dump_cache.py can dump pn, pv and packages for it, now enhance it dump everything. Here is the usage: $ /path/to/dump_cache.py --help usage: dump_cache.py [-h] [-r RECIPE] [-m MEMBERS] [-s] cachefile bb_cache.dat's dumper positional arguments: cachefile specify bb_cache.dat optional arguments: -h, --help show this help message and exit -r RECIPE, --recipe RECIPE specify the recipe, default: all recipes -m MEMBERS, --members MEMBERS specify the member, use comma as separator for multiple ones, default: all members -s, --skip skip skipped recipes Use dump_cache.py --help to get help Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: fixes for resolve_file()Robert Yang2018-02-093-7/+2
| | | | | | | | The resolve_file() calls mark_dependency(), so the one which calls resolve_file() doesn't need call mark_dependency() again. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: improve debug messageRobert Yang2018-02-091-1/+4
| | | | | | | | | | | * Print message when cachefile is found/not can help debug. * Update "Using cache in" to "Cache dir:" since it was the same as the debug message of "codeparser & file checksum caches", which caused confusion. And whether the cache file will be used or not is still unknown at that time, so just print the cache dir. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: don't stop file notifier when cooker is shutdownRobert Yang2018-02-092-2/+2
| | | | | | | | | | | | | | | | | | | It should be live/exited with server rather than cooker, fixed: $ bitbake --server-only -T -1 Set MACHINE = "qemux86" in conf/local.conf $ bitbake quilt Set MACHINE = "qemuppc" in conf/local.conf $ bitbake quilt [snip] ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed. [snip] This is because the server doesn't know local.conf is changed since the notifiers are stopped, so it doesn't reparse, and then we would get the errors, let the notifiers live/exited with server can fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix for BBFILE_PATTERN matches bbappendRobert Yang2018-02-091-3/+7
| | | | | | | | | | | | | | | | | | | The old code couldn't handle nestled layers correctly, e.g.: parent_layer/sub_layer/foo.bb Note there are two layers, parent_layer and sub_layer. And in parent_layer/conf/layer.conf: BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/" This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb, so it warns that no files matched sub_layer, this is the expected behavior, but it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it should warn and let the user fix the problem. Check the bbappend in already "matched set" before return it as matched by "unmatched set" can fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: skip when BBFILE_PATTERN is emptyRobert Yang2018-02-091-0/+1
| | | | | | | There is nothing to do when BBFILE_PATTERN is empty. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/git: Use 'git-make-shallow' from bin directoryNathan Rossi2018-02-092-5/+5
| | | | | | | | | | | | Move the code that existed in tests/fetch.py for determining the path to 'git-make-shallow' into the git module and reference it. This ensures that 'git-make-shallow' is always available and the desired version regardless of the path variable or whether git exposes the command. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* main/runqueue: Rework runall task and add runonly optionRichard Purdie2018-02-093-14/+46
| | | | | | | | | | | | | | | | | | | | | | | | | The runall commandline option was confusing people. There are in fact two different behaviours people may want. a) For a given target (or set of targets) look through the task graph and run task X only if its present and would have been built. b) For a given target (or set of targets) look through the task graph and run task X if any recipe in the taskgraph has such a target even if it wasn't in the original task graph. I've decided to interpret the existing "runall" option as b), even if right now if behaves like a). For a), which is a valid use case, this patch adds a "runonly" option. With both behaviours present, I'm hoping we can then kill off the "fetchall", "checkuriall" and other tasks from OE metadata and replace them with this option. This would significantly speed up task graph processing. (Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g" from 22s to 8s). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Commented out Placeholder ParagraphKristi Rifenbark2018-02-061-2/+7
| | | | | | | Fixes [YOCTO #12030] Signed-off-by: Kristi Rifenbark <kristi@buzzcollectivemarketing.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Rewrite and optimize recrdepends handlingRichard Purdie2018-01-291-51/+83
| | | | | | | | | | | | | | | | This is a performance sensitive piece of code and the shear number of recursive loops is causing a significant and unscalable performance pain point. This change moves to a two step approach, firstly generating a list of recursive dependencies for any task, then applying this to the recursive tasks, iterating over things until no further dependencies are added. It was noticed an optimisation is possible and the list of recursive tasks need not contain the taskname, only the base task id. This allows a significant performance improvement and limits the size of the resursive task lists, improving speed. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Optimize recrdepends handlingRichard Purdie2018-01-291-10/+13
| | | | | | | | | | | | | | We can optimise the loops slightly so we only process given substrings once rather than many times. This means expanding out add_resolved_dependencies. Also add a function which allows replacement of the task element of a task id, reducing the amount of string handling we're doing in a performance critical loop. Its also clear that later code adds to the tasks depends so we don't need to add .depends() to extradeps at the start. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Remove unused variablesRichard Purdie2018-01-291-2/+0
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Fix recidepends handlingRichard Purdie2018-01-291-41/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | Currently we only run through the recidepends/recrdepends code once. This means that we can miss some expansions of dependency trees where one rec{r,i}depends tasks depends on another rec{r,i}depends task. In reality we need to iterate over the data until we stop adding dependencies. In doing this we can't show quite so granular progress information since we don't know how many times we'll need to do this. This does slow down the runqueue prepare phase however some optimisations are possible and can be handled in subsequent patches. This fix means some missing dependencies, such as: <image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X -> <ca-certs>:do_package_write_X -> debianutils-native (via PAKAGE_WRITE_DEPS) are now found/added. [YOCTO #12510] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/git: Add pack-refs command to mirror handling codeRichard Purdie2018-01-281-0/+1
| | | | | | | | | | | We've noticed issues on our infrastucture iterating over the many tag/branch/head reference files that some git repositories may contain. By issuing the pack-refs command, we move these all to a single file which speeds up operations with the mirror repos in the downloads directory in general. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Sort task graph (dot) file outputRichard Purdie2018-01-281-6/+6
| | | | | | | This means two different graph files can easily be compared, currently you'd have to sort them as the output is randomized. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Removed stray parenthesisKristi Rifenbark2018-01-161-1/+1
| | | | | | | Fixes [YOCTO #12016] Signed-off-by: Kristi Rifenbark <kristi.rifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Updated Hello World tutorialKristi Rifenbark2018-01-161-3/+21
| | | | | | | Fixes [YOCTO #12016] Signed-off-by: Kristi Rifenbark <kristi.rifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Replaced bad linkKristi Rifenbark2018-01-161-13/+13
| | | | | | | | | | | | Fixes [YOCTO #11675] There was a "hambedded" link that was broken. The link was intended to show the bitbake.conf file. I replaced it with the link to show the actual bitbake.conf file. Signed-off-by: Kristi Rifenbark <kristi.rifenbark@gmail.com> Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Updated copyright year.Scott Rifenbark2018-01-161-1/+1
| | | | | Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Updated 'bitbake -h' output example.Scott Rifenbark2018-01-161-22/+25
| | | | | Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Updated "Events" for BuildStarted eventScott Rifenbark2018-01-161-0/+2
| | | | | | | | | | BitBake now fires off one "BuildStarted" event per configuration when it is configured for multiple configurations (multiconfig). I updated the bullet item for "bb.envent.BuildStarted()" to indicate that behavior. Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/__init__: Disable pseudo in runfetchcmd()Peter Kjellerstedt2018-01-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | If a fetcher, e.g., git, is run when pseudo is active it will think it is running as root. If it in turn uses ssh (as git does), ssh too will think it is running as root. This will cause it to try to read root's ssh configuration from /root/.ssh which will fail. If ssh then needs to ask for credentials it will hang indefinitely as there is nowhere for it to ask the user for them (and even if there was it would not access the correct private keys). The solution to the above is to temporarily disable pseudo while executing any fetcher commands. There should be no reason for them to be executed under pseudo anyway so this should not be a problem. RP Ammendum: We finally did get more information about how to reproduce this problem, something needs to trigger bb.fetch2.get_srcrev() in a pseudo context, for example when AUTOREV is in use or the recipe doesn't have a defined SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger an ls-remote of the remote repo and if that happens under pseudo, the wrong ssh credentials may be attempted which can hang. [YOCTO #12464] Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* events: Drop unused cfg propertyRichard Purdie2018-01-071-9/+0
| | | | | | | The cfg property on events doesn't add much, all code appears to access "data" at this point. Remove it to clean up the interface. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/fetch: Allow bit/bitbake-selftest to function correctlyRichard Purdie2018-01-071-1/+1
| | | | | | | Without this the paths to git-make-shallow are incorrect and cause test failures if bitbake isn't executed from cwd or PATH. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/event: tests: Add event classes test classJair Gonzalez2018-01-071-0/+499
| | | | | | | | | | | This change adds EventClassesTest class to bb/tests/event.py, including 47 new test cases for the public interfaces of the bitbake event and related helper classes. [YOCTO #10773] Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/event: Remove empty assignments from event testsJair Gonzalez2018-01-071-4/+4
| | | | | | | Remove assignments from non-returning calls. Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/event: Add test for bb.event.worker_fireJair Gonzalez2018-01-071-0/+8
| | | | | | | Test the triggering of bb.event.worker_fire callback. Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/event: Include tests for event filtering featuresJair Gonzalez2018-01-071-9/+111
| | | | | | | | | | | | | | | | | | This change includes unit tests for the following functions, helper class and methods in bitbake.event: - set_eventfilter(func) - set_UIHmask(handlerNum, level, debug_domains, mask) - getName(e) - class UIEventFilter(object) def __init__(self, level, debug_domains) def update(self, eventmask, level, debug_domains) def filter(self, event) [YOCTO #10773] Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: corrupted patch for bldcollector URL fileDavid Reyna2018-01-021-1/+1
| | | | | | | | | | Fix a fatal error introduced by a corrupted patch file submission. [YOCTO #12459] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* build.py: Don't return dependencies for tasks which don't existRichard Purdie2017-12-191-0/+6
| | | | | | | | If for example you deltask do_build but it has recrdeps set, its confusing to have that list returned when the task no longer exists (same would apply to deps too if it was set after the deltask). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: add 'nobuild' option to ToasterDavid Reyna2017-12-185-28/+58
| | | | | | | | | | | | Add a 'nobuild' option for starting Toaster without the project and hosted builds support. This allows a Toaster host to provide local build statistics without opening the host to external users building projects. [YOCTO #12315] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: landing page show cmdline captureDavid Reyna2017-12-181-1/+7
| | | | | | | | | | | The Toaster initial landing page needs to show that it still has the command line build capture support in addition to the new project support. [YOCTO #12316] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: enable custom env support for shell callsDavid Reyna2017-12-181-11/+15
| | | | | | | | | | Allow for custom environment additions for git cloning, for example for anspass support. [YOCTO #12193] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: some recipe events do not include packagesDavid Reyna2017-12-181-1/+5
| | | | | | | | | | Allow for "SinglePackageInfo" events that do not include package data, for example OPKGN equal 'lib32-*' or 'lib64-*'. [YOCTO #12204] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: update Toaster for Django 1.11David Reyna2017-12-186-6/+6
| | | | | | | | | Toaster needs to accomodate API changes in Django 1.11. [YOCTO #12192] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: unset BBPATH before starting bbserver for backwards compatibilityDavid Reyna2017-12-181-6/+12
| | | | | | | | | | | | | | | The latest build enviroment script is now setting BBPATH. This breaks building previous releases from a master-based Toaster, because with this newly inherited BBPATH value the 'bitbake.lock' file ends up in the Toaster build directory instead of the project's build directory. Toaster should always clear BBPATH so that the bbserver's environment is clean (enough). [YOCTO #12363] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* providers: Fix determinism issueRichard Purdie2017-12-101-3/+3
| | | | | | | | | | | | | | | | We saw builds where runtime providers were sometimes changing order and the build result was therefore non-deterministic. For example it could show: DEBUG: providers for lib32-initd-functions are: ['lib32-lsbinitscripts', 'lib32-initscripts'] or DEBUG: providers for lib32-initd-functions are: ['lib32-initscripts', 'lib32-lsbinitscripts'] which could cause a test to pass or fail. This change ensures we don't rely on the random order of dictonaries in memory and act deterministically. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: fix unpack of .txz filesAndré Draszik2017-11-291-1/+1
| | | | | | | | .txz is the same as .tar.xz, and can be found in the wild. Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/wget: use with to ensure the response is closedRoss Burton2017-11-211-1/+2
| | | | | Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/wget: improve proxy objectRoss Burton2017-11-211-0/+1
| | | | | | | | The connection cache class uses a dummy file object but it doesn't have a closed attribute, so we can't use it in a context manager. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix typo in bitbake -g messagePaul Eggleton2017-11-111-1/+1
| | | | | | | Flatened -> flattened. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: ensure get_recipe_info() returns Null if recipe not foundPaul Eggleton2017-11-111-7/+10
| | | | | | | | If a matching recipe is not found then return Null instead of raising KeyError because we were blindly using None as a key for pkg_fn. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-layers: show-recipes: fix help to mention -i supports multiple classesPaul Eggleton2017-11-111-1/+1
| | | | | | | | The -i option supports more than one class, but the help didn't mention that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-layers: remove-layer: support removing multiple layers at a timePaul Eggleton2017-11-111-9/+12
| | | | | | | | If you can add multiple layers at once, it stands to reason that you should also be able to remove more than one at a time. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-layers: add-layer: enable adding multiple layers at oncePaul Eggleton2017-11-111-11/+13
| | | | | | | | | | | | | Allow specifying multiple layers with bitbake-layers add-layer so that you can add more than one in a single command. This is not just useful, it's actually pretty important if you need to add a layer and its dependencies at the same time - since we now go through a parse process when the layer is added, without this you have to add them all in just the right order and wait for the parse each time which is somewhat painful. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Fix typo builable -> buildableRichard Purdie2017-11-111-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: Ensure we clean up loggersRichard Purdie2017-11-081-2/+8
| | | | | | | | This is primarily paranoid but ensure we remove any loggers we setup either directly or indirectly so the initial state is restored after we exit. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* event: Ensure we clean up loggersRichard Purdie2017-11-081-0/+4
| | | | | | | | Whilst we're likely exiting in this case, clean up the loggers we add so that in the case of certain server retries there is no possibility multiple loggers stack up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>