summaryrefslogtreecommitdiffstats
path: root/lib/bb/cache.py
AgeCommit message (Collapse)Author
2011-05-06bitbake/cache.py: Ensure skipped recipes make it into the cache to avoid ↵Richard Purdie
reparsing (From Poky rev: 001a555c2f755d4f8a69b113656d9307ca7ee597) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06bitbake/ast.py: Only run finalise() for the specified variantRichard Purdie
Allows the heavy finalise function to only be run for the case we're interested in when running tasks, saving some processing time. (From Poky rev: 9211fd9c375489c73924fd43f1f8a0da2c4290bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-18runqueue.py: Enable PSEUDO (fakeroot) before the forkMark Hatle
PSEUDO when used as the fakeroot program is usually preloaded into memory, but disabled by default. The trigger for enabling it is a set of environment variables and a fork() or exec*() operation. We need to setup the environment, specifically PSEUDO_DISABLED=0, prior to the fork() so that python tasks can be run under PSEUDO control. This patch is based on the work in Poky primarily from Richard Purdie. See the following Poky commits for detailed history on this change: commit 1f3e313fd5cc5ae8ea838bf8fcdedace3cb72584 Author: Richard Purdie <rpurdie@linux.intel.com> Date: Wed Dec 8 00:08:04 2010 +0000 bitbake Revert bitbake exec() and go back to fork() for performace wins (first draft) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> commit 52aada2eaf08d57d5f715f155f2d878831dbaab0 Author: Richard Purdie <rpurdie@linux.intel.com> Date: Tue Dec 21 00:51:24 2010 +0000 bitbake/runqueue.py: Somehow the python environment mapping is failing so do it manually Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> commit 1b08a7eb8b708f4d0fc119cf89deb450fa62fea1 Author: Richard Purdie <richard.purdie@linuxfoundation.org> Date: Mon Feb 28 15:31:20 2011 +0000 bitbake/cache/runqueue.py: Move workload for recipe parsing to the child process Parsing the recipe in the parent before forking off the child worker can mean the parent doesn't hit the idle loop and becomes a bottleneck when lauching many short lived processes. The reason we need this in the parent is to figure out the fakeroot environmental options. To address this, add the fakeroot variables to the cache and move recipe loadData into the child task. For a poky-image-sato build this results in about a 2 minute speedup (1.8%). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Integrated and merged into bitbake upstream Signed-off-by: Mark Hatle <Mark.Hatle@windriver.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-03-03bitbake/cache: bump cache version after recent changesJoshua Lock
(From Poky rev: 08c7a59b5a5d24cf8c03b9da8278d30c0c175f5e) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-03bitbake/[cooker|cache]: cache summary, license and group. Add to targets treeJoshua Lock
Add summary, license and group metadata to RecipeInfo and the cache. Unfortunately this impacts parse speed but gives us a much richer set of metadata to expose through UI's which can be accessed via the generateTargetsTree command. (From Poky rev: a13304e1c6500db95823961a5a34849db46037e3) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-03bitbake/cache: store a list of inherited files in the cacheJoshua Lock
(From Poky rev: 920c402342bd490cd94b365c3e151de735dec0d6) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-21fix parameter order for flaglist()Dongxiao Xu
(From Poky rev: 5da9423217a5cd204d17a0f7bbe248082a1b257c) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-21Introduce stamp-extra-info task flag into stamp filenamesDongxiao Xu
For certain tasks, we need additional information in build stamp file other than the task name and file name. stamp-extra-info is introduced as a task flag which is appended to the stamp file name. [Code simplifcations/tweaks from Richard] (From Poky rev: 80cf3e405c923ed95dd09936739e816dbdd92299) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-20cache: fix docstring usageChris Larson
Use comments instead of docstrings where appropriate in CacheData. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-01-07Merge branch 'poky-sync'Chris Larson
* poky-sync: build: kill unneeded environment setup bits parse: save python functions into the metadata Include exported variables in task process env runqueue: start implementing quieterrors Overhaul environment handling runqueue: resurrect use of file objects for pipein/pipeout runqueue: clean up message log levels runqueue: use correct task ID when checking validity of setscene tasks Add support for 'noexec' tasks runqueue: implement cache checking from metadata runqueue: pass task hash information to tasks runqueue: avoid unnecessary delays Implement task signatures data: Add emit_func and generate_dependencies codeparser: Implement persistent cache data_smart: track variable references codeparser: add module for parsing shell/python runqueue: implement scenequeue fetch: add SRC_URI checksum Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-01-07cache: don't expand variables for skipped recipesChris Larson
Errors can result from these expansions, but for skipped recipes, we shouldn't care about those failures. This fixes the same issue which Richard Purdie fixed in poky, commit 847b717. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-01-04cache: defer marking fn as cleanBernhard Reutner-Fischer
Only mark fn as clean if it is clean. This saves us from removing (prematurely added) fn from our clean set and saves me a few percent of runtime (and misleading debugging output from remove()). Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-29runqueue: implement cache checking from metadataRichard Purdie
This lets us speed up setscene processing, by letting the function from the metadata inform us about the necessity of running the setscene functions. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-29Implement task signaturesRichard Purdie
Includes functionality to find out what changes between two different signature data dumps. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-12-16Show the user progress when loading the cacheBob Foerster
Signed-off-by: Bob Foerster <robert@erafx.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-29cache: change to more incremental formatChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-23cache: bump cachever per masterChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-23cache: ensure 'pn' is included in the pkgvarsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21cooker: stop loading the cache for -bChris Larson
Previously, the cache was actually being loaded from disk twice whenever using -b or -e -b. This also moves the bb_cache instance into the CookerParser, as it's not needed by the cooker itself at all. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21cache: don't add info to cache if cache is disabledChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21Simplify cache syncingChris Larson
Rather than adding nocache items to the cache, then copying the cache and removing them to sync it, don't add them in the first place. Also use 'with' for the cachefile. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21Implement parallel parsing supportChris Larson
This utilizes python's multiprocessing module. The default number of threads to be used is the same as the number of available processor cores, however, you can manually set this with the BB_NUMBER_PARSE_THREADS variable. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21cache: don't write out the cache unnecessarilyChris Larson
If the only recipes's we reparsed this run were those flagged as not to be cached, there's no point in re-saving the cache, as those items won't be included anyway. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-21cache: create and use a RecipeInfo classChris Larson
This class holds the particular pieces of information about a recipe which are needed for runqueue to do its job. By using it, I think we improve code clarity, reduce method sizes, reduce overuse of primitive types, and prepare for parallel parsing. In addition, this ditches the leaky abstraction whereby bb.cache attempted to hide the difference between cached data and a full recipe parse. This was a remnant from the way things used to be done, and the code using it had to know the difference anyway. If we choose to reimplement caching of the full recipes, we can do it in bb.parse, in a completely transparent way. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-19cache: make loadDataFull a classmethodChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18cache: Add debug msg for a nonexistant dep fileChris Larson
If a recipe depends on a file, and that file is out of date, we show a message, but if that file was removed, we do not, until now. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18cache: use new style classesChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18cache: pyflakes/pep8/pylint cleanupChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18cache: use set for clean, checkedChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18cache: make use of defaultdictChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-18Pre-explode rundeps/runrecs in CacheDataChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-15bb.cache: only log if the respective action was takenBernhard Reutner-Fischer
This avoids alot of misleading log-messages like "Removing FOO from cache" if FOO was not in the cache and as such is not a removal candidate. Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-10-27Make 'cache is clean' message debugChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-24Fix another little bug with undefined var from cacheChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-24bb.cache: fix bug from last commit with loading from cacheChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-23Reduce bb.cache memory usage a bitChris Larson
- Don't store key/value pairs when the value is None - Delete the depends_cache when we're done with it This reduces the memory usage after sync on initial parse by roughly 11.5% on this machine. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-03Switch bitbake internals to use logging directly rather than bb.msgChris Larson
We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-08-03Add support for .bbappend files (see mailing lists for detais)Richard Purdie
(From Poky rev: 63e6ba85677b8aa9f4cf9942a1fccbb8a8c72660) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-03cache: If one virtual of a recipe is uncached, remove all versions from the ↵Richard Purdie
cache (From Poky rev: 7edde59de4fd8b19af55e8377138db6ee3dbb3d8) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-06-10Use bb.utils.mkdirhier in bb.cache to shut up the pending deprecation warningChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: do not chdir unnecessarilyBernhard Reutner-Fischer
previously we called chdir() twice for every target, this patch reduces the amount of chdir() calls via openembedded master from some 16000 to 4. Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: use a set() for __dependsBernhard Reutner-Fischer
to make updating depends easier/more intuitive/eventually faster Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: use max() to determine newest_mtimeBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: use os.path ops instead of statBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: fix typo in error message of cacheValidUpdate()Bernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-04-20Don't try to expand non-string valuesChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-11Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Import fixupsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Formatting cleanupsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Ensure we always utilize the correct messaging domainsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-06-15 13:54:12 +0000