aboutsummaryrefslogtreecommitdiffstats
path: root/bitbake
AgeCommit message (Collapse)Author
2016-12-16bitbake: BBHandler: use with instead of open/closeRoss Burton
This is more pythonic and can handle unclosed file warnings better than the previous code structure. (Bitbake rev: 50633012a64a3b5f0662145e29ff426374fb7683) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16bitbake: cooker/command: Drop expanded_dataRichard Purdie
Some of our metadata assumes that BuildStarted and BuildCompleted events see the same data store. This is the case for buildTarget but not for buildFile and recent changes mean this is now a problem. The update_data() call is now an empty operation and there is no difference between the expanded_data and data so we can simply remove the expanded_data and its references and use data everywhere. This has been inteded for a while but the above issue makes this more pressing to finally clean up. (Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16bitbake: bitbake: remove True option to getVar calls (take 2)Joshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (a follow on patch to fix up a few recent introductions) (Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: lib/bb/build: enable access to logger within tasksPaul Eggleton
In certain circumstances it can be useful to get access to BitBake's logger within a task; the main example is in OpenEmbedded's image construction code where we want to be able to check the log file for errors and warnings, but we don't want to see any errors or warnings that were emitted through the logger; so we need a way to exclude those. In order to do this, pass the logger object into the task via a BB_TASK_LOGGER variable, and add a logging handler class to bb.utils that can be added to it in order to keep a list of warnings/errors that have been emitted. (Bitbake rev: f1cd6fab604f14d8686b1d783cbfe012d923ee42) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: server/process: don't change UI process signal handler on terminatePaul Eggleton
On terminating the connection to the server, we were disabling SIGINT - and this is executed on the UI side. I'm not sure whether the intention here was to undo the SIGINT disabling we did in the server, and it was just a mistake that it disabled rather than restored and it's run on the wrong side, or whether we wanted to stop the user from breaking out of the shutdown code - the commit message provides no clues either way. Regardless, we do not want to permanently disable Ctrl+C here - it's legitimate to terminate the connection to the server and then re-establish it within the same process; at least currently, devtool modify by virtue of using tinfoil in two separate parts of the code does this, and the result of this disabling is that during the second tinfoil usage we can potentially be parsing all recipes without the ability to easily interrupt the process. (Bitbake rev: 58c60a951229dcbd8253863fb24228d046c23f6e) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: siggen: add means of ignoring basehash mismatchPaul Eggleton
If you run the setVariable command to set variables then you end up causing the basehash to not match the previously computed values, which triggers error messages. These mismatches are expected, so add a means of disabling them. (Bitbake rev: 5a80c0e210f26526afbe8f266b7b1a9c03334967) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: runqueue: enable setVariable command to affect task executionPaul Eggleton
Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. (Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: data_smart: support serialisationPaul Eggleton
The COW object used within VariableHistory can't be serialised itself, but we can convert it to a dict when serializing and then back when deserialising. This finally allows DataSmart objects to be serialized. NOTE: "serialisation" here means pickling, not over XMLRPC or any other transport. (Bitbake rev: bbbb2a53d5decf3b613a92c4ff77c84bfc5d4903) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: cooker: allow buildFile warning to be hidden programmaticallyPaul Eggleton
If we want to use this function/command internally, we don't want this warning shown. (Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: tinfoil: pass datastore to server when expanding python referencesPaul Eggleton
If you're expanding a value that refers to the value of a variable in python code, we need to ensure that the datastore that gets used to get the value of that variable is the client-side datastore and not just the part of it that's on the server side. For example, suppose you are in client code doing the following: d.setVar('HELLO', 'there') result = d.expand('${@d.getVar("HELLO", True)}') result should be "there" but if the client part wasn't taken into account, it would be whatever value HELLO had in the server portion of the datastore (if any). (Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: tinfoil: implement server-side recipe parsingPaul Eggleton
It's not really practical for us to parse recipes on the client side, we need to do it on the server because that's where we have the full python environment (including any "pure" python functions defined in classes). Thus, add some functions to tinfoil do this including a few shortcut functions. (Bitbake rev: 8f635815d191c9d848a92d51fdbf5e9fd3da1727) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: remotedata: enable transporting datastore from the client to the serverPaul Eggleton
For the purposes of server-side parsing and expansion allowing for client-side use of the datastore, we need a means of sending a datastore from the client back to the server, where the datastore probably consists of a remote (server-side) original plus some client-side modifications. To do this we need to take care of a couple of things: 1) xmlrpc can't handle nested dicts, so if you enable memres and simply try passing a serialised datastore then things break. Instead of serialising the entire datastore, just take the naive option of transferring the internal dict alone (as a list of tuples) for now. 2) Change the TinfoilDataStoreConnector object into simply the handle (number) when transmitting; it gets substituted with the real datastore when the server receives it. (Bitbake rev: 784d2f1a024efe632fc9049ce5b78692d419d938) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: tinfoil: rewrite as a wrapper around the UIPaul Eggleton
Rewrite tinfoil as a wrapper around the UI, instead of the earlier approach of starting up just enough of cooker to do what we want. This has several advantages: * It now works when bitbake is memory-resident instead of failing with "ERROR: Only one copy of bitbake should be run against a build directory". * We can now connect an actual UI, thus you get things like the recipe parsing / cache loading progress bar and parse error handling for free * We can now handle events generated by the server if we wish to do so * We can potentially extend this to do more stuff, e.g. actually running build operations - this needs to be made more practical before we can use it though (since you effectively have to become the UI yourself for this at the moment.) The downside is that tinfoil no longer has direct access to cooker, the global datastore, or the cache. To mitigate this I have extended data_smart to provide remote access capability for the datastore, and created "fake" cooker and cooker.recipecache / cooker.collection adapter objects in order to avoid breaking too many tinfoil-using scripts that might be out there (we've never officially documented tinfoil or BitBake's internal code, but we can still make accommodations where practical). I've at least gone far enough to support all of the utilities that use tinfoil in OE-Core with some changes, but I know there are scripts such as Chris Larson's "bb" out there that do make other calls into BitBake code that I'm not currently providing access to through the adapters. Part of the fix for [YOCTO #5470]. (Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: command: provide a means to shut down from the client in memres modePaul Eggleton
In memory resident mode we don't really want to actually shut down since it's only the client going away. (Bitbake rev: 74db369c46043116359101cab70486afd82372c0) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: data_smart: implement remote datastore functionalityPaul Eggleton
This allows you to maintain a local reference to a remote datastore. The actual implementation of the remote connection is delegated to a connector object that the caller must define and supply. There is support for getting variable values and expanding python references (i.e. ${@...} remotely, however setting variables remotely is not supported - any variable setting is done locally as if the datastore were a copy (which it kind of is). Loosely based on an earlier prototype implementation by Qing He. (Bitbake rev: a3edc3eefa2d03c4ad5d12187b32fa4dc495082a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: server/xmlrpc: send back 503 response with correct encodingPaul Eggleton
If you send back a string here you get "TypeError: 'str' does not support the buffer interface" errors in bitbake-cookerdaemon.log and "IncompleteRead(0 bytes read, 22 more expected)" errors on the client side. (Bitbake rev: 0d659a7dfe5fb096f8aa4380320f9e2a464b3cb5) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: knotty: fix --observe-only optionPaul Eggleton
If we're in observe-only mode then we cannot run commands that would affect the server's state, including getSetVariable, so prevent that from being called in observe-only mode. (Bitbake rev: 2c5a8661430edebff67ab4a108995033d182b5d6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: knotty: make quiet option a level optionPaul Eggleton
Allow you to specify -q / --quiet more than once to reduce the messages even further. It will now operate as follows: Level Option Result ----- ------ ---------------------------------------- 0 Print usual output 1 -q Only show progress and warnings or above 2 -qq Only show warnings or above 3+ -qqq Only show errors (Bitbake rev: 6cf2582e17c28ca04f5cfb59858c4a9778c700d4) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14bitbake: data_smart: fix resetting of reference on variablehistoryPaul Eggleton
There is no "datasmart" member, only dataroot. This dates back to the original implementation of variable history support - it's surprising we haven't noticed the issue until now, but I guess it's rare to change a copy of a datastore in a manner which using the old reference would cause an issue. (Bitbake rev: febd5534b07edfdef15cedb0578730c582c7373f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: views Remove old code that converts template context to JSONMichael Wood
Remove the template context to JSON decorator function as this is deprecated by having a proper REST API. (Bitbake rev: b65a8193368ffa1d15af24a6acde8dce6bd4d383) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: js test Update js tests to use correct url for layer RESTMichael Wood
Update js tests so that we use the new Layer REST API. (Bitbake rev: 81764ce3ebf0d2fcb3dc6965f6f07931d39e5524) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: Switch front end to use Layer get REST APIMichael Wood
Switch the front end to use the proper REST API for retrieving layer information. (Bitbake rev: 5ea25c49091f4d4b5007af948e063ed25ba5766f) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: typeaheads Layers add url to layer REST API to the layerMichael Wood
Add the url to the Layer typeahead so that this can be used later on by the front end code to look up layer details. (Bitbake rev: d195f24a1b30ae8698bff5e87308347b9596a2e2) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: api Add GET REST API for Layer informationMichael Wood
Add a get API for returning information about layers. (Bitbake rev: 3fa5170c98c699d2a7a8380b696fc599efcc5dee) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: importlayer Add git revision typeahead to that input fieldMichael Wood
Add the front end mechanism to load the typeahead for the git revision field on importing a layer. Also fix one indentation issue and update the js test. (Bitbake rev: 28114be42174095b812a93d4b5a0e01e953d74f8) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: typeaheads Add a git revisions suggestionsMichael Wood
When we're importing a layer it's useful to suggest available git revisions of the layers. This uses git ls-remote to fetch the revisions and then filter on this. Caching is added for this typeahead to avoid having to fetch this information multiple times in a single session. [YOCTO #8429] (Bitbake rev: a94ae3ad0cb07a52004143c6f86f371b9e330e9f) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: typeaheads widgets Fix flake8 issues and remove redundant codeMichael Wood
Fix flake8 issues and remove redundant __init__ function definitions from typeaheads (likely a copy and paste error). (Bitbake rev: be1f9f48da480d813e3364815cb3e002ba70dd22) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: tests Update import layer test to use new Layer add apiMichael Wood
(Bitbake rev: d4c79cd60c12d329c533add73e88b7184dca6ca3) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: importlayer Use new layer add APIMichael Wood
Switch the importlayer.js to use the new REST API for importing a new layer. (Bitbake rev: 6475fd7e0d2488bf300b75322f2c00297cd1440b) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: api Add layer Add apiMichael Wood
Add layer adding REST api and remove old views method. (Bitbake rev: 0c8e41d2217fd568a84e857d1be230fcfd4bb5c7) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: models Layer_Version get_detailspage_url optional project_idMichael Wood
Allow passing none as the project id, this is convenient for layers which belong to projects already and therefore have their own project field. Add documentation string to function (Bitbake rev: 78dc7b9f1801e7f4c266ba1369e5706f177ddaa1) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: models Layer fix whitespaceMichael Wood
(Bitbake rev: 8116cb4f71de09eff124cd48d80cc7a7c64da5e1) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12bitbake: toaster: tablejs Add visual indicator for table data loadingMichael Wood
Add a visual indicator when the data is changing in the table, for instance if it's being re-ordered, searched or paginated. [YOCTO #10104] (Bitbake rev: 554c4992b33b77526b4b37c7484f1dd00032ddef) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: bitbake-user-manual: Added new "Line Joining" section.Scott Rifenbark
Fixes [YOCTO #10444] Added a new section about syntax called "Line Joining". The section describes how you can use the backslash character (\) to joing stuff. (Bitbake rev: a2768ecae7846d72a1bdb7cbbc5e8d242af854f6) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: bitbake-user-manual: Refreshed the 'bitbake --help' output.Scott Rifenbark
(Bitbake rev: d7121d919beaa4be2c18d8f0bbd4afe01590eb7b) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: toaster: Reference pip3 not pipRichard Purdie
Now that we use python3, we should refer to pip3, not pip. [YOCTO #10774] (Bitbake rev: 99136f5f591deef0c96d9aea2dbea1c216f38121) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: cooker: Handle inofity queue overflows more gracefullyRichard Purdie
If many files change and the inotify queue overflows, rather than print a traceback, invalidate the caches and warn the user. [YOCTO #10676] (Bitbake rev: 058f8517c041b80e8b591ad7d34a68281b2d03fc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: cooker: Fix world taskgraph generation issueRichard Purdie
The processing of the "do_" prefix to tasks is currently inconsistent and has resulted in "bitbake world -g" being broken as task prefixes don't get handled correctly. Make the "do_" task prefix handling consistent through various codepaths. [YOCTO #10651] (Bitbake rev: 3d7186353e804c9410096c408bc337a98c8b33fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: utils: Avoid traceback errorsRichard Purdie
Avoid errors like: ERROR: Exception handler error: 'NoneType' object has no attribute 'decode' (Bitbake rev: 1aeb45abe56061f044c2347889c191d5256ff21f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: runqueue: Send BB_TASKDEPDATA for setscene tasksRichard Purdie
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore generate and send this data. In this case its a "pre collapsed" tree but that is fine for the use cases in question. (Bitbake rev: 38b857d086af43af6ea3aa60d3876a2c9b225401) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: runqueue: Add the taskhash to BB_TASKDEPDATARichard Purdie
Its useful to know the task hash in code using TASKDEPDATA so add this data to the data structure. The recipe specific sysroots in OE need this data. (Bitbake rev: 758867e8dc74283bb1f031e158ec54cefdd5c2a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake: bitbake-worker: Further IO performance tweaksRichard Purdie
Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable (Bitbake rev: 9375349e27b08b4d1cfe4825c042d4c82120e00b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: lib/bs4: Fix imports from html5lib >= 0.9999999/1.0b8Daniel Lublin
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules have moved from _base to base. Handle this, while staying compatible with earlier versions. (Bitbake rev: 1679188f9c55c615cae780f2b5e6852dea9cf2ec) Signed-off-by: Daniel Lublin <daniel@lublin.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: cooker: convert type which needs to be marshalledSujith H
We assume that the value taken by variable v can be string, integer or any type which can be marshalled by xmlrpc. This change would help us to convert the non marshallable types to string. So that we don't get exception from xmlrpc. [YOCTO #10740] (Bitbake rev: efb0e47479b3526bc047112f7200087c5844bba4) Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: toaster: browser tests - add Selenium Docker container as driverbrian avery
Adds the ability to specify a Selenium Docker container server as a driver. This allows for repeatable tests independent of host. Currently we assume you are using the Firefox container. Instructions are located in the README in tests/browser. (Bitbake rev: 7df842f8f8b2ae640109ed06729ab59c9469fc64) Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: depexp: fix string formatting.Ismo Puustinen
The parameters to Python string formatting need to be inside a tuple. (Bitbake rev: 3c82af11b89cf251c3e56725a1eed2d3f4bd835b) Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: monitordisk: add eventPatrick Ohly
The current disk usage is interesting and may be worth logging over time as part of the build statistics. Instead of re-implementing the code and the configuration option (BB_DISKMON_DIRS), the information gathered by monitordisk.py is made available to buildstats.bbclass via a new event. This has pros and cons: - there is already a useful default configuration for "interesting" directories - no code duplication - on the other hand, users cannot configure recording separately from monitoring (probably not that important) (Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: monitordisk.py: minor code and comment cleanupPatrick Ohly
There's no need to encode and decode the hash key as a single string, a tuple works just fine. Iterating over entries can be written more concisely. Entries in the stat results are integers, not floating point values. (Bitbake rev: 3c943e989964382c0b819d92de26a0c914ebed33) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: runqueue.py: monitor disk space at regular time intervalsPatrick Ohly
Hooking the disk monitor into the regular heatbeat event instead of the runqueue solves two problems: - When there is just one long running task which fills up the disk, the previous approach did not notice that until after the completion of the task because _execute_runqueue() only gets called on task state changes. As a result, aborting a build did not work in this case. - When there are many short-lived tasks, disk space was getting checked very frequently. When the storage that is getting checked is on an NFS server, that can lead to noticable traffic to the server. (Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake: cooker process: fire heartbeat event at regular time intervalsPatrick Ohly
The intended usage is for recording current system statistics from /proc in buildstats.bbclass during a build and for improving the BB_DISKMON_DIRS implementation. All other existing hooks are less suitable because they trigger at unpredictable rates: too often can be handled by doing rate-limiting in the event handler, but not often enough (for example, when there is only one long-running task) cannot because the handler does not get called at all. The implementation of the new heartbeat event hooks into the cooker process event queue. The process already wakes up every 0.1s, which is often enough for the intentionally coarse 1s delay between heartbeats. That value was chosen to keep the overhead low while still being frequent enough for the intended usage. If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying the delay in seconds between these heartbeat events. (Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>