aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2021-04-12bitbake-user-manual: add REQUIRED_VERSION and adjust PREFERRED_VERSION entrypaule/doc-git-pwPaul Eggleton
Add REQUIRED_VERSION, add a reference to it in PREFERRED_VERSION and adjust the opening statement to read slightly better. Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
2021-04-12bitbake-user-manual: document no support for using passwords in git URLsPaul Eggleton
This is based on the comment added in revision aded964eed4ce5a725ed1ab477efabc86b1aa481. Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
2021-04-09bitbake: Update version to 1.50.0 stable release seriesRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06runqueue: Further fixes for confused setscene tasksRichard Purdie
There is further evidence of tasks ending up being "covered" and "notcovered" which shouldn't happen and is bad. The code that caused this problem last time appears to have issues where stamps for tasks already exist. Split out the setscene stamp checking code to a separate function and use this when checking "hard dependencies" (like pseudo-native) so that if the stamps exist and it will be "covered", it is not put on the notcovered list. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23doc: Update links to documentationMichael Opdenacker
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23runqueue/event: Add an event for notifying of stale setscene tasksRichard Purdie
Use the new functionality in build.py to identify stale setscene tasks and send an event to the metadata listing them. The metadata then has the option of performing cleanup operations if it thinks that appropriate. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23build: Add find_stale_stamps functionRichard Purdie
Add a new function which compares the stamp filename we want (including taskhash) with what is in the stamp directory (using the clean mask). This tells us which stamp files are stale and are due to be rerun. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22doc: fix syntax error in layer.conf exampleRobert P. J. Day
While this example really needs to be rewritten to not define multiple patterns in the same layer.conf, as long as it's there, it might as well be syntactically correct. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22doc: fix glossary link for BB_INVALIDCONF variableRobert P. J. Day
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-22doc: mention that addtask handles multiple dependenciesRobert P. J. Day
Add a note explaining that "addtask" can accept multiple dependencies, just in case someone runs across such an example and is confused. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15doc: move BBFILES_DYNAMIC for alphabetical orderRobert P. J. Day
Since BBFILES_DYNAMIC does not have a "BB_" prefix, it belongs further down in the variable glossary. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15codeparser: Fix TypeError in bitbake debug modeTomasz Dziendzielski
Commit 75f87db413 fixed the confusion between bitbake and python logger but in codeparser still old method of setting debug level was used causing TypeError, because debug level value was incorrectly returned and assigned to event.msg. | File "./bitbake/lib/bb/ui/knotty.py", line 660, in main | event.msg = event.fn + ': ' + event.msg | TypeError: can only concatenate str (not "int") to str [YOCTO #14298] Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15bblayers/query: Update to handle REQUIRED_VERSIONRichard Purdie
Ii is unclear whether any changes are needed to bblayers are needed to handle the extra data from REQUIRED_VERSION. Update to at match the new API, at present it doesn't look necessary to handle the required version data. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: cooker: Add REQUIRED_VERSION checksCharlie Davies
Add logic to check that if REQUIRED_VERSION has been set that the cooker class method findBestProviders properly handles the case where the REQUIRED_VERSION has not been found. Fixes [YOCTO #10096] Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: lib: add Required Version to Bitbake --show-versions commandCharlie Davies
This commit adds a new column to the Bitbake --show-versions command called Required Version. This column will display any packages which have a REQUIRED_VERSION successfully set. Fixes [YOCTO #10096] Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: providers: check for REQUIRED_VERSION in _filterProvidersCharlie Davies
Before the REQUIRED_VERSION variable was introduced the PREFERRED_VERSION variable allowed for a fallback to the next most suitable version. Since REQUIRED_VERSION does not allow a fallback to a different version implement a check in the _filterProviders function to make sure that if a requested REQUIRED_VERSION is not found then the function returns no eligible providers have been found. Fixes [YOCTO #10096] Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: providers: introduce logic for REQUIRED_VERSION variableCharlie Davies
This commit adds checks during the findPreferredProvider function for a new variable REQUIRED_VERSION. This can be set, in exactly the same manner as PREFERRED_VERSION, on a per package basis to enforce the use of a particular version of a package. REQUIRED_VERSION is similar in behaviour to PREFERRED_VERSION except if the version specified by REQUIRED_VERSION is not found an error occurs and the execution of Bitbake stops. Fixes [YOCTO #10096] Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: providers: consistent single line formatting between functionsCharlie Davies
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12bitbake: providers: remove unneeded logging callCharlie Davies
The logging line for when no eligible providers are found only adds unnecessary noise and is not required. As the stack unwinds better logging occurs which satisfactorily describes the no eligible providers error condition. Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12gitignore: ignore runqueue-tests/bitbake-cookerdaemon.logPaul Gortmaker
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12tests/color: Fix event register to pass the datastoreRichard Purdie
Adding the "if d is None" to the event handling code meant some of these tests stopped working. The reason is that len(d) was zero but not equal to None. Passing the data object to the register() function in the test correctly registers the event handler and avoids the problem, it just happened to work previously, incorrectly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11event: Fix another 'if d' test to test for None explictlyRichard Purdie
This should have been fixed in the previous patch, well spotted Chris! Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11event: Fix multiconfig event handler change performance regressionsRichard Purdie
There were two issues in this code, firstly the code could stack duplicates in the variable, secondly, calling "if data" caused the datastore to compute len(data) which is comparitively expensive. Checking "if data is not None" is much much faster/cheaper. The issue was clear from "bitbake -p -P" output where the time in register() showed large amounts of time in the __len__ function of the datastore. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11tinfoil: Honor quiet when parsing recipesManuel Leonhardt
When using parse_recipes, honor quiet so that scripts and custom plugins for recipetool are able to mute progress bars from bitbake that would otherwise print to STDOUT. Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11runqueue: Print pseudo.log if fakeroot task failedTomasz Dziendzielski
Currently if pseudo fails we can only see the path to pseudo.log. If we have no access to server and can only rely on bitbake log then debugging becomes impossible. This printing needs to be added in runqueue level, not inside task execution, because in some cases task fails with pseudo abort really early and we don't even see any log. In this change I'm adding pseudo log printing in every fakeroot task failure that logged `mismatch`, `error` or `fatal` to logfile, because we have no other way to communicate with pseudo if it failed or not. Only lines from last pseudo server execution will be printed. Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Adrian Walag Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com> Signed-off-by: Wiktor Baura <wbaura@gmail.com> Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11event.py: fix regression about INVALIDCONFChen Qi
A previous commit uses __BBHANDLERS_MC to record all handlers, but it does not take into consideration of INVALIDCONF, thus causing regression. We need to record the name before returned AlreadyRegistered, otherwise, when reparsing due to INVALIDCONF, bascially all handlers are not called. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11docs: Add AZ_SAS definition to glossaryAlejandro Hernandez Samaniego
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11docs: Add Az fetcher documentationAlejandro Hernandez Samaniego
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11bitbake: Add Azure Storage fetcher implementationAlejandro Hernandez Samaniego
Allows bitbake to fetch from an Azure Storage account. The fetcher submodule is compatible with the az:// URI protocol, its functionality is based on bitbakes wget fetcher, superior in performance to using a propietary tool like azcopy which can handle cloud storage account operations with more functionality (that we dont need in a fetcher) but less compatibility. A sample URI uses can be defined in the following way: SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz" This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.: SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n" PREMIRRORS_prepend = "\ git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ " Can also be used with non-public access Azure Storage accounts/containers via a Shared Access Signature by declaring the AZ_SAS variable which will be automatically used by the fetcher: AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature" Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09__init__.py: Fix bitbake debug log handlingRichard Purdie
For a while I've been puzzled as to why debug logging from runqueue wouldn't appear on the console with -DD. The logic in the bbdebug handling is inverted so fix it and now we see the expected messages from runqueue with -D and -DD. This should then let us debug other issues using those log messages. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09runqueue: Add setscene task overlap sanity checkRichard Purdie
We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-09runqueue: Fix task execution corruption issueRichard Purdie
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21fetch2/wget: Avoid crashing when connection drops mid checkstatusAlejandro Hernandez Samaniego
If an exception is raised when running host python code, the fetcher immediately crashes, this might be temporary depending on the servers reliability. Catch the exception when the connection was reset and try once again to fetch the data. File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto 0665: if self._timeout_occurred: 0666: raise OSError("cannot read from timed out object") 0667: while True: 0668: try: *** 0669: return self._sock.recv_into(b) 0670: except timeout: 0671: self._timeout_occurred = True 0672: raise 0673: except error as e: Exception: ConnectionResetError: [Errno 104] Connection reset by peer Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-20event: Fix broken builds when multiconfig has a hyphen in the nameJoshua Watt
5f7fdf7b2d ("bitbake: event: Prevent bitbake from executing event handler for wrong multiconfig target") broke multiconfig builds contain a hyphen, since it's attempt to use the multiconfig as part of a function name and python functions are not allowed to contain a hyphen. Rework the bitbake multiconfig test to test a multiconfig with a hyphen and one with an underscore to validate this doesn't break in the future. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-20Force parser shutdown after catching an exceptionJan Brzezanski
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-20bitbake: providers: use pythonic empty list checkCharlie Davies
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-20bitbake: providers: fix incorrect return type bugCharlie Davies
If no eligible providers are found then an integer zero is returned. This causes the following error, in two possible places in taskdata.py, when the return value is used in a list comprehension: [snip] eligible = [p for p in eligible if not p in self.failed_fns] TypeError: 'int' object is not iterable [\snip] Fix by returning the variable eligible itself, of type list. Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16contrib: Add Dockerfile for building hash serverJoshua Watt
Adds a Dockerfile for build the reference hash equivalence server in a container Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16bitbake: Bump version to 1.49.2Richard Purdie
This allows metadata to rely upon BB_DEFAULT_UMASK. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16bitbake-worker/runqueue: Add support for BB_DEFAULT_UMASKRichard Purdie
Currently each task has to have a umask specified individually. This is leading to determinism issues since it is easy to miss specifying this for an extra task. Add support for specifing the default task umask globally which simplifies the problem. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-11event: Prevent bitbake from executing event handler for wrong multiconfig targetTomasz Dziendzielski
When multiconfig is used bitbake might try to run events that don't exist for specific mc target. In cooker.py we pass `self.databuilder.mcdata[mc]` data that contains names of events' handlers per mc target, but fire_class_handlers uses global _handlers variable that is created during parsing of all the targets. This leads to a problem where bitbake runs event handler that don't exist for a target or even overrides them - if multiple targets use event handler with the same name but different code then only one version will be executed for all targets. See [YOCTO #13071] for detailed bug information. Add mc target name as a prefix to event handler name so there won't be two different handlers with the same name. Add internal __BBHANDLERS_MC variable to have the handlers lists per machine. Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10bitbake: Bump release to 1.49.1Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10logging: Make bitbake logger compatible with python loggerJoshua Watt
The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09bblayers/action: When adding layers, catch BBHandledExceptionRichard Purdie
When adding a layer, parse error can occur, raising BBHandledException. Catch this and error, aborting the layer add to meet user expectations. [YOCTO #14054] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09cooker: Ensure reparsing is handled correctlyRichard Purdie
From tinfoil, if you edit bblayers.conf and break it, then call parseConfiguration (e.g. by adding a bad layer with bitbake-layers), the system doens't show any parse error yet it should. Add in a call to the updateCache function so that things really are reparsed when requested. Partially fixes [YOCTO #14054] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09BBHandler: Don't classify shell functions that names start with "python*" as ↵Tomasz Dziendzielski
python function If shell function name starts with 'python' or 'fakeroot' parser wrongly assumes it's python/fakeroot function. [YOCTO #14204] Use regex lookahead assertions to check if 'python' expression is followed by whitespace or '(' and if 'fakeroot' is followed by whitespace. Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09hashserv: Add get-outhash messagePaul Barker
The get-outhash message can be sent via the get_outhash client method. This works in a similar way to the get message but looks up a db entry by outhash rather than by taskhash. It is intended to be used as a read-only form of the report message. As both handle_get_outhash and handle_report use the same query string we can factor this out. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09hashserv: server: Support searching upstream for outhashPaul Barker
Use the new get-outhash message to perform a read-only query against an upstream server (if present) when a reported taskhash/outhash combination is not found in the current database. If a matching entry is found upstream it is copied into the current database so it can be found by future queries. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09hashserv: Add short forms of remaining command line argumentsPaul Barker
Short form arguments are added for convenience. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09hashserv: Support upstream command line argumentPaul Barker
The hashserv server already implements support for pulling hash data from another "upstream" server. Add the -u/--upstream argument to the bitbake-hashserv app to expose this functionality to users. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>