aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2022-06-17cooker: Drop sre_constants usagejansa/1.48Richard Purdie
As reported by Martin Jansa <Martin.Jansa@gmail.com>: bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-16tinfoil/data_smart: Allow variable history emit() to function remotelyRichard Purdie
We can't access the emit() function of varhistory currently as the datastore parameter isn't handled correctly, nor is the output stream. Add a custom wrapper for this function which handles the two details correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-16bin/bitbake-getvar: Add a new command to query a variable value (with history)Richard Purdie
We've talked about having this for long enough. Add a command which queries a single variable value with history. This saves "bitbake -e | grep" and avoids the various pitfalls that has. It also provides a neat example of using tinfoil to make such a query. Parameters to limit the output to just the value, to limit to a variable flag and to not expand the output are provided. [YOCTO #10748] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-03knotty: reduce keep-alive timeout from 5000s (83 minutes) to 10 minutesRoss Burton
The keep alive timeout is excessively long at 83 minutes (5000 seconds), reduce this to 10 minutes: this should be long enough that it rarely triggers in normal builds, but when it does it has useful information. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dcf52157d3635925491783be656c6b76d1efe1a4) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-03knotty: display active tasks when printing keepAlive() messageRoss Burton
In interactive bitbake sessions it is obvious what tasks are running when one of them hangs or otherwise takes a long time. However, in non-interactive sessions (such as automated builds) bitbake just prints a message saying that it is "still alive" with no clues as to what tasks are active still. By simply listing the active tasks when printing the keep alive message, we don't need to parse the bitbake log to identify which of the tasks is still active and has presumably hung. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 30f6c3f175617beea8e8bb75dcf255611e3fc2fd) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-11bitbake: adjust parser error check for python 3.10 compatibilityAlexander Kanavin
The change was introduced in https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75 Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8d3c6cbbe6ee734495713ae3b99c609527842506) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-08bitbake: correct deprecation warning in process.pyAlexander Kanavin
Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aff52fe21a0b27f6302555c1e52a864550eb46ce) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-08bitbake: correct the collections vs collections.abc deprecationAlexander Kanavin
This becomes a hard error in python 3.10. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-03build: Handle SystemExit in python tasks correctlyRichard Purdie
If a python task fails with sys.exit(), we currently see no TaskFailed event. The high level code does detect the exit code and fail the task but it can leave the UI inconsistent with log output. Fix this be intercepting SystemExit explicitly. This makes python task failures consistent with shell task failures. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-03build: Avoid duplicating logs in verbose modeRichard Purdie
With "bitbake -v", for task failures you'd see the log output twice. Avoid this by using the existing "did we print info" switch. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-03process: Don't include logs in error message if piping themRichard Purdie
If the caller is piping the logs, they likely don't want them in the error exception as well. This removes duplicate output from the build output allowing the UI level controls on whether to show logs to work correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-03fetch2: Allow Fetch.download() to warn instead of errorPeter Kjellerstedt
Under some situations it can be allowed for Fetch.download() to fail to fetch a file without causing bitbake to fail. By adding only_warn=True as argument to Fetch.download(), it will call logger.warning() instead of logger.error() and thus not cause build failures. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
2022-02-23tests/fetch: Handle upstream master -> main branch changeRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie
We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 30s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03fetch/git: Handle github dropping git:// supportRichard Purdie
github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-19test/fetch: Update urls to match upstream branch name changesRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-22tests/fetch2: Use our own git server for dtc test repoRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02providers: replace newly added logger.warn() with logger.warning()Denys Dmytriyenko
Commit https://git.openembedded.org/bitbake/commit/?id=78cd63285713fde59506eb2e71a7b7ee59a594ff converted logger.info() to logger.warn(), which is deprecated and instead should use logger.warning(): https://lists.openembedded.org/g/bitbake-devel/topic/82742194#12377 Signed-off-by: Denys Dmytriyenko <denis@denix.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30data_smart: Allow colon in variable expansion regexRichard Purdie
Now that ":" is a valid character in variable key names, it needs to be allowed by the variable expansion code too, to match. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30data_smart/parse: Allow ':' characters in variable/function namesRichard Purdie
It is becomming increasingly clear we need to find a way to show what is/is not an override in our syntax. We need to do this in a way which is clear to users, readable and in a way we can transition to. The most effective way I've found to this is to use the ":" charater to directly replace "_" where an override is being specified. This includes "append", "prepend" and "remove" which are effectively special override directives. This patch simply adds the character to the parser so bitbake accepts the value but maps it back to "_" internally so there is no behaviour change. This change is simple enough it could potentially be backported to older version of bitbake meaning layers using the new syntax/markup could work with older releases. Even if other no other changes are accepted at this time and we don't backport, it does set us on a path where at some point in future we could require a more explict syntax. I've tested this patch by converting oe-core/meta-yocto to the new syntax for overrides (9000+ changes) and then seeing that builds continue to work with this patch. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30BBHandler: Don't classify shell functions that names start with "python*" as ↵Tomasz Dziendzielski
python function If shell function name starts with 'python' or 'fakeroot' parser wrongly assumes it's python/fakeroot function. [YOCTO #14204] Use regex lookahead assertions to check if 'python' expression is followed by whitespace or '(' and if 'fakeroot' is followed by whitespace. Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-13bitbake: providers: selected version not available should be a warningRoss Burton
If the selected version if not available, bitbake will happily attempt to build something else. This should be a loud warning not a small note. (Bitbake rev: 78cd63285713fde59506eb2e71a7b7ee59a594ff) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-04fetch/gitsm: Fix crash when using git LFS and submodulesNiels Avonds
Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] Signed-off-by: Niels Avonds <niels@codebits.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-04runqueue: Fix deferred task issuesRichard Purdie
In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-04bitbake-server: ensure server timeout is a floatRoss Burton
bitbake-server is spawned by process.py and passes the arguments it is given to ProcessServer. There's some type confusion here: bitbake-server is called with a string representation of the timeout, which may be None. If the timeout is not set, pass 0 instead of None. Inside bitbake-server a ProcessServer is created which expects the timeout to be a float not a string, so always float() the value. [ YOCTO #14350 ] Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c93ae1f861208f6d39fd15c84fbcd0e2b54331f5) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27cooker: Include all packages a recipe provides in SkippedPackage.rprovidesPeter Kjellerstedt
The provided packages by a skipped recipe are supposed to be listed in SkippedPackage.rprovides, which is used when generating a meaningful error message when a build fails because of a skipped package. Previously this variable only contained the contents of ${RPROVIDES}. However, most recipes don't define RPROVIDES, they define RPROVIDES_<pkg> for each package they provide. Additionally, the recipe provides the packages in PACKAGES without them being included in ${RPROVIDES}. Before this change, having a runtime dependency on a skipped non-recipe package would result in a build error stating that the build failed because the package was skipped, but without providing any reason for why it was skipped. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27cache: Make CoreRecipeInfo include rprovides_pkg for skipped recipesPeter Kjellerstedt
This will be needed by SkippedPackage in the cooker. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-24Force parser shutdown after catching an exceptionJan Brzezanski
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-24runqueue: Add setscene task overlap sanity checkRichard Purdie
We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-24runqueue: Fix task execution corruption issueRichard Purdie
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-09__init__.py: Fix bitbake debug log handlingRichard Purdie
For a while I've been puzzled as to why debug logging from runqueue wouldn't appear on the console with -DD. The logic in the bbdebug handling is inverted so fix it and now we see the expected messages from runqueue with -D and -DD. This should then let us debug other issues using those log messages. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-11lib/bb/fetch2/__init__.py: drop _PYTHON_SYSCONFIGDATA_NAME unsettingAlexander Kanavin
With introduction of python3targetconfig class in core this is no longer needed. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-02fetch/git: download LFS content too during do_fetchMatt Hoosier
Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-26data_smart: Ensure hash reflects vardepvalue flags correctlyRichard Purdie
The get_hash() function is used to decide if the base configuration has changed and hence whether a reparse is required. The vardepvalue flag's value was not expanded but it is often used in contexts like: METADATA_REVISION = "${@base_detect_revision(d)}" METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" which in it's unexpanded form means reparsing doesn't happen when it should as the data appears unchanged. Update get_hash to expand the values of vardepvalue so reparsing works as expected. This avoids basehash mismatch errors such as the one recently caused by using METADATA_REVISION in poky.conf's DISTRO_VERSION variable. The issue there could be exposed by a recipe using DISTRO_VERSION with the sequence: bitbake os-release <change the revision of the metadata with a dummy commit> bitbake os-release -C install which was caused because METADATA_REVISION changed but the metadata didn't reparse. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-26main: Handle cooker daemon startup errorStacy Gaikovaia
On startup, bitbake spawns a cooker daemon and waits for it's acknowledgement signal. If the acknowledgement doesn't happen in time,the bitbake object will fail to initialize and exit. The error that occurs in this case isn't handled by the existing try - catch block because SystemExit inherits from a different base Exception class. This commit adds SystemExit to the list of expected bitbake server startup errors. [YOCTO #13993] Signed-off-by: Stacy Gaikovaia <stacy.gaikovaia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-26tests/fetch: Update upstream master->main branchname transitionRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-19toaster: Update documentation links to new URLsReyna, David
Update the Toaster documentation links to the new Yocto Project URLs. [YOCTO #14092] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-19bitbake: Bump version to 1.48.0 ready for the new releaseRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-16ui/toasterui: Fix startup faults from incorrect event sequencingRichard Purdie
Toaster has been failing to start correctly when in interactive mode. The issue is due to setEventMask being called (which triggers parsing) before the environment has been sent from the UI over to the server. This means PATH isn't setup, which causes the sanity checks on HOSTTOOLS to fail in base.bbclass. The fix is to ensure the environment is sent to the server before other commands are run. The pain in debugging this highlights other improvements to the logging are needed. [YOCTO #14079] Reviewed-by: Tim Orling <timothy.t.orling@linux.intel.com> Tested-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-15toaster: Enable Gatesgarth branch in place of ZeusDavid Reyna
Toaster directly supports the last two stable branches of Yocto Project. With "Gatesgarth " being released, it is time to replace "Zeus". [YOCTO #14086] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-15lib/bb/ui/knotty: fix typo in parseprogressTim Orling
After parseprogress.finish() it was intended to set parseprogress to None, but a typo means this is not happening. Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-15bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI urlCharlie Davies
This commit replaces the instances where escaped double quotes are used to support SRC_URI url containing spaces with the more pythonic shlex.quote(). Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-11process: Show command exceptions in the server log as wellRichard Purdie
There are autobuilder logs where the server commands are failing but we have no debug info in the server log. Improve this to try and understand what is failing. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-09tinfoil: When sending commands we need to process eventsRichard Purdie
The server may be displaying useful information for the user through log messages so we should display anything that has been sent. Its either this or expecting every UI to implement this code around every command call which isn't good API. [YOCTO #14054] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-09command: Ensure exceptions inheriting from BBHandledException are visibleRichard Purdie
Previous changes allowed BBHandledException to be detected but not exceptions which inherit from it. Fix this. The code really needs totally reworking to preserve the exceptions. [YOCTO #14054] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-05sphinx: rename Makefile.sphinxNicolas Dechesne
Now that the DocBook files are removed, we can rename the top level Makefile. Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-10-05sphinx: remove DocBook filesNicolas Dechesne
The BitBake documentation was migrated to Sphinx. Let's remove the deprecated DocBook files. Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-10-05docs: static: theme_overrides.css: fix responsive design on <640px screensQuentin Schulz
From experience the body takes the whole space anyway and the text stays within the screen boundaries by default, no need to make the min-width 640px then. Signed-off-by: Quentin Schulz <foss@0leil.net> Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-10-05docs: sphinx: report errors when dependencies are not metNicolas Dechesne
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-10-05docs: update README file after migrationg to SphinxNicolas Dechesne
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>