aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2023-05-05tests/fetch: Default to https git protocol where possibleRichard Purdie
The recommendation from server maintainers is that the https protocol is both faster and more reliable than the dedicated git protocol at this point. Switch to it where possible. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-02cooker: Log config and parse cache status changesRichard Purdie
It can be hard to tell why bitbake does some things slowly. Log the changes to the base configuration and parse cache status so that it becomes clear from the logs when the cache invalidation causes a slowdown. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-27bitbake-user-manual: fix BB_RUNFMT's default valueUlrich Ölmann
According to the code the default value is slightly different (see [1] which reflects the current tip commit at the time of writing). [1] https://git.openembedded.org/bitbake/tree/lib/bb/build.py?id=d97d62e2cbe4bae17f0886f3b4759e8f9ba6d38c#n237 Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22bitbake-user-manual: add BB_HASH_CODEPARSER_VALSdankm/bb-go-fetcherPaul Eggleton
New variable for 2.4. Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22bitbake-user-manual: add addpylib and BB_GLOBAL_PYMODULESPaul Eggleton
New directive and variable for 2.4. Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22bitbake-user-manual: document BB_CACHEDIRPaul Eggleton
New variable for 2.4. Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-20cooker: do not abort on single ctrl-cEnrico Scholz
After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately when a single ctrl-c is pressed. Patch restores the previous behavior where a single ctrl-c waits for active tasks to finish. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094 Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-20npmsw fetcher: Avoid instantiating Fetch class if url list is emptySvend Meyland Nicolaisen
Recipes containing both git and npmsw sources in the SRC_URI fail during fetch from the shrinkwrap. It seems that when the fetcher is fetching from the shrinkwrap, the SRCREV variable has been deleted but it till ends up fetching from the git source resulting in an error because SRCREV is undefined. The root cause of this is that the Fetch class defaults to urls from the SRC_URI when the urls parameter contains an empty list. This patch will ensure that Fetch is not instantiated if the urls list is empty. Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13runqueue: fix PSI check calculationChen Qi
The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13bitbake: ConfHandler: Allow variable flag name with a single characterKai Kang
Update regex pattern to allow variable flag name with a single character. Regression tests have also been updated in `bb.parse` and `bin/bitbake-selftest -k ParseTest` has been successfully executed. Eliminate a trailing space as well. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13event: add bb.event.ParseErrorMingli Yu
Add bb.event.ParseError to let error-report can catch this kind of error. Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-06fetch2/crate: create versioned 'name' entriesEnrico Scholz
It is common for rust packages to depend on different versions of the same crate. E.g. | crate://crates.io/windows_x86_64_msvc/0.42.2 \ | crate://crates.io/windows_x86_64_msvc/0.48.0 \ Identification only by the plain crate name makes the sha256sum ambiguous | SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" | SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a" and requires lot of manual work to fix the SRC_URI list. Making the 'name' property unique by appending the version allows direct copy & paste of reported sha256sum errors to complete the crates list. Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-05bitbake: Bump to version 2.4.0Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-03fetch2: Display all missing checksum at onceFrederic Martinsons
Instead of only display the first and stop. For recipe (crago based) that can contains several artifacts to fetch with their checksum, it will particularly handy to display all of missing one. An example of error message would be NOTE: Executing Tasks ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe: SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4" SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b" SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9" SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b" ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe') ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025 ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1' Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-29bitbake: bitbake-user-manual: Update EXCLUDE_FROM_WORLDRobert Yang
Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-29cache: Make EXCLUDE_FROM_WORLD booleanRobert Yang
Fixed: Set EXCLUDE_FROM_WORLD = "0" in recipe_A: $ bitbake world -g Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem: EXCLUDE_FROM_WORLD = "1": Not build in world EXCLUDE_FROM_WORLD = "0": Add back to world build Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-28fetch2/local: Mention the value of localpath in failure messageRichard Purdie
We have an obtuse rare and intermittent bitbake fetch failure. Understanding the value of ud.localpath at the time of failure would be helpful to debug it further. Tweak the debugging to provide this in the output. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-27fetch/git: Fix local clone url to make it work with repoRobert Yang
The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23crate.py: make checksum verification mandatoryFrederic Martinsons
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-23fetch2: Add checksum capability for crate fetcherFrederic Martinsons
This change brings checksum verification of each crate in a recipe, e.g | SRC_URI += " \ | crate://crates.io/aho-corasick/0.7.20 \ | crate://crates.io/atomic-waker/1.1.0 \ | crate://crates.io/cc/1.0.79 \ | " | | SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" | SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599" | SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f" That will require to move the checksum initialization after the possible call to urldata_init method in order for the crate fetcher to parse the url. Another way of doing could have been implementing a decodeurl method that would have been specific for each fetcher class. Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-23fetch2/sftp: Fix fetching URIs with spacesDaniel Ammann
Spaces can be used in URIs with %20, but the URI needs to be quoted. Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-23ConfHandler: Allow the '@' character in variable flag namesJames R T
This patch enables the usage of the '@' character in variable flag names. One use case of variable flags is to assign the network namespaces of some systemd services/targets to configure other parts of the build process of some system. The filenames of systemd services/targets might contain the '@' character if they are template unit files that can take in a single parameter/argument and be instanced multiple times, as indicated by systemd's official manual page. The '@' character is disallowed as the first character in a variable flag name. Imposing more restrictions on the first character is a compromise to make parsing easier and to allow for more options in the future to extend the syntax. This patch is successfully verified by creating a custom BitBake recipe that sets and unsets the value of a variable flag with the '@' character in its name and ensuring that no ParseError is being thrown. Regression tests have also been added to `bb.parse`. `bin/bitbake-selftest` has also been successfully executed and all tests passed. Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-23bitbake-user-manual: fix links to supported release manualsMichael Opdenacker
Links to manuals for supported releases were all broken. Only the pre-sphinx ones are still valid. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-21fetch2: Add autorev warning when it is set too lateRichard Purdie
Bitbake expects a consistent metadata environment when it parses. There are plenty of ways you can set a recipe to autorev at a point where parts of the fetcher have already been triggered leading to obsure bugs which I struggled to debug, let alone anyone not familar with the code. If anyone is running into the message from the commit, the issue is likely one of timing. Keep in mind that the anonymous python code in base.bbclass will expand variables like FILESPATH, WORKDIR and others which contain PV. The recipe needs to be set to AUTOREV before that anonymous python runs. In particular, that means you can't set SRCREV = "${AUTOREV}" in other anonymous python, it needs to happen earlier. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-21fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEENRichard Purdie
For readability of following patches, rename this internal variable to allow for others in a similar format. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17toaster: update fixtures for mickledoreTim Orling
After updating gen_fixtures.py for 'mickledore' 4.2, run './gen_fixutures.py --all' to update oe-core.xml and poky.xml. Drop langdale as support is ending soon. Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17toaster: update gen_fixtures.py for mickledoreTim Orling
Update for 'mickledore' 4.2.0 release and bitbake 2.4 branching. Drop 'langdale' as support will be ending soon. Update stable branches to latest patch release. Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17build: Make python output print to stdout when running with -v (verbose)Mark Asselstine
When tasks are run with -v (verbose) on the bitbake commandline, shell tasks print their stdout, python tasks do not. This change redirects the python task's print output to an in memory buffer. After the task is executed the output is printed to stdout via the logger. This makes the python task behavior match the shell task behavior when running with -v. The contents of the task's log files remain unchanged after this change. This approach should keep the correct order in most cases, however, if the python task accesses the logger directly, that content will appear before other output. On the other hand, this change should negate the need for python tasks to access the logger directly. Special care is taken to save/restore the existing stdout and stderr and preventing sending output directly to the logger when there are "recursive" calls, for instance when a python function calls a shell function, avoiding printing things potentially out of order and/or multiple times. The logging-test.bb in meta-selftest can be used to review this change. This has been tested with the full bblogging oeqa tests. [Yocto #14544] Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17crate.py: authorize crate url with parametersFrederic Martinsons
This allow to have classic fetch parameters (like destsuffix, sha256, name ...) not being considered by crate fetcher itself (and so mess up its download) Moreover, it allow to overload the name of the downloaded crate (maybe usefull if there is a naming clash between two crates coming from different repositories) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-11wget.py: Combine urlopener exceptionsMark Hatle
No reason to have three identical exception handles, refactor to catch any of the exceptions with the same block of code. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-11wget.py: Add catch TimeoutError exceptionMark Hatle
We've observed TimeoutError exceptions during the sstate-cache mirror fetch, it appears that due to the number of (invalid) files requested the remote side is eventually dropping the connection (not closing it) which can result in a TimeoutError exception being sent, while rate it is different from the urllib.error.URLError or ConnectionResetError. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-03-07bitbake-user-manual: update Hello World exampleFawzi KHABER
Fix broken link "Obtaining bitbake". Update documentation for the bitbake hello world example, the output was outdated. Fix LAYERSERIES_COMPAT warning by adding dunfell as default compatible release. Add proper formating for base.bbclass command. Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reviewed-by: Ever ATILANO <ever.atilano@smile.fr> Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr> Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-07doc: ref-variables: add LAYERSERIES_COMPAT to term glossaryFawzi KHABER
Bitbake throws a warning if the layer compatibility is not defined since cca81e33b58c390dcf5cc3a31555a43b79177166. The description of this variable comes from the Yocto Project manual. Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reviewed-by: Ever ATILANO <ever.atilano@smile.fr> Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr> Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-07server/xmlrpc: Fix after currentAsyncCommand locking changesRichard Purdie
After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3, "process/cooker/command: Fix currentAsyncCommand locking/races", command.py assumes it has access to the process server but the xmlrpc backend was passing in the xmlrpc server object leading to errors like: xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'"> Fixing to pass the process server to command.py resolves this issue. [YOCTO #15008] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-25documentation: bitbake: add file-checksums to varflags sectionRichard Elberger
Fixes [YOCTO #11605] by: - Adding definition of file-checksums to Variable Flags section. - Describe data to add to list which adds external file dependencies. - Write example on usage to prepend a value to file-checksums list. Signed-off-by: Richard Elberger <rich@richelberger.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-25fetch/npmsw: add more short forms for git operationsMark Asselstine
>From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-25gitsm: Fix path construction for relative submodule URIMike Crowe
The submodule repository URI contains a path to something not necessarily on the local filesystem. This means that we can't use realpath to normalise it without risking getting bad results if the path happens to match something on the local filesystem. This situation can cause very confusing errors if that matching local path happens to be a symlink to somewhere else. Using normpath rather than realpath means that the path simplification follows simple rules on the string rather than looking at the local filesystem and avoids problems. Signed-off-by: Mike Crowe <mac@mcrowe.com> Co-authored-by: Dave Craig Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24utils: Use internal fetcher function to avoid duplicationRichard Purdie
We currently have two lists of "proxy" or "fetcher" environment exports. Make the one in utils match the one on the fetcher which has a more complete list of variables now. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24fetch2/wget: Drop unused importRichard Purdie
This import is no longer used anywhere so can be removed. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARSCaner Altinbasak
These environment variables are needed by gclient and needed to be passed into fetcher. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24cooker: Fix memory resident cache invalidation issueRichard Purdie
We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24lib/bb/siggen: fix debug() callRoss Burton
Bitbake f68682 changed the logger's debug() method to be compatible with logging.debug(), but this caller was still using the old API where you can pass an integer as the first argument: WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]') Instead, call bbdebug() which has the priority argument. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23bitbake: Bump to version 2.3.1Richard Purdie
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our verison number to a development series version. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie
Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22fetch/git: Replace mkdtemp with TemporaryDirectory and avoid exception maskingPaulo Neves
Due to using mkdtemp instead of TemporaryDirectory we needed to manually cleanup the directory in a try finally block. With tempfile.TemporaryDirectory we can handle the cleanup with a "with" statement and not need to manually clean up oursevels. Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22fetch/git: Removed unused variables in _contains_lfsPaulo Neves
branchname was set but never used in the context of _contains_lfs method. Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22tests/fetch: Add real git lfs tests and decoratorPaulo Neves
Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22tests/fetch: git-lfs restore _find_git_lfsPaulo Neves
Not restoring the mocked _find_git_lfs leads to other tests failing. Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20server/process: Improve idle thread exception handlingRichard Purdie
If the inotifier code has an exception, bitbake currently hangs. Catch any exception and exit if seen. Also check the idle thread is alive and exit if it disappears. This should stop bitbake hanging if such a situation arises in future such as this example: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20cooker: Ensure lock is held with changing notifierRichard Purdie
We've seen a couple of cases which bitbake hangs due to an inotifer exception such as: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Ensure we don't destory the inotifier when the idle thread is reading is by holding the lock during setup/teardown. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>