aboutsummaryrefslogtreecommitdiffstats
path: root/lib
AgeCommit message (Collapse)Author
2 daysasyncrpc: Check websockets versionmaster-nextjoshua Watt
Checks that the minimum version of the websockets module is present, and if not raises an ImportError. This allows the user to get earlier feedback if using websockets is going to succeed Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 dayscooker: Handle ImportError for websocketsjoshua Watt
Handles ImportError when creating a hash equivalence to ping the server. This notifies user earlier with a more precise error if websockets can't be used, and also prevents passing a known bad upstream value to the local server Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 dayssiggen/runqueue: Report which dependencies affect the taskhashjoshua Watt
Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: add bitbake selftestsMichael Opdenacker
Run them with "bitbake-selftest prserv.tests" Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: import simplificationMichael Opdenacker
Simplify the importone() hook: - to make it independent from the "history" mode which is client specific. - remove the "history" parameter - we want all values to be imported for binary reproducibility purposes. - using the store_value() function (which warrants you don't save the same value twice and doesn't write when you're using a read-only server) is enough. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: store_value() improvementsMichael Opdenacker
Add a test_checksum_value() to test whether a (version, pkgarch, checksum, value) entry already exists in the database. This is used to protect the store_value() function from an error when trying to store a duplicate entry in the database. Also check whether the current database is open in read-only mode. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: avoid possible race condition in database codeMichael Opdenacker
Remove a possible race condition by allowing a read-only server to create the PR table anyway. This avoids a failure if both a read-only and read-write server try to access an empty database at the same time. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <jpewhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: enable database sharingMichael Opdenacker
sqlite3 can allow multiple processes to access the database simultaneously, but it must be opened correctly. The key change is that the database is no longer opened in "exclusive" mode (defaulting to shared mode). In addition, the journal is set to "WAL" mode, as this is the most efficient for dealing with simultaneous access between different processes. In order to keep the database performance, synchronous mode is set to "off". The WAL journal will protect against incomplete transactions in any given client, however the database will not be protected against unexpected power loss from the OS (which is a fine trade off for performance, and also the same as the previous implementation). The use of a database cursor enabled to remove the _execute() wrapper. The cursor automatically makes sure that the query happens in an atomic transaction and commits when finished. This also removes the need for a "dirty" flag for the database and for explicit database syncing, which simplifies the code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: add "upstream" server supportMichael Opdenacker
Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one. This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y"). Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format." Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision. This is done by going on supporting the "history" mode that wasn't used so far (revisions can return to a previous historical value), in addition to the default "no history" mode (revisions can never decrease). Rather than storing the history mode in the database table itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode is now passed through the client requests. As a consequence, the table name is now "PRMAIN", which is incompatible with what was generated before, but avoids confusion if we kept the "PRMAIN_nohist" name for both "history" and "no history" modes. Update the server version to "2.0.0". Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: move code from __init__ to bitbake-prservMichael Opdenacker
This script was the only user of this code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysprserv: declare "max_package_pr" client hookMichael Opdenacker
Add missing declaration for the max_package_pr client hook Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysfetch2/svn: Fix mirroring issue with svnKari Sivonen
Add return false to supports_checksum for svn fetcher which fhis fixes MIRROR usage for svn uris. Also add a testcase. [YOCTO #15473] Signed-off-by: Kari Sivonen <kari.sivonen@live.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysbuild: Handle conflict between cwd and cleandirsRichard Purdie
If the cwd of the task is also a cleandirs, you would see warnings from bitbake about being unable to obtain cwd during the task execution. Tweak the code to detect this and avoid the warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysfetch2/crate: add upstream latest version check functionAlexander Kanavin
This is actually rather easy: crate web API provides a json with all the versions, for example: https://crates.io/api/v1/crates/cargo-c/versions Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 daysbitbake-layers: adapt force option to not use tinfoilSimone Weiß
Fixes [YOCTO #15417] When a layer adds a new dependency after it was added to a conf, it can not be removed w/o this dependency in the setup. Even the dependent layer can not be added, as the tinfoil setup will fail. Adapt --force to not perform the tinfoil at all, the use will be at own risk, i.e. the added layers might not parse properly afterwards. This is not merged into the force option with -F as it even changes the loading of plugins from other layers and is hence even more invasive as force. Instead force can now be speciefied multiple times and is counted. Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2 dayscooker: Use hash client to ping upstream serverHEADmasterjoshua Watt
The cooker attempts to connect to the upstream hash equivalent server to warn the user early if it is misconfigured. However, this was making the assumption that it was a raw TCP connection and failed when attempting to use a websocket upstream server. Fix this by creating an hash client and using the ping API to check the server instead of using a raw socket. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-20fetch2/gcp: Add missing runfetchcmd importSven Schwermer
This adds the missing import. This bug was introduced with 1ab1d36c. Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16bitbake: Bump to version 2.9.0 development version postreleaseRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16bitbake: Bump to version 2.8.0yocto-5.02024-04-scarthgap2.8.0Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16hashserv: client: Fix mode state errorsJoshua Watt
Careful reading of the code can contrive cases where poorly timed ConnectionError's will result in the client mode being incorrectly reset to MODE_NORMAL when it should actual be a stream mode for the current command. Fix this by no longer attempting to restore the mode when the connection is setup. Instead, attempt to set the stream mode inside the send wrapper for the stream data, which means that it should always end up in the correct mode before continuing. Also, factor out the transition to normal mode into a invoke() override so it doesn't need to be specified over and over again. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16siggen: Capture SSL environment for hashserverJoshua Watt
Now that the bitbake hash server supports SSL connections, we need to capture a few environment variables which can affect the ability to connect via SSL. Note that the variables are only put in place to affect the environment while actually invoking the server [RP: Tweak to use BB_ORIGENV as well] [RP: Tweak to handle os.environ restore correctly] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14BBHandler: Handle unclosed functions correctlyRichard Purdie
A function accidentally defined as: somefunction() { : } which is unclosed due to the space at the end, would currently silently cause breakage. Have the parser throw and error for this. [YOCTO #15470] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: remove unnecessary codeMichael Opdenacker
In db.py, the ifnull() statement guarantees that the SQL request will return a value. It's therefore unnecessary to test the case when no value is found. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: correct error messageMichael Opdenacker
according to db.py, prserv.NotFoundError is returned here when adding a new value to the database failed Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: remove redundant exception handlerMichael Opdenacker
This exception handler is already present in db.py's get_value() code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: add extra requestsMichael Opdenacker
Useful for connecting a PR server to an upstream one - "test-package" checks whether the specified package version and arch is known in the database. - "test-pr" checks a specified output hash is found in the database. Otherwise it returns 'None' instead of a new value. - "max-package-pr" returns the highest PR number for (version, arch) entries in the database, and None if not found Add new DB functions supporting the above, plus test_value() which tells whether a given value is available for the specified package and architecture. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: capitalization and spacing improvementsMichael Opdenacker
Choosing only one style of capitalization Add extra space after some commas too Remove idle spaces Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14asyncrpc: include parse_address from hashservMichael Opdenacker
Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: use self.logger instead of logger directlyMichael Opdenacker
In both the PRServerClient and PRClient objects. This aligns with what is done in hashserv/server.py and makes it possible to benefit from possible specializations of the logger in the corresponding super classes, instead of using always the global logger. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: use double quotes by defaultMichael Opdenacker
To aligh with the hashserv code Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: simplify the PRServerClient() interfaceMichael Opdenacker
serv.py: simplify the PRServerClient() interface by passing the server object instead of multiple arguments, and then retrieving the data through this object. This replicates what is done for ServerClient() in hashserv/server.py Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-11wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley
Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-23siggen: Add support for hashserve credentialsJoshua Watt
Adds support for hashserver credentials to be specified in the SignatureGenerator Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21fetch2/git: Install Git LFS in local repository configDerek Erdmann
Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21utils: better estimate number of available cpusFelix Moessbauer
When running in a cgroup which is limited to a subset of cpus (via cpuset.cpus), cpu_count() should return the number of cpus that can be used instead of the number of cpus the system has. This also aligns the semantics with nproc. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21fetch2: Fix misleading "no output" msgViswanath Kraleti
When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21fetch2: handle URIs with single-valued query parametersRoss Burton
Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21lib/bb: support NO_COLORPeter A. Bigot
Red text on a black background can make it difficult for people with visual impairments to read the text of error messages. Respect the presence of a non-empty NO_COLOR environment variable as an indication the user doesn't want colorization to be automatically enabled. See: https://no-color.org/ Signed-off-by: Peter A. Bigot <pab@pabigot.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21bitbake: fetch2/git: Escape parentheses in git src namePhilippe Rivest
FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest <privest@genetec.com> Signed-off-by: Philippe Rivest <technophil98@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21bitbake-worker: allow '=' in environment variable valuesdavid d zuhn
Limit the split to key & value (2 items) instead of the n items one can get if there are '=' characters in the value. Fixes [YOCTO #15447] Signed-off-by: david d zuhn <david.zuhn@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-21bitbake: improve descriptions of '-S printdiff'Alexander Kanavin
Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-05utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()Michael Opdenacker
preserved_envvars() is used when the BB_ENV_PASSTHROUGH environment variable is not set. Therefore, its code shouldn't return this variable. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reported-by: Robert P. J. Day <rpjday@crashcourse.ca> Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-01prserv/serv: Fix a PID file removal race on prserv stopYoann Congal
A race condition has happened where the exiting server removed the PID file between the existence check and the removal, resulting in a FileNotFoundError exception. The fix is to ignore the FileNotFoundError exception, the existence check is now redundant so remove it to simplify. Fixes [YOCTO #14341] Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-28fetch/git: Avoid clean upon failureRichard Purdie
Currently when git fetches fail, it destroys all the existing local clone data. For large repositories this can introduce long build delays when for example, you just typo'd the git revision hash. The git fetcher should be able to recover most directories so when the fetch is for a git repo, avoid removing things unless clean is explicitly called (e.g. a -c cleanall task). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-28taskexp_ncurses: fix execution example in introductory commentUlrich Ölmann
Drop the ".py" file extension from the "-u" option's argument that has been overlooked while applying the original patch (see [1]) to make the example work. While at it sort the recipes' names consistently with respect to what is found in the self-test examples below. [1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/ Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24asyncrpc: Add support for server headersJoshua Watt
Adds support for asyncrpc servers to send connection headers to clients on connection. Since this is a breaking protocol change, clients must opt-in to expect headers from the server, corresponding to a version bump in the client protocol. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24layerindexlib: fix missing layer branch backtraceTim Orling
When a LayerBranch (a specific layer at a specific release) does not exist in the layerindex database ("Layerindex Metadata"), the dependency would throw a backtrace. Instead fail early and provide an error message. Since layerindexlib will also check the local layers, inform the user that a local checkout might resolve the situation. Recommend that they reach out to the layer maintainers and layer index admins to properly fix it for everyone. While we are here, remove some trailing whitespace. [YOCTO #15365] [YOCTO #13954] Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23fetch2: Ensure that git LFS objects are availablePhilip Lorenz
The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23bitbake: progressbar: accept value over initial maxvalEnguerrand de Ribaucourt
There is a very rare case where the maxval is improperly computed initially for cache loading progress, and the value will go over. Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress() # we might have calculated incorrect total size because a file # might've been written out just after we checked its size In that case, progressbar will receive a value over the initial maxval. This results in a ValueError stack trace as well as bitbake returning 1. Traceback (most recent call last): File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main cacheprogress.update(event.current) File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update raise ValueError('Value out of range') ValueError: Value out of range This fix mirrors the behavior of MulticonfigCache and accepts the new value as the new maxval. This is also what the percentage printout is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage() I encountered this issue randomly while working on a project with VSCode saving files while commands where fired. Note: This file is a fork from python-progressbar. It hasn't been refreshed in 8 years. We did only two commits, 5 years ago with minor modifications. This new change is also not how the upstream project is behaving. Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23runqueue: Add support for BB_LOADFACTOR_MAXRichard Purdie
Some ditros don't enable /proc/pressure and it tends to be those which we see bitbake timeout issues on, seemingly as load gets too high and the bitbake processes don't get scheduled in for minutes at a time. Add support for stopping running extra tasks if the system load average goes above a certain threshold by setting BB_LOADFACTOR_MAX. The value used is scaled by CPU number, so a value of 1 would be when the load average equals the number of cpu cores of the system, under one only starts tasks when the load average is below the number of cores. This means you can centrally set a value such as 1.5 which will then scale correctly to different sized machines with differing numbers of CPUs. The pressure regulation is probably more accurate and responsive, however our graphs do show singificant load spikes on some workers and this patch is aimed at trying to avoid those. Pressure regulation is used where available in preference to this load factor regulation when both are set. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>