| Age | Commit message (Collapse) | Author |
|---|
|
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
|
|
It is helpful to have timestamps on the ping failures so that they
can be matched against the bitbake logs. It is also useful to understand
how long the server takes for form a reply verses when it is sent.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To learn more about the server timeout issues, be clear in the error
message about which command is showing the timeout. It is currently
unclear if this is the original command or a ping to the server.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.
Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.
This will hopefully resolve various ping timeout issues that were being reported.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
* only as debug output, because otherwise this is printed
very often even when nothing much is happening, e.g.
this is from relatively short build with cca 2000 tasks:
$ grep "Using only" log/cooker/qemux86-64/console-latest.log | sort | uniq -c
78337 NOTE: Using only 1/8 tasks due to exceeded max pressure
577 NOTE: Using only 2/8 tasks due to exceeded max pressure
114 NOTE: Using only 3/8 tasks due to exceeded max pressure
36 NOTE: Using only 4/8 tasks due to exceeded max pressure
81 NOTE: Using only 5/8 tasks due to exceeded max pressure
73 NOTE: Using only 6/8 tasks due to exceeded max pressure
82 NOTE: Using only 7/8 tasks due to exceeded max pressure
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Verify that an existing layer path was given when adding a new
layer.
Manually using the shell for globbing is unnecessary, use the glob
function instead for cleaner code.
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a SECURITY.md file with hints for security researchers and other
parties who might report potential security vulnerabilities.
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
|
|
New variable for 2.4.
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
New directive and variable for 2.4.
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
New variable for 2.4.
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately
when a single ctrl-c is pressed.
Patch restores the previous behavior where a single ctrl-c waits for
active tasks to finish.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Recipes containing both git and npmsw sources in the SRC_URI fail
during fetch from the shrinkwrap. It seems that when the fetcher is
fetching from the shrinkwrap, the SRCREV variable has been deleted but it
till ends up fetching from the git source resulting in an error because
SRCREV is undefined. The root cause of this is that the Fetch class defaults
to urls from the SRC_URI when the urls parameter contains an empty list. This
patch will ensure that Fetch is not instantiated if the urls list is empty.
Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current PSI check calculation does not take into consideration
the possibility of the time interval between last check and current
check being much larger than 1s. In fact, the current behavior does
not match what the manual says about BB_PRESSURE_MAX_XXX, even if
the value is set to upper limit, 1000000, we still get many blocks
on new task launch. The difference between 'total' should be divided
by the time interval if it's larger than 1s.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update regex pattern to allow variable flag name with a single character.
Regression tests have also been updated in `bb.parse` and
`bin/bitbake-selftest -k ParseTest` has been successfully executed.
Eliminate a trailing space as well.
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add bb.event.ParseError to let error-report can catch this kind of error.
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is common for rust packages to depend on different versions of the
same crate. E.g.
| crate://crates.io/windows_x86_64_msvc/0.42.2 \
| crate://crates.io/windows_x86_64_msvc/0.48.0 \
Identification only by the plain crate name makes the sha256sum
ambiguous
| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
and requires lot of manual work to fix the SRC_URI list.
Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of only display the first and stop.
For recipe (crago based) that can contains several artifacts
to fetch with their checksum, it will particularly handy
to display all of missing one.
An example of error message would be
NOTE: Executing Tasks
ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe:
SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe')
ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025
ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1'
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
Set EXCLUDE_FROM_WORLD = "0" in recipe_A:
$ bitbake world -g
Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem:
EXCLUDE_FROM_WORLD = "1": Not build in world
EXCLUDE_FROM_WORLD = "0": Add back to world build
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have an obtuse rare and intermittent bitbake fetch failure. Understanding
the value of ud.localpath at the time of failure would be helpful to debug it
further. Tweak the debugging to provide this in the output.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:
https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:
* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
This change brings checksum verification of each crate
in a recipe, e.g
| SRC_URI += " \
| crate://crates.io/aho-corasick/0.7.20 \
| crate://crates.io/atomic-waker/1.1.0 \
| crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"
That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.
Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Spaces can be used in URIs with %20, but the URI needs to be quoted.
Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
This patch enables the usage of the '@' character in variable flag
names.
One use case of variable flags is to assign the network namespaces of
some systemd services/targets to configure other parts of the build
process of some system. The filenames of systemd services/targets might
contain the '@' character if they are template unit files that can take
in a single parameter/argument and be instanced multiple times, as
indicated by systemd's official manual page.
The '@' character is disallowed as the first character in a variable
flag name. Imposing more restrictions on the first character is a
compromise to make parsing easier and to allow for more options in the
future to extend the syntax.
This patch is successfully verified by creating a custom BitBake recipe
that sets and unsets the value of a variable flag with the '@' character
in its name and ensuring that no ParseError is being thrown. Regression
tests have also been added to `bb.parse`.
`bin/bitbake-selftest` has also been successfully executed and all tests
passed.
Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Links to manuals for supported releases were all broken.
Only the pre-sphinx ones are still valid.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Bitbake expects a consistent metadata environment when it parses. There
are plenty of ways you can set a recipe to autorev at a point where parts
of the fetcher have already been triggered leading to obsure bugs which
I struggled to debug, let alone anyone not familar with the code.
If anyone is running into the message from the commit, the issue is
likely one of timing. Keep in mind that the anonymous python code
in base.bbclass will expand variables like FILESPATH, WORKDIR and others
which contain PV. The recipe needs to be set to AUTOREV before that
anonymous python runs.
In particular, that means you can't set SRCREV = "${AUTOREV}" in other
anonymous python, it needs to happen earlier.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For readability of following patches, rename this internal variable to
allow for others in a similar format.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After updating gen_fixtures.py for 'mickledore' 4.2,
run './gen_fixutures.py --all' to update oe-core.xml
and poky.xml.
Drop langdale as support is ending soon.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update for 'mickledore' 4.2.0 release and bitbake 2.4 branching.
Drop 'langdale' as support will be ending soon.
Update stable branches to latest patch release.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When tasks are run with -v (verbose) on the bitbake commandline, shell
tasks print their stdout, python tasks do not.
This change redirects the python task's print output to an in memory
buffer. After the task is executed the output is printed to stdout via
the logger. This makes the python task behavior match the shell task
behavior when running with -v. The contents of the task's log files
remain unchanged after this change.
This approach should keep the correct order in most cases, however, if
the python task accesses the logger directly, that content will appear
before other output. On the other hand, this change should negate the
need for python tasks to access the logger directly.
Special care is taken to save/restore the existing stdout and stderr
and preventing sending output directly to the logger when there are
"recursive" calls, for instance when a python function calls a shell
function, avoiding printing things potentially out of order and/or
multiple times.
The logging-test.bb in meta-selftest can be used to review this
change. This has been tested with the full bblogging oeqa tests.
[Yocto #14544]
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allow to have classic fetch parameters
(like destsuffix, sha256, name ...) not being
considered by crate fetcher itself (and so mess
up its download)
Moreover, it allow to overload the name of the downloaded
crate (maybe usefull if there is a naming clash between
two crates coming from different repositories)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
No reason to have three identical exception handles, refactor to catch any
of the exceptions with the same block of code.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
We've observed TimeoutError exceptions during the sstate-cache mirror fetch,
it appears that due to the number of (invalid) files requested the remote
side is eventually dropping the connection (not closing it) which can result
in a TimeoutError exception being sent, while rate it is different from the
urllib.error.URLError or ConnectionResetError.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Fix broken link "Obtaining bitbake".
Update documentation for the bitbake hello world example, the output was
outdated.
Fix LAYERSERIES_COMPAT warning by adding dunfell as default compatible release.
Add proper formating for base.bbclass command.
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake throws a warning if the layer compatibility is not defined since
cca81e33b58c390dcf5cc3a31555a43b79177166. The description of this variable
comes from the Yocto Project manual.
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3,
"process/cooker/command: Fix currentAsyncCommand locking/races", command.py
assumes it has access to the process server but the xmlrpc backend was
passing in the xmlrpc server object leading to errors like:
xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'">
Fixing to pass the process server to command.py resolves this issue.
[YOCTO #15008]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #11605] by:
- Adding definition of file-checksums to Variable Flags section.
- Describe data to add to list which adds external file dependencies.
- Write example on usage to prepend a value to file-checksums list.
Signed-off-by: Richard Elberger <rich@richelberger.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
>From the npm-install documentation [1] the CLI provides a set of
short forms when the install fetches from git. These include
"github:"
example: npm install github:mygithubuser/myproject
"gist:"
example: npm install gist:101a11beef
"gitlab:"
example: npm install gitlab:mygitlabuser/myproject
"bitbucket:"
example: npm install bitbucket:mybitbucketuser/myproject
Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in
npm shrinkwrap version] by Stefan Herbrechtsmeier added support for
the "github:" but the others would marked as 'Unsupported dependency'.
The other prefixes are added in this commit, along with extending the
tests to cover some of these.
However, there is one more short form for github which npm-install
allows which forgoes the prefix altogether.
example: npm install mygithubuser/myproject
Unfortunately this format is a bit problematic as it lacks any easily
identifiable 'marker' to match against, and it could be either the
github short form or install from folder format. Experimentation shows
that the folder format requires a leading './' or '/', so we use this
to rule out the ambiguity.
If this approach to folder and github formats disambiguation is
incorrect it won't matter anyways as the folder format is unrecognized
by the code as-is and thus with this change or without, things would
fail.
Since we have to be less strict in the check for git operations we
move it to be the last install format which we check, such that the
less ambiguous formats can be sorted out first.
[1] https://docs.npmjs.com/cli/v9/commands/npm-install
[Yocto #14236]
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The submodule repository URI contains a path to something not
necessarily on the local filesystem. This means that we can't use
realpath to normalise it without risking getting bad results if the path
happens to match something on the local filesystem. This situation can
cause very confusing errors if that matching local path happens to be a
symlink to somewhere else.
Using normpath rather than realpath means that the path simplification
follows simple rules on the string rather than looking at the local
filesystem and avoids problems.
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Co-authored-by: Dave Craig
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This import is no longer used anywhere so can be removed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These environment variables are needed by gclient and needed to be
passed into fetcher.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Joshua Watt
Richard Purdie
Martin Jansa
Chen Qi
Marta Rybczynska
Samantha Jalabert
Paul Eggleton
Enrico Scholz
Svend Meyland Nicolaisen
Kai Kang
Mingli Yu
Frederic Martinsons
Robert Yang
Daniel Ammann
James R T
Michael Opdenacker
Tim Orling
Mark Asselstine
Mark Hatle
Fawzi KHABER
Richard Elberger
Mike Crowe
Caner Altinbasak