| Age | Commit message (Collapse) | Author |
|---|
|
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
|
|
Adds a subcommand to clean unused outhash entries from the server based
on age
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds a subcommand to invoke the remove API on the server
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than outputting the string "None" for undefined variables, output
only a linefeed (the same as for variables that are defined to the empty
string).
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If an undefined variable or variable flag is specified, bitbake-getvar
will now fail with an error message indicating this.
The error can be supressed with --ignore-undefined, which matches the
previous behavior.
This also changes the errors related to specifying --flag or --unexpand
without --value so that they are sent to stderr rather than stdout.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It does not make any sense to get log output from bitbake-getvar when
the --value option is used as the log output is sent to stdout and thus
interferes with the output of the variable's value.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Initializing Tinfoil with setup_logging = False only has an effect when
recipe parsing is not needed. To make it work regardless of if --recipe
is used, manipulate the quiet parameter to Tinfoil.prepare() instead.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.
Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.
The idea is that this would make layer overrides possible:
OVERRIDES .= ":layer-${FILE_LAYERNAME}"
which then opens possibilities like:
WARN_QA:append:layer-core = " patch-fuzz"
as an example where OE-Core could enable specific QA tests only for that
specific layer.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the export/unexport/network flags are acted on only by presence
or lack of in the data store. This is deliberate due to performance reasons
since a given recipe shell task might export ~80-90 variables and this means
expanding flags for every one of them.
This does catch users unaware since values which expand to nothing don't work
e.g. ${@""} and setting values to "0" wouldn't work either.
The downside to this patch is slow down in parsing speed of around 4%.
The network flag is easier to change and doesn't affect performance, it was
made to operate the same as export/unexport for consitency reasons.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake-getvar does not have a way to silence bitbake
server's logger and that makes the tool hard to use for
text processing. This is especially true when one wants to
get a bitbake value to be piped to some other utility and
instead we get uncontrolled logging messages or warnings
together with bitbake's variable value.
Example without quiet:
bitbake-getvar --value MACHINE
NOTE: Starting bitbake server...
qemux86-64
With quiet:
bitbake-getvar --value MACHINE --quiet
qemux86-64
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
One of the challenges in maintaining the code is that it sometimes uses
a datacaches structure and sometimes a datastore. Rather than continue
the current dual API madness, have the worker contexts create a dummy
datacaches structure with the entries we need. Whilst this does need to
be kept in sync with the real structure, that doesn't change and this
allows the code to be simplified.
With this new approach, we can unify the stamps dependency code again.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When setting up the worker we were transfering large amounts of data
which aren't needed until task execution time.
Defer the fakeroot and taskdeps data until they're needed for a specific
task. This will duplicate some information when executing different tasks
for a given recipe but as is is spread over the build run, it shouldn't
be an issue overall.
Also take the opportunity to clean up the silly length argument lists
that were being passed around at the expense of extra dictionary keys.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous cleanups meant that when the cooker was started, profiling
was always disabled as configuration was sent to the server later and this
was too late to profile the main loop.
Pass the "profile" option over the server commandline so that we can
profile cooker itself again, the setting can now take effect early enough.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Plugins may want to use it (e.g. the layers-setup plugin that would
want to discover writer sub-plugins with it), and so it makes sense
to make tinfoil available a bit eariler.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have some confusion for users since some classes are meant to work
in the configuration space (or "globally") and some are meant to be
selected by recipes individually.
The cleanest way I could find to clarify this is to create "classes-global"
and "classes-recipe" directories which contain the approproate classes and
have bitbake switch scope between them at the appropriate point during
parsing. The existing "classes" directory is always searched as a fallback.
Once a class is moved to a specific directory, it will no longer be found
in the incorrect context. A good example from OE is that
INHERIT += "testimage"
will no longer work but
IMAGE_CLASSES += "testimage"
will, which makes the global scope cleaner by only including it where it
is useful and intended to be used (images).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where copyright headers were not present, add them to make things
clear.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows OE to depend on the unihash copy functionality.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This ignores flake8 rules:
* E402 module level import not at top of file
* E501 line too long
Signed-off-by: Marius Kriegerowski <marius.kriegerowski@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the upcoming LTS, it is time we changed the bitbake version so
move to 2.0.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH
BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
(Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).
Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.
This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We occasionally see bitbake-worker failing and from the logs, an unpickle error
occurs. Add more debug so we can further debug this next time it fails.
[YOCTO #14595]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the build is interrupted, handle the shutdown of pseudo even in this
case to avoid data corruption inside docker containers.
[YOCTO #14555]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have a problem where pseudo server processes exist after bitbake exits
and hold the pseudo database in memory. In a docker container, the processes
will be killed as the container is destroyed with no warning and no opportunity
to write the data to disk. This leads to permissions/inode corruptions and
data loss.
Send a shutdown message to pseudo which in new versions of pseudo will flush
the database, thereby fixing some of the issues people using docker containers
see.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We really do want to see those, as they tend to turn into
hard errors eventually, as what happened with collections
vs collections.abc in python 3.10.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If bitbake-worker fails, return an error code showing that. Also
make the thread cleanup code explict in a finally clause as it would
otherwise hang.
[YOCTO #14393]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #13659]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[updated for asyncrpc changes]
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've talked about having this for long enough. Add a command which queries a single
variable value with history. This saves "bitbake -e | grep" and avoids the
various pitfalls that has.
It also provides a neat example of using tinfoil to make such a query.
Parameters to limit the output to just the value, to limit to a variable flag
and to not expand the output are provided.
[YOCTO #10748]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the previous patch this code is now pointless as we'd have hit a TypeError
before now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Joshua Watt
Peter Kjellerstedt
Etienne Cordonnier
Richard Purdie
Paulo Neves
Frank de Brabander
Alexander Kanavin
Marius Kriegerowski
Scott Murray
Robert Yang
Martin Jansa
Paul Barker
Ross Burton