| Age | Commit message (Collapse) | Author |
|---|
|
Add logic to check that if REQUIRED_VERSION has been set that the cooker
class method findBestProviders properly handles the case where the
REQUIRED_VERSION has not been found.
Fixes [YOCTO #10096]
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit adds a new column to the Bitbake --show-versions command
called Required Version. This column will display any packages which
have a REQUIRED_VERSION successfully set.
Fixes [YOCTO #10096]
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.
In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.
Thus we force the shutdown after catching an exception.
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
From tinfoil, if you edit bblayers.conf and break it, then call
parseConfiguration (e.g. by adding a bad layer with bitbake-layers),
the system doens't show any parse error yet it should.
Add in a call to the updateCache function so that things really
are reparsed when requested.
Partially fixes [YOCTO #14054]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.
Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Recent changes mean data might not be setup. If its not, avoid tracebacks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Blocks SIGINT in the worker processes to prevent them from running the
parent process signal handler, which causes them to deadlock under
certain circumstances.
[YOCTO #14034]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The original code saved BBFILES back to BBFILES without any changes which isn't
usefule, so remove that line. Now save prioritized BBFILES to
BBFILES_PRIORITIZED which can accelerate the query a lot for the one which
relies on it such as bb.utils.get_file_layer().
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Otherwise this can interfer with multiprocessing exit handling.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you make parsing fail (e.g. add something like:
X := "${@d.getVar('MCMACHINES').split()[1]}"
to meson.bbclass, then run "while true; do bitbake -g bash; done"
it will eventually hang. It appears the cancel_join_thread() call the
parsing failure triggers, breaks the results_queue badly enough that it
sits in read() indefintely (called from self.result_queue.get(timeout=0.25)).
The timeout only applies to lock aquisition, not the read call.
I've tried various other approaches such as using cancel_join_thread()
in other places but the only way things don't lock up is to avoid
cancel_join_thread() entirely for results_queue.
I do have a concern that this may adversely affect Ctrl+C handling
but equally, its broken now already and this appears to improve
things.
[YOCTO #14034]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
During cooker shutdown, its possible the parser isn't cleaned up. Fix
this (which may partially explain why threads were left hanging around
at exit).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This could potentialy account for some of the missing thread cleanup
we're seeing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Calling sys.exit() in the middle of the code is rather antisocial. We catch
this in various places but we shouldn't have to. In all these cases we have
already sent events explaining to the user what happened. This means the
correct exception is BBHandledException.
The recent startup changes have moved the point a lot of this code gets
called to inside the UI, with memres it would have always been possible
from there anyway. This change makes things much more consistent.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Only the env variables which were added to the datastore were being
updated. We need to update the whole copy, we just only trigger a
reparse if any of the variables making it into the datastore change.
This avoids a bug where variables such as DISPLAY in a new UI context
would break under memory resident bitbake.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changes to configuration.env were not updating BB_ORIGENV, fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.
The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the eventlog is not handled correctly for memory resident
bitbake. Fix this by allowing adpations to configuration changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than passing debug/verbose/debug_domains around, pass the
computed output of these. Ensure that the cooker sets the levels
to match the levels currently set in the UI and generally try and
make it easier to understand what the code is doing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Hongxu Jia reported a problem where the bb_cache files were not always being
written out correctly. This was due to the sync thread being terminated
prematurely.
Whilst the preceeding changes mean the exit handler for this thread is now
correctly called since we switch to using sys.exit() instead of os._exit(),
this write can happen after we drop the bitbake lock, leading to potential
races. Avoid that headache by adding in explicit thread join() calls before
we drop the lock (which atexit or Finalize can't do).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Each run through the parser would leak a thread from the queue created to
shut the parser down. Close this down correctly and clean up the code flow
slightly whilst in the area, making sure this thread does shut down correctly
(we don't care if it loses data).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"bitbake mc:arm:bash mc:arm:busybox"
works but
"bitbake multiconfig:arm:bash multiconfig:arm:busybox"
does not. The reason is the list is modified whilst iterating.
Don't do that.
[YOCTO #13607]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids a traceback if an invalid multiconfig is referenced in the bitbake
commandline and tweaks the message to make it more understanable.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.
The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.
Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.
The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The cooker had a multiconfig parameter for the findProviders() and
findBestProviders() API, but it was being ignored.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If for example a tinfoil connection edits the datastore, a subsequent
connection can be "corrupted" by those changes. By setting the parse
status of the caches as False at exit, the behaviour becomes the same
as a newly setup server as a new data store is setup.
This avoids problems in tests when BB_SERVER_TIMEOUT is set as the
server is properly reset between connections.
[YOCTO #13812]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When parsing, reset the loghandler when finished, else the messages
can be misleading.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.
Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).
This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Removed the deprecated methods as it will only cause problems later on,
and since warn() just calls warning(), it shouldn't change anything
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This should have been removed together with expanded_data in commit
e3694e73 (cooker/command: Drop expanded_data).
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The hash server process is terminated and waited on with join(), so it
should not be a daemon. Daemonizing it cause races with the server
cleanup, especially in the selftest because the process may not have
terminated and cleanup up its socket before the test cleanup runs and
tries to do it.
[YOCTO #13542]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82.
This ignores the layer priority, making the issue much worse.
E.g. I'm seeing a lot of failures caused by missing users, because
base-passwd bbappends applied in unexpected order caused different
passwd.master to be found in re-ordered FILESPATH.
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order. Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.
Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The information of recipe-depends.dot is misleading.
e.g.
$ grep xz recipe-depends.dot | grep bzip2
"bzip2" -> "xz"
"xz" -> "bzip2"
Users would wonder why they get some circular dependency.
The information is derived from removing the task names
of task-depends.dot. It's not giving people any additonal
information, and it's misleading.
So we remove the generation of this file.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At exit the hashserv code was causing tracebacks as join() wasn't
being called from the thread that started the process. Ensure that
the hashserver is started from the pre_serve hook which is the
final thread the cooker runs in. This avoids the traceback at the
expense of some horrific poking into data stores which will ultimately
need improving through a proper API.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press *one* Ctrl-C when the parsing process is at about 50%, then the processes
are not exited:
Keyboard Interrupt, closing down...
Timeout while waiting for a reply from the bitbake server
It hangs at process.join(), according to:
https://docs.python.org/3.7/library/multiprocessing.html
Cleanup the queue before call process.join() can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There were hard to debug lockups when trying to use threading to start
hashserv as a thread. Switch to multiprocessing which doesn't show the
same locking problems.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Charlie Davies
Jan Brzezanski
Joshua Watt
Richard Purdie
Tomasz Dziendzielski
Peter Kjellerstedt
Robert Yang
Frazer Clews
Martin Jansa
Wes Lindauer
Chen Qi