| Age | Commit message (Collapse) | Author |
|---|
|
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When a single signature is locked, dependent task checksum calculations
fail. This in turn is because get_unihash cannot be cached correctly
by this function. Remove that has turned out to be a poor optimisation
to avoid that bug.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We try to write sstate with group-write permissions so that sstate-cache
can be shared between multiple users. However the siginfo files are
created in various tasks which may set their own umask (such as
do_populate_sysroot, 0022). This results in no group write permission
on the intermediate directories, which is fatal when sharing a cache.
Fix this by wrapping the siginfo mkdir in a umask change to 0002.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This change improves performance by reducing runtime about 33% for typical
inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't
mention 'virtual' anymore.
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Before this fix, this example basepath (a):
virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile
would get incorrectly "cleaned" into:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb
When searching backwards in `a` trying to isolate the 'virtual:xyz' to add
it to the end of the string, we need to consider `a` still has the recipe
path and taskname. So stoping the rsplit after only 1 split is not enough.
We want to reach the second ':' from the end.
This way, we obtain:
helloworld/helloworld_1.2.3.bb:do_compile:virtual:native
reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Passing all the data caches to the task hashing functions allows them to
correctly account for mcdepends in task signatures. This allows tasks to
be correctly re-run when a mcdepends changes.
By default, the legacy behavior is maintained for derived signature
generators by passing a special proxy object that can either be used to
access all multiconfigs or the legacy behavior. If a derived signature
generator is updated, it can set the supports_multiconfig_datacaces
property to instruct bitbake it deals with multiconfigs properly.
[YOCTO #13724]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a variable is in the signature whitelist, we'd currently expand
it, then later ignore the data. This is problemtic for code which
has effects when expanded, recently source date epoch in OE-Core
for example.
We don't actually need to do this, if we pass the whitelist into
the earlier function it can avoid the expansion. This also also
give a small performance boost since we avoid running code in some
cases.
[YOCTO #13581]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.
Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).
This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If locked sigs are in use this function makes little sense, need to
avoid generating mismatch warnings.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add unihash cache of values to speed up cache lookup.
This avoids the overhead of the disk based check functions.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the cache can grow huge since any previously used hash is
retained in the cache. This change moves to use one hash per task
which improves the speed of the functions considerably. Currently
performance is an issue, as are very large cache files and cache
load time.
By moving to a single hash per task, the shorted filename as a key
is no longer usable as the same recipe has multiple variants for
the same filename so this has to change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Hack the hashserv to allow extra data to be injected into the hashserv
method. This allows OE-Core to handle cases where there are multiple
sstate objects for the same taskhash, e.g. native/cross objects based
upon BUILD_ARCH or the host distro (when uninative isn't used).
This has been tested and proven to be very effective. We will likely
rework the code to improve how this is handled but for now this
improves automated builds until we can get to that refactoring and
more invasive changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a significant performance issue in get_unihash(). The issue turns out
to be the lookups of setscene tasks. We can fix this by using a set() instead of
the current list.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit c4b8440f730c33eaf9f818b856ae81b2f1017fec.
The logic in this change is flawed and needs to be re-thought.
|
|
We're seeing huge slowdowns on large builds on the autobuilder. A qemux86 world
build was running really slowly, a primary feature was lots of rehashing going on
due to an sstate change which caused a rebuild when all output should be identical.
This was traced to the hundreds of thousands of calls to get_unihash() from
get_taskash(). If we simplify the unitaskhashes data structure, we can bypass
the function call and access the data directly. In local profile charts,
this significanly sped up process_possible_migrations(), both on profiles
and visually.
Whilst this change doesn't aid readability, it does solve an otherwise
huge performance issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its possible the new unihash may not exist in sstate. Currently the code
would create an sstate object with the old hash however this updates it to
create the object with the new unihash.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
rehash
An example:
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started
ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded
Is caused by a rehash in a dependency happening somewhere earlier in the build
and the taint being reset.
Change the code so that nostamp taints are preserved to avoid the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a hash is reported to the hash server, the stamp written out by the
current task didn't account for any new hash the server may have provided.
Fix this so the correct stamp is written. This means "bitbake X; bitbake X"
no longer rebuilds lots of things when hashequiv is active.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The full paths make the cache useless in the sdk. They also bloat the
cache size. They're for human debugging benefit only so compromise and
reduce this to the filename.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the unihash in the output filename of sigdata files else the contents
of stamp directories is misleading. Write the unihash into the singature to
make it clear what happened.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.
Avoid reporting hashes for non setscene tasks since we'd never
query that.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The setscenetasks list needs to be available in the worker contexts
else the signature behaviour there mismatches what the server does.
Add the data to get/set_taskdata to ensure this happens.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The HashConnectionError class was moved to the client module and needs
to be updated.
[YOCTO #13537]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a test which tests the runqueue adaptations for hash equivalency.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than metadata driven sqlite databases for communication, use
bitbake's unitaskhashes variable instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The addition of some debugging code meant that comparisions between sig
files with a taint and without a taint weren't working. Tweak the logic
to avoid tracebacks if one side doesn't have a taint.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that runqueue optimises based on task hash, we need to ensure
tasks have unique hashes even in the simplest siggen mode. Use
the task name to calculate a unique hash.
This fixes runqueue tests when hash optimisations are added.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After the unihash changes the default signature handler didn't work. Tweak it
to adapt to those changes (allowing the runqueue tests to work).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.
mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.
[YOCTO #11168]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is useful for debugging.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This can make people easier to understand bitbake-diffsigs/dumpsig's output,
otherwise, it's hard to know it is a random uuid unless look into the code.
E.g.:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*
* Before:
Taint (by forced/invalidated task) changed from nostamp:fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp:28192187-5021-40c1-9e21-45483b62c910
* Now:
Taint (by forced/invalidated task) changed from nostamp(uuid4):fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp(uuid4):28192187-5021-40c1-9e21-45483b62c910
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There was already a fix to ignore some multiconfig dependencies but its
'opposite' case wasn't covered. Cover that combination to so as to avoid
tracebacks in multiconfig builds.
[YOCTO #13090]
[YOCTO #13130]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Similarly to the codeparser change, change to sha256 hashes due to worries
over collisions. The main impact of this change is slightly slower parsing
time as well as longer sstate file names.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Abstracts the function to get the unique hash for a task. This hash is
used as in place of the taskhash for the purpose of determine how other
tasks depend on this one. Unless overridden, the taskhash is the same as
the unique hash, preserving the original behavior.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The mechanism used to get the hash for a stamp file is split out so that
it can be overridden by derived classes
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The colors specified for use with bitbake-diffsigs were adapted for a
dark theme, e.g., by setting the background color to black, which made
it look very bad when used with a light theme.
To make it look good both with a dark or a light theme, it is better
to drop the background color. It is also better to leave out the color
altogether for the title and just use bold. Finally, dropping bold for
the red and green texts indicating removed/added values better matches
other colorized diff implementations as, e.g., git diff.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Joshua Watt
Tomasz Dziendzielski
Richard Purdie
Ross Burton
Jean-Francois Dagenais
Aníbal Limón
Robert Yang
Peter Kjellerstedt