| Age | Commit message (Collapse) | Author |
|---|
|
Currently each task has to have a umask specified individually. This
is leading to determinism issues since it is easy to miss specifying
this for an extra task.
Add support for specifing the default task umask globally which
simplifies the problem.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.
This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.
See [YOCTO #13071] for detailed bug information.
Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should not be using sys.argv inside the server to decide which targets
the user added on the commandline. There might not even be a commandline.
Thankfully the targets variable is easily accessible in this context
and contains this exact data we want.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In rare cases the pickled data from a task contains "</event>" which
causes backtrace. This can be reproduced with something like:
do_unpack_prepend () {
bb.warn("</event>")
}
There are several solutions but the easiest is to catch this exception
and look for the next marker instead as this should be the only way such
an unpickle error could occur.
This fixes rare exceptions seen on the autobuilder.
Also add in other potential exceptions listed in the pickle manual
page so that better debug is obtained should there be an error in
this code path in future. exitcode doesn't need the same handling
since we control what is in that data field and it could never contain
</exitcode>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Passing all the data caches to the task hashing functions allows them to
correctly account for mcdepends in task signatures. This allows tasks to
be correctly re-run when a mcdepends changes.
By default, the legacy behavior is maintained for derived signature
generators by passing a special proxy object that can either be used to
access all multiconfigs or the legacy behavior. If a derived signature
generator is updated, it can set the supports_multiconfig_datacaces
property to instruct bitbake it deals with multiconfigs properly.
[YOCTO #13724]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Lowers the level of the log message when setscene tasks have completed.
This message can occur multiple times when hash equivalence is enabled,
since the runqueue switches between executing setscene tasks and normal
tasks. Since this is primarily of use when debugging hash equivalence,
use the hash equivalence logger at VERBOSE level.
[YOCTO #13813]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously --runonly=do_task would give a misleading error like:
ERROR: Could not find any tasks with the tasknames ['do_task'] to run
within the recipes of the taskgraphs of the targets...
The problem is that BitBake tried to find "do_do_task". So teach it to
only add the do_ prefix if it's not already there.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This should give performance improvements to functions using this list of
tasks (sets are used for most of the other code for this reason, not sure
why this wasn't a set in the first place).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A corner case was identified where tasks with valid stamps from previous
builds need to be accounted for in the new sanity test in the migration
code. Add a variable to track such completed tasks to ensure the sanity
test works correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've seen a number of mystery failures where task B would run despite
task A, its dependency not having run. An example would be do_compile
when do_unpack didn't run.
This has been tracked down to this code block. In theory it shouldn't
trigger however it can and has due to bugs elsewhere. When it does, it
causes significant weird failures and possible build corruption.
Change the code to abort the build. This avoids any chance of corruption
and should ensure the issues get reported, putting an end to the weird
build failures.
There may be some cases where this triggers and it shouldn't, we'll work
through those as they arise and are identified.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Regardless of whether we remapped the hash on the server or not, we need
to have bitbake work as if we did as we need to match how the stamp files
look.
This change resolves build failures where tasks were rerunning when they
shouldn't.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This doesn't save much time but does make the profile counts for the
function more accurate which is in itself useful.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We were seeing this running thousands of times with hashequiv, do
the filtering where it makes more sense and make it persist.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the calls to difference_update out a code level which improves efficiency
significantly.
Also further combine the outer loop for efficiency too.
These two changes remove a bottleneck from the performance charts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Creating a new sorted list of sstate tasks each iteration through runqueue is
extremely ineffecient and was compounded by the recent change from a list to set.
Create one sorted list instead of recreating it each time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The "no dependencies" task case was not being correctly considered in this
code and seemed to be the cause of occasionaly task hash mismatch errors
that were being seen as the dependencies were never accounted for properly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The looping over multiple changed hashes causes many calls to get_taskhash
and get_unihash which are potentially slow and then overwritten.
Instead, batch up all the tasks which have changed unihashes and then
do one big loop over the changed tasks rather than each in turn.
This makes worlds of difference to the performance graphs and should speed
up build where many tasks are being rehashed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Batch all updates to scenequeue data together in a single invocation
instead of checking each task serially. This allows the checks for
sstate object to happen in parallel, and also makes sure the log
statement only happens once (per set of rehashes).
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When follow the instructions of multiconfig from Yocto dev manual that
set in core-image-sato recipe:
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
it show too many annoying warnings look like:
| WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
| after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
Treat them as infomations rather than warnings.
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
migrations
Setscene hard dependencies were not being correctly handled during task migration.
For example, do_package of recipe X might become valid due to hashequiv yet we're
still rebuilding pseudo-native, a harddep of do_package. This would mean
it would try to execute that setscene task despite pseudo not being present.
Fix this by ignoring tasks with failed harddependencies. This does mean
stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately
the new runqueue model can likely better handle these cases than the older codebase
could but that is for another more invasive patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The key problem is that currently running setscene tasks are not
accounted for when processing task migrations. This means can allow
two of the same task to execute at the same time with unpredictable
effects.
This change allows us to stop doing that and refactor the code slightly
to make it clearer that these conditions don't arrive even with
deferred tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bibake is currently too 'chatty' when hash equivalence is enabled. Fix
this by only printing the log output if a rehash happens and it matches
an sstate object.
Also, pass a summary option to the hash checking function. This was
already changed to a mechanism which allows addition of new parameters
so this should be backwards and forwards compatible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A minor performance optmisation to keep lists smaller when running large
builds. We can do this since once a task has been built, we don't need
to worry about it. This improves a major bottleneck that shows up on
performance profile charts in dryruns.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are some runqueue code paths where the unihash cache would not be
saved where for example only parsing or an occurred. Save the cache at the
end of runqueue generation to ensure entries are cached.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.
Avoid reporting hashes for non setscene tasks since we'd never
query that.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently runqueue will rerun setscene tasks multiple times as hashes
change. This has caused numerous problems since a setscene task may
become "unavailable" for some future signature combination and the code
then can't easily "unskip" tasks its already passed into the execution
queue.
At least for now, only run setscene once and assume they're equivalent
at that point. In practise that has been much more stable in testing.
Tweak the test to match the change in behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tasks were not migrating consistently, particularly:
* if a task was rehashed which had already run
* if a task which was valid became invalid due to a rehash
We need to always run the migration code for rehashed tasks and then
reprocess them for hash validity. This means rearranging the code.
It also means several tests are no longer correct and can't be written
correctly to work on all possible workflows so those are removed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
According to sstate_checkhashes which is defined in sstate.bbclass, the
currentcoun should be a number (0, not None).
Fixed:
$ bitbake base-files -Sprintdiff
> bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete))
TypeError: %d format: a number is required, not NoneType
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Specifying the force flag will now cause runall and runonly to
invalidate the tasks before running them. This allows a --runall or
--runonly to force the tasks to run, even if they would have otherwise
been skipped, e.g.:
bitbake -f --runall fetch
Will run all do_fetch tasks even if they wouldn't be necessary (for
example, skipped by setscene)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than repeatedly calling mc_from_tid() do this in the parent,
removing around a million function calls. Takes time spent in this
function from 40s to 36s.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are other data structures which can be reprocessed at the same
time as holdoff_tasks, further improving build efficiency in various
places.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't need to process the holdoff task list until we're executing tasks
which saves some data manipulation, at the cost of some data structures
not being correct at all times. This saves significant amounts of time
in various profile charts of larger builds.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've observed do_package and do_package_setscene running in parallel. The
reason is that holdoff_tasks wasn't getting updated. Looking at the code, it
would seem the reason is that the task was in pending_migrations and hence
changed wasn't set and holdoff_tasks wasn't updated.
Fix this. It only affects builds with rehashing enabled.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake <target> --runonly=fetch
failed as the target_tids list included entries which were no longer targeted
task ids. Fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Tomasz Dziendzielski
Joshua Watt
Chris Laplante
Frazer Clews
Kai Kang
Robert Yang