| Age | Commit message (Collapse) | Author |
|---|
|
Extend the SignatureGenerator API by adding a new argument to
get_taskhash() for defining the file checksum cache to use. If the
checksum cache argument is not provided, we use the cache in bb.fetch2,
as before. This is a step towards removing checksum cache from
bb.fetch2.
Unfortunately, extending the API is a bit more involved than just adding
a new optional argument because it has a sort of cyclic dependecy: the
API is defined in bitbake, the implementation may be modified elsewhere
(e.g. oe.sstatesig in oe-core layer) by overriding SignatureGenerator's
method(s), which are eventually consumed back at bitbake. In bitbake, we
must be prepared to use the old interface because oe-core may override
our new shiny API with an old one. Thus, we need to inspect the API in
RunQueueData and adjust call arguments accordingly. If the signature
generator instance implements the new API, we use a checksum cache
instance specific to runque. Print a warning if an implementation of the
old API is detected.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
Just utilize the checksum cache in bb.fetch2 as it is not used for
anything else.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
When multiple recipes which both provide something are being built, bitbake
informs us that most likely one of them provides something the other doesn't,
which is usually correct, but unfortunately it's rather painful to figure out
exactly what that is.
This patch dumps two sets of information, one is the provides information for
each recipe, filtered so only common components are removed. The other is a list
of dependees on the recipe, since sometimes this can easily identify why something
is being built.
Its not straightforward for bitbake to obtain the information but since the
warning/error code path isn't the normal one, we can afford to go through some
less than optimal processing to aid debugging.
Also provide the same information even if we're showing a warning since its still
useful.
[YOCTO #8032]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
which needs to run"
This reverts commit b22592af8145a8c7c4ada2fa7c1dee2e753eca46.
That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.
If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.
The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.
Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.
If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.
The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.
After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.
Optimized the code: got rid of extra loop and temporary
list variable virts.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.
Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.
If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
function
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This has always bothered me:
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
This patch changes the messages to be consistent.
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, if you have hard dependencies between setscene tasks (like avahi on
base-passwd through useradd.bbclass), other dependencies may not be installed
even if these exist in sstate. For example, avahi -> expat -> pigz-native
(and avahi -> base-passwd) yet if you cleansstate base-passwd:
bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate
bitbake avahi | tee
you will currently see pigz-native being rebuilt even though it was in
sstate. The fix for this is to continue to iterate dependency chains
around hard blocked dependencies as per this patch.
After this patch is applied, you will see pigz-native installed from sstate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to have an event emitted when all of the sceneQueue tasks
have completed since the metadata can hook this for processing.
Therefore add such an event.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you hit Ctrl+C at the right point, the system processes the request
but merrily continues building. It turns out finish_runqueue() is called
but this doesn't stop the later generation and execution of the
runqueue.
This patch adjusts some of the conditionals to ensure the build really
does stop.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In dry run mode, stamps for noexec tasks are being written out which
is incorrect. Avoid this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When looking at a list of tasks, do_patch and do_unpack were being
given equal priority when one clearly depends on another. The
reason for this was the default task weights of 0 being to tasks.
This is therefore changed to 1 to allow correct weighting of dependencies
which means the scheduler has better information available to it about
tasks.
Weight endpoints differently (10) for clearer debugging of priorities.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The zero priority task should be run first but was being confused with
the None value the priority field defaulted to. Check for None
explicitly to avoid this error.
In the real world this doesn't change much but it confused the debug
output from the schedulers.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The first part of the sstate code checks en-mass whether given checksums
are available. The next part of the code then either triggers those
setscene tasks either running them or skipping them if they've been
covered by others.
The problems was that this second part would always skip a task if it
was unavailable in the first part, even if it would have otherwise been
covered by other tasks.
This mean the mere presence of an artefact (or lack of presence) could
cause a different build failure.
The issue reproduces if you run a build and populate an sstate feed, then
run a second build off that feed, then run a third build off the sstate
feed of the second build (which is reduced in size).
The fix is rather than immediately skipping tasks if the checksum is
unavailable, create a list of missing tasks, then, if that task cannot
be covered by others we can skip it later. The deferral makes the
behaviour the same even when the cache is "incomplete".
[YOCTO #6081]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a setscene task has [depends], its possible they may still get executed out
of order. The issue is that the dependencies are set to set() for all tasks
involved. This patch adds back in explict dependencies within these chains
to avoid the setscene task failures.
[YOCTO #6069]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the commandline options to be processed in the dump signature
code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The runqueue should be using the "realtask" ID to lookup the task
hash, not the "task" ID. This patch resolves corruption issues where
incorrect task hashes were displayed within toaster.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use of waitpid on the worker processes is a bad idea since it conflicts
directly with subprocess internals. Instead use the poll() method
and returncode to determine if the process has exitted, if it has,
we can shut down the system.
This should resolve the hangs once and for all, famous last words.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're running into processes using 100% cpu. It appears theses are locked in
a subprocess.poll() type loop where the process has exited but the code is
looping as its not handling the ECHILD error.
http://bugs.python.org/issue14396
http://bugs.python.org/issue15756
This is likely due to one or both of the above bugs. The question is what actually
grabbed the child exit code as it wasn't this code. Its likely there is therefore
some other code racing and taking that code, it may be some kind of race like:
http://hg.python.org/cpython/rev/767420808a62/
where the fix effectively catches the childs codes in a different part of the system.
We could try and get everyone onto python 2.7.4 where the above bugs are fixed however
for now its safer to admit defeat and go back to polling explictly for our worker exit
codes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Catching all child exit status values is a bad idea. Setting an http sstate mirror
is a great way to view that spectacularly break things. The previous change did
have good code changes so don't revert those parts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are several problems. Firstly, a return value of "None" can mean
there is a C signal handler installed so we need to better handle that
case. signal.SIG_DFL is 0 which equates to false so we also need to
handle that by testing explicitly for None.
Finally, the signal handler *must* call waitpid on all child processes
else it will just get called repeatedly, leading to the hanging behaviour
we've been seeing. The solution is to only error for the worker children,
we warn about any other stray children which we'll have to figure out the
sources of in due course.
Hopefully this patch gets things working again properly though.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Failures on the autobuilder look like this handler is recursing. That
shouldn't be possible but it doesn't hurt to code as such.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've noticed hanging processes which appear to be looping around
waitpid. Its possible multiple calls to teardown are causing problem
or in theory multiple registrations (although the code should not
allow that). Regardless, put better guards around signal handler
registration.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we didn't setup any workers (such as bitbake -S), this would error
since we're trying to set a signal handler to None. This patch
avoids that problem.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sigchld handler was reaping any processes and this was leading to
confusion with any other process handling code that could be active.
This patch:
a) Ensures we only read any process results for the worker processes
we want to monitor
b) Ensures we pass the event to any other sigchld handler if
it isn't an event we're interested in so the functions are properly
chained.
Together this should resolve some of the reports of unknown processes
people have been reporting.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of a significant number of calls to waitpid, register a SIGCHLD
handler instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A run of "bitbake bash -c unpack" when the task has already been
completed resulted in about 9000 calls to logger.debug(). With this
patch which comments out some noisy/less usefull logging and moves
other logging calls outside loops, this number is reduced to 1000
calls. This results in cleaner logs and gives a small but
measurable 0.15s speedup. The log size dropped from 900kb to 160kb.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the worker has already gone missing (e.g. SIGTERM), we should
gracefully handle the write failures at exit time rather than throwing
ugly tracebacks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the worker (or fakeworker) process disappears for some reason, the
system doesn't currently even notice. To fix this, we call waitpid
periodically, looking for exit events of our children. If these
occur, we can gracefully shutdown the server.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
slef.self is clearly meant to be self, fix typo.
Otavio spotted and reported, thanks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* exception like this keeps spinning quite quickly generating GBs of logs
better to kill it asap and show invalid pickle
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A previous commit of mine used the target variable for two different uses
resulting in a lot more sstate being installed than is needed.
Fix the variable to use two different names and unbreak the setscene
behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Markus Lehtonen
Richard Purdie
Ed Bartosh
Alex Franco
Mariano Lopez
Christopher Larson
Gary Thomas
Robert P. J. Day
Martin Jansa