| Age | Commit message (Collapse) | Author |
|---|
|
Extend the SignatureGenerator API by adding a new argument to
get_taskhash() for defining the file checksum cache to use. If the
checksum cache argument is not provided, we use the cache in bb.fetch2,
as before. This is a step towards removing checksum cache from
bb.fetch2.
Unfortunately, extending the API is a bit more involved than just adding
a new optional argument because it has a sort of cyclic dependecy: the
API is defined in bitbake, the implementation may be modified elsewhere
(e.g. oe.sstatesig in oe-core layer) by overriding SignatureGenerator's
method(s), which are eventually consumed back at bitbake. In bitbake, we
must be prepared to use the old interface because oe-core may override
our new shiny API with an old one. Thus, we need to inspect the API in
RunQueueData and adjust call arguments accordingly. If the signature
generator instance implements the new API, we use a checksum cache
instance specific to runque. Print a warning if an implementation of the
old API is detected.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
Just utilize the checksum cache in bb.fetch2 as it is not used for
anything else.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
sstate.bbclass for example writes siginfo files to a separate location
but we need to read taint data from the standard path.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The taint values need to be passed from the server to the workers to
ensure they see the same stamp values. Also ensure that the "nostamp:"
prefix isn't included in the checksum value to match the server
calculation. This ensures the checksums are all consistent.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In theory all the information to recalcuate the task signatures was written
into the siginfo/sigdata files. In reality, some of the information was
written into the filename.
Firstly this patch duplicates that info into the file itself just for easy
of use since its small.
Secondly, we abstract out the existing "calculate the checksum" code for
the taskhash, and add a function to calculate the bashhash based on the
informaiton within the file.
Finally, we call these functions when we're writing out the data to check
that the data we're writing is consistent. I've found a couple of places
it wasn't and its good to know about these in advance, rather than having
a siginfo/sigdata file which a given hash in its filename but a contents
which give a different result.
This should all combine to avoid a certain class of checksum bugs making
it into world, and identifying problems in advance.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When I enabled debugging of the checksum code, I found the value calculated
from siginfo/sigdata files for do_fetch tasks never matched. This was due
to an error in the way the data was being stored for these, it wasn't ordered
correctly. This patch fixes things so the checksums calculated from
siginfo/sigdata files is correct when file checksums are present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The real method is a few lines later, this one is incorrect and
just causing confusion. Remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This makes no sense as just a note, its at least a warning and useful
to get an idea of which codepath is failing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you run bitbake-diffsigs against two differing sigdata files from
nostamp tasks it shows no difference despite the differing checksum.
Change the code so this shows up as a nostamp 'taint' and at least
makes the issue clearer to the end user.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the number of task dependencies change you currently get
a traceback when using diffsigs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.
[YOCTO #7233]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is useful code to double check the computed checksum value if nothing
else. Might as well have it in tree.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.
This fixes a case where two different checksums could show no output with
bitbake-diffsigs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With a shared work task like gcc, the task can be run from a variety of
different recipes which may have different virtual extensions in place.
Depending on whether gcc-runtime or nativesdk-gcc-runtime's do_preconfigure
task is called for example will change the sorting of the task hashes due
to the way clean_basename currently works.
The correct thing to do here is sort on the base filename first, then any
extension when ordering the hashes. This means we do account for things
like recipes with both a native and non-native dependency but we also fix
the shared work case where we don't care whether it was a virtual version
or not.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The big warning printed when people use -f is easily ignored/forgotten.
To raise user awareness, print a warning any time we include a tainted
stamp file into a build instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the commandline options to be processed in the dump signature
code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we set:
BB_SIGNATURE_HANDLER = "noop"
Then we would get the following errors:
[snip]
File "runqueue.py", line 876, in RunQueue._start_worker(fakeroot=False, rqexec=None):
"fakerootnoenv" : self.rqdata.dataCache.fakerootnoenv,
> "hashes" : bb.parse.siggen.taskhash,
"hash_deps" : bb.parse.siggen.runtaskdeps,
AttributeError: 'SignatureGenerator' object has no attribute 'taskhash'
[snip]
This patch fixes the problem.
[YOCTO #5741]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* bitbake -S throws exception when 'noop' signature handler is used
[YOCTO #5738]
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.
The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).
Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.
We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The output when comparing siginfo files for dict_diff is reversed and shows
additions when things were removed and vice versa. This patch reverses the operation
so the changes are shown correctly and makes the output less confusing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the code prints all differences. If the task dependencies have changed hash,
we recurse into those and print those differences as well. This leads to a lot
of output. The reality is if the parents changed signature, we might as well just
say that and recurse with no other output since we're much more interested in how
the parents changed in nearly all cases. The changes in the parent are probably
the same ones we'd have printed at each level anyway.
By doing this we focus the output more carefully on the thing the user wants/needs
to see.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid storing paths to files in SRC_URI when writing out the the
file checksums to siginfo files. This prevents a move of the source
directory being reported by bitbake-diffsigs as files being removed and
then added (the signature itself is not affected since the file paths
have never been included in the signature).
This has required the format of the file checksums in the siginfo file
to be changed from a dict to a list of tuples (in order to handle
multiple files with the same name under different paths, which is
uncommon but possible); the code remains backwards-compatible with older
siginfo files that use a dict however.
Fixes [YOCTO #5245].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is clearly a missing parameter from this error message, add it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* without this change it's hard to see what's changed:
bitbake-diffsigs sstate-cache/*/sstate-depmodwrapper-cross-cortexa8-vfp-neon-oe-linux-gnueabi-1.0-r0-cortexa8-vfp-neon-*_populate-sysroot.tgz.siginfo
basehash changed from 62f28c13172d38af917716d6d4b15ac5 to 9cf38842bcc79e0a4356856a64141548
Variable TUNE_CCARGS value changed from -march=armv7-a -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8 to -march=armv7-a -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8
Hash for dependent task depmodwrapper-cross_1.0.bb.do_install changed from 4494cb6c117309a9b4c541ad86c0aac5 to 1e3bcf2d14e5bd30a38350b41bb98db3
* with this change, we can see extra trailing space in first version:
basehash changed from 62f28c13172d38af917716d6d4b15ac5 to 9cf38842bcc79e0a4356856a64141548
Variable TUNE_CCARGS value changed from ' -march=armv7-a -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8 ' to ' -march=armv7-a -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8'
Hash for dependent task depmodwrapper-cross_1.0.bb.do_install changed from 4494cb6c117309a9b4c541ad86c0aac5 to 1e3bcf2d14e5bd30a38350b41bb98db3
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Though the value of variables in the BB_BASEHASH_WHITELIST is kept out of the
checksums, dependency on them is not, at least for variables and non-task
functions. In the code, the whitelist is removed from the overall task dep
list, but not the individual variable deps. The result of this is that
functions like sysroot_stage_all and oe_runmake end up with whitelisted
variables like TERM listed in their dependencies, which means that doing
a 'unset TERM' before building will result in all checksums for tasks that
depend on those changing, and shared state reuse not behaving correctly.
This is only really a potential issue for variables from the environment, as
it's the existance/removal of the variable that's an issue, not its value, and
the other whitelisted variables are set in our metadata. This which means in
practical terms the only cases where this is likely to be an issue are in
environments where one of the following are unset: TERM, LOGNAME, HOME, USER,
PWD, SHELL. This may seem like an unlikely circumstance, but is in fact a real
issue for those of us using autobuilders. Jenkins does not set TERM when
executing shell, which means shared state archives produced by your jenkins
server would not be fully reused by an actual user.
Fixed by removing the whitelisted elements from the individual variable deps,
not just the accumulated result.
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The "*" part of the mask is to ensure we clean both any stamp, and any
setscene varient. It turns out we would also trample other tasks,
e.g. do_package_write could trample do_package_write_rpm. do_package also
tramples do_package_write_* but this is less of an issue since the other
tasks depend on it.
Rather than use the wildcard, we can just use a list instead.
[YOCTO #3484]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently when we execute a task, we don't remove other potentially stale
stamps. This can mean if you switch between two different versions of a
recipe without a clean, the build can get very confused.
This patch adds in functionality to allow a wildcard expression of stamp
files to be removed when creating a new stamp file. This patch adds in
the core of the code to enable this but it also requires metadata support
to enable it.
When writing this improvement I went through several different options but
this was the only way I could find to allow things like noexec tasks to
function correctly (where stamps need to be created without the data store).
[YOCTO #2961]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the ability to compare the two most recent runs of a specified task,
and follow dependent hash changes recursively. This enables you to trace
back and find exactly why a task was re-run after the fact.
Note that this relies on the metadata providing a function, hooked in
as bb.siggen.find_siginfo, which allows searching in the appropriate
places to find signature data files.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These functions become a little bit more reusable if they return a list
containing the output rather than just printing it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
before:
virtual:nativeautomake_1.12.1.bb.do_compile
after:
virtual:native:automake_1.12.1.bb.do_compile
This separation ensures that the key is readable, and if necessary,
parsable.
Unfortunately this invalidates previous native sstate signatures with
OE-Core - not much that can be done about that; however that occurs
frequently during the development cycle so it's par for the course.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We had one section of the code mixing tabs with spaces, which is
particularly undesirable with python code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use tempfile.mkstemp to create a temporary file in the sstate dir and move it
into place after closing. The previous code would fail in the chmod() if two
users were running jobs that touched the same signature file.
Signed-off-by: Jeffrey C Honig <jeffrey.honig@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Log a note when applying a taint to a task signature (e.g. when using
the -f or -C command line options) so that the user knows this has been
done.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.
This means you can now do the following:
bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage
and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.
Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.
This fix is required for [YOCTO #2615] and [YOCTO #2256].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a dependency to each variable on a filtered list of its varflags.
This is intended to catch things such as SRC_URI checksums, varflags
controlling extra functionality from classes (e.g. the recently updated
update-alternatives class in OE-Core), etc. and ensure their values
influence the sstate checksums.
There is an exclusion list which needs to be set via bitbake.conf
(BB_SIGNATURE_EXCLUDE_FLAGS), if this is not set then the functionality
is disabled. The existing vardepsexclude mechanism can also be used to
exclude undesired varflags, but they must be fully specified, e.g.:
do_patch[vardepsexclude] += "do_patch[someflag]"
Implements [YOCTO #2517].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We anticipate signatures being shared amongst peers in a group so ensure
they have rw-rw-r-- permissions to facilitate this.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The Python documentation states:
"When opening a file, it’s preferable to use open() instead of invoking
the file constructor directly." [1]
Further in Python 3 direct use of the file constructor is no longer
possible.
1. http://docs.python.org/library/functions.html#open
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bitbake-runtask doesn't work since there is no taskhash (it worked
in the past, but seems recent changes made it doesn't work again), use
the basehash when there is no taskhash, just like what dump_sigtask
does.
[YOCTO #1229]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If an error occurs during finalise() (when for example we might get an
expansion error in SRCPV), log a note which mentions which recipe was
being finalised.
Fixes [YOCTO #1782]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Markus Lehtonen
Richard Purdie
Paul Eggleton
Robert Yang
Martin Jansa
Christopher Larson
Jeffrey C Honig
Joshua Lock