| Age | Commit message (Collapse) | Author |
|---|
|
Also correct the description of variant2virtual().
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.
Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.
The idea is that this would make layer overrides possible:
OVERRIDES .= ":layer-${FILE_LAYERNAME}"
which then opens possibilities like:
WARN_QA:append:layer-core = " patch-fuzz"
as an example where OE-Core could enable specific QA tests only for that
specific layer.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
Set EXCLUDE_FROM_WORLD = "0" in recipe_A:
$ bitbake world -g
Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem:
EXCLUDE_FROM_WORLD = "1": Not build in world
EXCLUDE_FROM_WORLD = "0": Add back to world build
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.
This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.
The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.
The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By writing the cache files only if there is data to write, we can save a
bit of time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Joshua Watt pointed out maintaining the counting on both sides of the
connection isn't needed. Remove the receiver side counting and simplify
the code, also allowing errors if the counts do go out of sync.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a bug in the current cache code where the restored data structures
were "inverted" leading to some very weird issues, not sure how anything worked
like that, this patch fixes it.
Also fix some issues with multiconfig cache ordering problems by resetting
the stream counters when appropriate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The data in SiggenRecipeInfo is large and has a lot of duplication. The size
causes a few problems, impacting:
- bitbake's overall memory usage
- the amount of data sent over IPC between parsing processes and the server
- the size of the cache files on disk
- the size of "sigdata" hash information files on disk
The data consists of strings (some large) or frozenset lists of variables.
To reduce the impact we can:
a) deplicate the data
b) pass references to the object on the second usage
(e.g. over IPC or saving into pickle).
This patch does this for SiggenRecipeInfo mostly behind the scenes
but we do need a couple of reset points so that streamed data is written
correctly on the second usage.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Being able to track siggen hash construction data can be useful for cache
debugging. For now, add an extra cache class which contains this information.
It can be enabled in the same way as the hob data cache through a feature flag
to cooker. This allows us to experiment with the data without carrying larger
patches around and ultimately may allow use to have a hash mismatch debugging
mode that is more easily enabled.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.
This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't test not using the cache and I'm not aware of anyone using this, it
would be hard to with modern bitbake.
Drop the conditional code and simply error if CACHE isn't set.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At this point users appear to all call add_info directly. Failing
to do that means the file dependency tracking code isn't active
so would cause problems. Therefore drop the unused function.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some parts of functions in Cache() were broken and unused, there was
also a totally unused function. This was historical as a result of the
cooker parsing process needing to handle cached entries in the main
thread but parsing actions in seperate processes.
Document the way it works, update the function name to be clear about
what it now does and drop the old code which was unused.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We see rare failures in eSDK generation with zero sized unihash cache files. This is
almost certainly due to races in the cache file being updated. Add a copy function
where the cache file can be copied with the lock held to avoid this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
File names containing colons cause split() to return a list with more
than two elements, which will lead to a stack trace ending in:
ValueError: too many values to unpack (expected 2)
Split only once at the last colon, thereby making sure that only two
elements are returned.
Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This code has been unchanged since 2006 apart from attempts to optimise
performance by minimising chdir() calls.
There is no reason the modern bitbake parser should be changing directory
all the time. We did have some path assumptions in the mists of time but
those were resovled and the code is deterministic and doesn't depend on
cwd now for parsing. We can therefore drop the changes in directory.
Also, TOPDIR is now being set by cookerdata in all cases so we don't
need the fallbacks in this code (which was used to effectively initialise
a value). We don't need to change TOPDIR when parsing a recipe, that makes
no sense. If we stop all the other messing around, we don't need to expand
TMPDIR either.
These changes have the potential to break some obscure use cases such
as an anonymous function assuming the current working directory, or some
case which depends on TOPDIR changing but I believe any such uses should
be fixed at this point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This becomes a hard error in python 3.10.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This change updates the datastore to use the new override syntax using
colons instead of underscores exclusively. It is expected layers would
have to be converted to work with bitbake after this change.
Supporting mixed syntax isn't possible, it is only feasible to have
one internal representation of overrides.
Whilst we can't warn for every possible override that may be set in the
old format, show errors for _append/_prepend/_remove since those should
never be present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.
In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This will be needed by SkippedPackage in the cooker.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.
To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.
For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.
So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.
Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The layout of the cache data hasn't changed, but the cache has now been
split into different files for multiconfig. If a user pulls in these
changes, it's possible that their base cache will still contain the
combined multiconfig cache entries, which are now unexpected and
generate errors like:
Unexpected multiconfig: foo
Bumping the version fixes this since the old cache data won't be
considered valid anymore.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The virtualfn variable is not defined, the filename is what should be
shown instead.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is expected that load_cachfile() returns an integer indicating how
many entries were loaded from the cache. In the event the cache needs to
be rebuilt, 0 must be returned to prevent python from attempting to add
an None and an integer together.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Improves the logging of Cache objects by prefixing the log messages with
the multiconfig name of the cache, so as to distinguish between multiple
instances of the class. Also adds a more log messages.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some companies are using large numbers of patch files in SRC_URI.
Rightly or wrongly that exposes a performance problem where the code
does not handle the large string manipulations in a way which works
efficienty in python.
This is a modified version of a patch from z00539568
<zhangyifan46@huawei.com153340508@qq.com which addresses the performance
problem. I modified it to use a more advanced regex, retain the "*" check
and cache the regex.
[YOCTO #13824]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These messages spam the logs for no good reason, they were useful for debugging
a particular problem long ago but are distracting noise now. Disable them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds a simple version of the MultiProcessCache which can be used to
save and load cache data, useful for a new usecase we have in
sigdata/runqueue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
So that people or other tools can easily know which one is being used, just
like what we did for run.do_task and log.do_task, otherwise, we have no way
to know it. I usually use "ls -t", but it isn't reliable since the one which
is being used may not the latest one.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The provides and rprovides had been set for skipped recipes, packages are
similar to them (all of them provide something), so also set it. This makes it
easier to figure out the RDEPENDS issues, for example, lmsensors
(lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol
(lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons,
then if we run:
$ bitbake lmsensors
ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol']
ERROR: Required build target 'lmsensors' has no buildable providers.
Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol']
We had no way to know who rprovides lmsensors-config-fancontrol, we can figure
it out by bitbake/contrib/dump_cache.py after this patch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whether skip or not, they are always set, so move the lines ahead to avoid
duplicated lines.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.
mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.
[YOCTO #11168]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove a deprecated warning and stop using our own deprecated API!
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* base_conditional was already removed from oe-core:
http://git.openembedded.org/openembedded-core/commit/?id=0391fcad9103abca0796a068f957d0df63ab4776
after the usage was migrated to oe.utils.conditional:
http://git.openembedded.org/openembedded-core/commit/?id=c97acbd034532895ce57c6717ed1b3ccc7900b0d
so we can handle just ".conditional" version
* add 1st parameter to variable dependencies, that way when you use
FOO = "${@oe.utils.conditional('VAR', 'VALUE', 'true', 'false')"
FOO variable will have dependency on VAR variable and you don't need
to add FOO[vardeps] += "VAR" manually every time you use
oe.utils.conditional
* this is similar to contains tracking from:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=3890
http://git.openembedded.org/bitbake/commit/?id=ed2d0a22a80299de0cfd377999950cf4b26c512e
http://git.openembedded.org/bitbake/commit/?id=0b9d117631ce909312d53b93289e61defc6be01c
but conditional is simpler, we don't need to handle the first
parameter as a set
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The cache code currently inserts PN into the package list if it isn't already
present. Whilst this ensures that the package list contains something which is
important for native recipes that don't set PACKAGES, it causes confusing
behaviour where a normal recipe doesn't have PN in PACKAGES: for example adding
dhcp to IMAGE_INSTALL will parse successfully but fail at rootfs time as the
dhcp recipe doesn't generate a dhcp package.
Solve this by only adding PN to the cache's package list if the package list is
empty. This results in the package list for recipes such as DHCP being correct,
but native recipes continue to have just PN in the list as before.
[ YOCTO #5533 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Peter Kjellerstedt
Richard Purdie
Robert Yang
Roland Hieber
Alexander Kanavin
Tomasz Dziendzielski
Joshua Watt
Maxime Roussin-Bélanger
Frazer Clews
Martin Jansa
Paul Eggleton
Ross Burton