| Age | Commit message (Collapse) | Author |
|---|
|
Make it easier for users to debug shell task failure by including
some breadcrumbs in the emitted .run file that (hopefully) points
to the .bb/.bbclass file where the shell function was defined.
Unfortunately this won't work with functions with _append
or _prepends, since BitBake wipes the filename/lineno information.
This shouldn't be too hard to fix; for now, you'll just see
comments like this for such functions:
[YOCTO #7877]
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a variable is in the signature whitelist, we'd currently expand
it, then later ignore the data. This is problemtic for code which
has effects when expanded, recently source date epoch in OE-Core
for example.
We don't actually need to do this, if we pass the whitelist into
the earlier function it can avoid the expansion. This also also
give a small performance boost since we avoid running code in some
cases.
[YOCTO #13581]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its hard to see what this exception adds in the current codebase. The logfile
attribute is effectively ignored, the exception doesn't serve a defined
purpose and mostly seems to be worked around.
Remove it entirely. If this does cause output problems, we'll figure
out better ways to address those.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Similarly to the codeparser change, change to sha256 hashes due to worries
over collisions. The main impact of this change is slightly slower parsing
time as well as longer sstate file names.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous check was in data.py which only can check code like "python
funcname()" in the dependency chain, but there are 3 kinds of python functions:
- python()
- def py_funcname()
- python funcname()
Add the checking to BBHandler to check and warn for all of them.
The warning looks like:
WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By creating a standalone function, we can add better functional testing
of this code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently remove operations are not being accounted for in the task
checksums. This is a fairly serious oversight and needs to be fixed.
To do so, we need internal data from getVarFlag combined with the
expanded variable data so that only "active" remove operators are
accounted for in the task checksum. We can get this from the new
optional removes attribute in the returned parser object.
The code can then use the data on active remove operators to account
for the removals in task checksum but only when the removal is active.
We have to be careful here not to reference any expanded data since this
may for example contain build paths. This means we can only map back
and reference the unsplit (and hence unexpanded) remove string which may
expand to multiple removal values.
[YOCTO #12913]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The datastore can assume internal API, this just removes the function indirection
overhead involved in this very common codepath (800,000 calls in parsing OE-Core).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Recipes/variables that raise a SkipRecipe exception are intentionally
skipped, and should not generate warnings.
[YOCTO #11319]
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The code that determines variable dependencies uses the codeparser to
find references to "contains" type operations e.g. bb.utils.contains().
That function can take multiple items to check, and all specified items
have to be present. However this code didn't handle that - it assumed
that only one item would be specified and thus it was treating the
multiple items as a single item with spaces in between. Split the value
and check if all words are present in order to determine whether the
check is "set" or "unset".
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Displaying the environment data in a sorted manner makes it easier
to compare data between different setups and means you can know
where to find specific entries.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Various run.do_xxx files export dozens of variables.
Presently they are in random order. Among other things,
this makes it difficult to compare two files for relevant changes.
This patch ensures they are enumerated/printed in a sorted
(alphabetical) order.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The old style bb.data.getVar/setVar API has long since been deprecated in
favour of d.getVar/setVar and friends.
Now we're about to change the default expansion parameter, drop the old APIs
to simplify the transition and ensure everyone is using the new style functions.
Conversion is trivial if there are remaining stragglers.
I've left bb.data.expand() for now since its more widely used but would make a good
follow up patch series.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix a bug where a totally wrong value of a variable would be exported if
an exception happened during d.getVar(). Also, print a warning if an
exception happends instead of silently ignoring it. It would probably be
best just to raise the exception, instead, but use the warning for now
in order to avoid breaking existing builds.
[YOCTO #10393]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The value used for exclusion was always being expanded. This is actually
a bad idea since in most cases you'd want to exclude an unexpanded
value and makes it impossible to use the variable as intended.
This adjusts things so the value is not expanded and we can correctly
remove things from checksums much more easily.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids a common issue where PACKAGECONFIG is emitted as a function in
bitbake -e when the 'python' flag exists. It isn't a python function unless
both 'func' and 'python' are set. This aligns with the behavior of
emit_func_python.
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Right now, if you have some python code like:
X = "a"
def somefunction(d):
d.setVar("X", "b")
d.setVar("Y", "${X}")
then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".
This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.
I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).
I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.
I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Expanding python functions for variable dependencies doesn't really make sense,
not least since this causes execution of any inline python, it also makes it
impossible to write expressions like d.expand("${X}") of d.setVar("X", "${Y}")
which may have the wrong values if expanded now.
This starts to standardise the approach across bitbake for handling python code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A newline is always appended to the function body when it's written
out, so strip any trailing newlines which may be there already.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This codepath can be triggered by a python indentation error for example.
Showing it as an ExpansionError is misleading.
Change the code to add a warning about where the failure came from (in
particular giving the variable key name that triggered it) but raise the
proper exception.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If parse_python() fails, the output is confusing. Passing in the extra
file/line data isn't expensive and improves readability significantly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Calling record() for each override alteration is slow. Since we now expand
overrides dynamically we don't have to record the log data at each alteration,
we can instead print it directly from the existing data stores at variable
history print time using the exact same data stores.
This massively improves performance of the data store when parsing
with bitbake -e for example, it will improve memory overhead as well.
The only downside is that VariableHistory has to poke into the datastore
for some of its data but that seems an acceptable tradeoff rather than
double caching.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should use the d.xxxVar syntax rather than the older function style.
Also replace 0/1 with the more pythonic True/False.
No functionality changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Minor formatting improvement in generated shell scripts.
Add a newline after the last export emitted by emit_func. Also, remove
chance of putting an extra newline in the middle of the exports, since
we never want to do that.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At least in theory, the order the keys are expanded in can make
a difference, particularly if there is key overlap.
We also want to ensure that any underlying base key is processed
before any overridden version of that variable (FOO before FOO_x)
which helps the update_data removal code I've been testing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This was incorrect, it *must* be done next to the rename as a previous
may overlap with the current one and we need to detect this case.
I'll add a test case to better catch this problem in future.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This comparison is interesting even in the case of empty vales. Enabling
this warning actually found a bug in the metadata in avahi. Make the
code handle None specifically and also remove the dead code path in the
second if statement.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By doing this we can take advantage of the expansion cache before
starting write operations on the data store.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the output in the python task/function run files is rather
incomplete and effectively useless. This enhances the code to take
advantage of the bitbake's dependency tracking and extend the output to
include dependencies. This makes the files more usable for debugging
purposes. Since this only happens at python function execution time, the
overhead is minimal in the grand scheme of things.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
On rare occasions it's useful to be able to exclude a part of a
variable's value from the variable's signature; for example if you want
to add an item to a list sometimes and not have the signature of the
variable change depending on whether the item is in the list or not. The
initial intended use case for this in OpenEmbedded is to allow adding a
function to SSTATEPOSTINSTFUNCS in buildhistory.bbclass and not have
that change any task signatures (so adding and removing
INHERIT += "buildhistory" won't lead to any rebuilds).
Part of the fix for [YOCTO #5897].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
pre/postfuncs were not being added to checksums. This meant that when reconfiguration
occurred, tasks were not always being rerun when they should. This include
sstate functions as well as systemd's do_install function in the OE metadata.
With the addition of postfuncs, its possible a shell task can have a python
pre/postfunc so we have to guard against this when generating shell output
in emit_func.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we don't sort the data, the values can reorder changing the
signatures meaning we get confused builds and significant cache
misses.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
One of the current frustrations with the sstate checksums is that
code like base_contains('X', 'y',...) adds a full dependency on X
and varies depend even on whitespace changes in X.
This patch adds special handling of the contains functions to expand
the first parameter and check for the flag specified by the second
parameter (assuming its a string).
The result is then appended to the value of the variable with a "Set"
or "Unset" status. If the flag is added/removed, the stored variable
value changes and hence the checksum changes. No dependency on X
is added so it is free to change with regard to other flags or
whitespace.
This code is far from ideal, ideally we'd have properly typed variables
however it fixes a major annoyance of the current checksums and
is of enough value its worth adding in a stopgap solution. It shouldn't
significantly restrict any propely typed variable implementation in
future.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
VAL = "" (not shown)
VAL = " " (shown as "")
VAL = " x" (shown as "x")
would all show up rather differently to what would be expected in the
bitbake -e output. This fixes things so they appear consistently.
The output for running some shell functions may also change slightly
but shouldn't change in a way that is likely to cause problems.
[YOCTO #5507]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the variable lookup to the outer loop for performance, replacing
a now unneeded parameter (after the previous changes).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Chris Laplante
Richard Purdie
Frazer Clews
Robert Yang
Nathan Rossi
Paul Eggleton
Juro Bystricky
Joshua Lock
Markus Lehtonen
Christopher Larson
Andre McCurdy
Richard Tollerton
Robert P. J. Day