| Age | Commit message (Collapse) | Author |
|---|
|
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes
except bb.parse.SkipRecipe:
> bb.data.setVar("__SKIPPED", True, d)
if include == 0:
AttributeError: module 'bb.data' has no attribute 'setVar'
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"inherit" already allows inheriting more than one class in a single
statement. The same also makes sense for "include" and "require",
because then one can generate a list of files to be included
dynamically also for the case that more than one file needs to be
included.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Writing .bbappends that only have an effect when some configuration
variable like DISTRO_FEATURES is changed becomes easier when allowing
"include" or "require" without a parameter. The same was already
allowed for "inherit".
Then one can write in a .bbappend:
require ${@bb.utils.contains('DISTRO_FEATURES', 'foo', 'bar.inc', '', d)}
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the regular expression for matching a variable name was amended
with allowing the ~ character as part of the variable name, this was
never done to the regular expression that matches export
lines. Similarly, the regular expression that was used for matching
unset variables also used the one without support for the ~ character.
This unifies the regular expressions. For good measures it also
corrects the regular expression used to match a variable flag name for
the unset command to match the one used when setting a variable flag.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is more pythonic and can handle unclosed file warnings better
than the previous code structure.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #10186]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR
Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Simple refactoring to allow for multiconfig support.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When parsing, we should reset the event handlers we registered when
done. If we don't do this, parse order may change the build, depending
on what the parse handlers do to the metadata.
This issue showed up as a basehash change:
ERROR: Bitbake's cached basehash does not match the one we just generated (
/media/build1/poky/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb.do_unpack)!
This is due to the eventhandler in nativesdk.bbclass being run, despite
this .bb file not inheriting nativesdk.bbclass. The parse order was
different between the signature generation and the main multithreaded
parse.
Diffsigs showed:
bitbake-diffsigs 1.0-r2.do_unpack.sigbasedata.*
basehash changed from 887d1c25962156cae859c1542e69a8d7 to cb84fcfafe15fc92fb7ab8c6d97014ca
Variable PN value changed from 'nativesdk-buildtools-perl-dummy' to '${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}'
with PN being set by the event handler.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Anonymous functions are python functions, set the variable
flags as such so we can detect them and avoid expansion where
needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now we're actively using the line numbers for other thins, having
magic values like IN_PYTHON_EOF causes problems, in particular, 32
bit overflow on 32 bit machines.
There is a neater way to signal eof to feeder(), just using an extra
parameter so use this instead and drop the IN_PYTHON_EOF magic values.
This has the added bonus that line numbers are then correct for
python functions at the end of files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.
The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.
This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).
Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined
ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
fire_class_handlers(event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
ret = handler(event)
File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined
----------------
After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
if isinstance(e, bb.build.TaskStarted):
> trigger = notexist
pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined
ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
fire_class_handlers(event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
ret = handler(event)
File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
trigger = notexist
NameError: global name 'notexist' is not defined
----------------
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Trying to expand a variable which isn't a string doesn't make sense.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.
While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.
Fix this by handling the ENOENT case specifically.
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).
To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we're not going to change the value of FILE, or we know it isn't
going to have changed (ext == bbclass), don't set FILE.
This avoids messy looking history of the variable as well as optimises
parsing speed slightly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.
This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.
The solution is to simply remove the bogus dependency.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst bitbake has done this for a long time, the behaviour of resolving
class files against cwd is not desirable. This can be seen during
base configuration parsing when looking for base.bbclass where a dependency
on cwd is added. If cwd then changes, the cache is invalid and triggers a
re-parse.
The only real option is to drop this entry and if files can't be found, we
fix BBPATH in the cases where it needs fixing. I didn't find any in the
random selection of layers I tested parsing locally.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add some sanity checks on the parsing state engine when returning data
so that incomplete functions raise parse errors.
This means a recipe doing:
do_somefunction {
echo 1
VAR = "1"
will now raise a ParseError. To get the right file/line information,
__infunc__ was changed to a list.
[YOCTO #7633]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The file inheritance logging has been here since the dawn of time. It
duplicates output many times over and logs to debug level 2. When running
with the debug option, its understandable the user may want to see the
paths of files included in the build.
These changes remove pointless/duplicate output and print the
include paths clearly at debug level one in a form which
users should be able to more easily parse.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was reported that bitbake -D made no mention of which append files it
was using. bitbake -DD does but it makes sense to increase the log level
of this piece of debug information.
[YOCTO #6262]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The optimisation where only the data we're interested in was finalised
was good but it turns out we can do better. In the case where a
class-extension is to be targeted, we can skip the other targets.
This change does that and speeds up parsing at the bitbake-worker
execution time. Specifically, you can see an improvement in the speed
of bitbake X -n.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When an evaluation was made for a configuration file the path to the
file was saved as a relative one. The change in this commit will save the
location as an absolute path. This way the user will have full information
regarding the location of the file where a variable was changed and the
line withing the file.
[YOCTO #5562]
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using wildcard .bbappend files with anonymous methods in them,
bitbake/python fails to parse the generated code since the '%' is encoded
in the generated method name.
Fix this by including '%' in the convert-to-underscore list during
method name mangling.
While we're at it, move the method name mangling translation table
to a class variable, as suggested by Chris Larson.
[YOCTO #5864]
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It doesn't really make sense to set PN from .conf files, for example.
More concretely, this avoids the config hash changing unnecessarily
within Hob due to PN effectively changing (since bblayers.conf is
parsed first and then .hob.conf).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently its near impossible to control task addition/deletion from
metadata context. This adds stong add/deltask API to bb.build
which is traditionally where it resided. The rather broken
remove_tasks function was removed, it didn't appear to do anything
useful or have any users.
This allows us to clean up hacks currently in use in metadata and use
standard API for it instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Back in the depths of time we did support task removal. In the pre
AST days it was nearly impossible to continue supporting it, it wasn't
used so it was dropped. With the modern codebase we can easily now support
deltask and it would be very useful within the metadata since it can
massively simplify dependency trees.
As an example, a core-image-sato had 47703 inter task dependencies before
this patch and a patch to native.bbclass, afterwards with the noexec tasks
deleted, we had 29883. Such a significant simplification is worthwhile
and justifies adding a deltask operation to the system.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Enrico Scholz
Patrick Ohly
Ola x Nilsson
Peter Kjellerstedt
Ross Burton
Joshua Lock
Markus Lehtonen
Jérémy Rosen
Christopher Larson
Paul Eggleton
Pascal Bach
Marius Avram
Jacob Kroon