| Age | Commit message (Collapse) | Author |
|---|
|
This patch subclasses TraceUnpackBase in order to implement full
upstream metadata collection and processing for do_unpack.
The final output is a compressed json file, stored in WORKDIR/temp
for each recipe. Data format is described in the help text of
bb.trace.unpack module, while some real-world examples can be found
in lib/bb/tests/trace-testdata
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
|
|
do_unpack currently unpacks all SRC_URI entries into WORKDIR, and can
even mix files coming from multiple SRC_URI entries into the same
subdir, making it hard to trace each source file found in WORKDIR back
to its corresponding upstream source.
Being able to trace source files to their corresponding upstream source
is fundamental for Software Composition Analysis (SCA), Software Bill
of Materials (SBoM) generation (create-spdx.bbclass), license compliance
checking and CVE checking.
To solve this issue, this patch implements a process that consists of:
1) unpacking each SRC_URI element into a temporary directory
2) collecting relevant metadata for Software Composition Analysis (file
sha1, upstream download location (in SPDX-compliant format), relative
path in the upstream repo/package, etc.);
3) moving everything to WORKDIR, and iterate with the next SRC_URI
element;
4) saving metadata in a json file after all SRC_URI elements have been
processed
By patching the relevant fetcher modules and adding a bb.trace module,
this patch implements steps 1,3,4 , while it provides only a bare-bone
implementation of step 2, in which all relevant raw metadata (file
paths, url, urldata, real destination dir, npmsw dependency tree, git
submodule revisions) are collected, but not processed nor saved.
This should allow to develop a full implementation of step 2 (data
collection) in a separate module independently from the development
of the rest of bb code, i.e. without the need of further patching bb
fetchers.
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
|
|
Bitbake expects a consistent metadata environment when it parses. There
are plenty of ways you can set a recipe to autorev at a point where parts
of the fetcher have already been triggered leading to obsure bugs which
I struggled to debug, let alone anyone not familar with the code.
If anyone is running into the message from the commit, the issue is
likely one of timing. Keep in mind that the anonymous python code
in base.bbclass will expand variables like FILESPATH, WORKDIR and others
which contain PV. The recipe needs to be set to AUTOREV before that
anonymous python runs.
In particular, that means you can't set SRCREV = "${AUTOREV}" in other
anonymous python, it needs to happen earlier.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For readability of following patches, rename this internal variable to
allow for others in a similar format.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After updating gen_fixtures.py for 'mickledore' 4.2,
run './gen_fixutures.py --all' to update oe-core.xml
and poky.xml.
Drop langdale as support is ending soon.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update for 'mickledore' 4.2.0 release and bitbake 2.4 branching.
Drop 'langdale' as support will be ending soon.
Update stable branches to latest patch release.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When tasks are run with -v (verbose) on the bitbake commandline, shell
tasks print their stdout, python tasks do not.
This change redirects the python task's print output to an in memory
buffer. After the task is executed the output is printed to stdout via
the logger. This makes the python task behavior match the shell task
behavior when running with -v. The contents of the task's log files
remain unchanged after this change.
This approach should keep the correct order in most cases, however, if
the python task accesses the logger directly, that content will appear
before other output. On the other hand, this change should negate the
need for python tasks to access the logger directly.
Special care is taken to save/restore the existing stdout and stderr
and preventing sending output directly to the logger when there are
"recursive" calls, for instance when a python function calls a shell
function, avoiding printing things potentially out of order and/or
multiple times.
The logging-test.bb in meta-selftest can be used to review this
change. This has been tested with the full bblogging oeqa tests.
[Yocto #14544]
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allow to have classic fetch parameters
(like destsuffix, sha256, name ...) not being
considered by crate fetcher itself (and so mess
up its download)
Moreover, it allow to overload the name of the downloaded
crate (maybe usefull if there is a naming clash between
two crates coming from different repositories)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
No reason to have three identical exception handles, refactor to catch any
of the exceptions with the same block of code.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
We've observed TimeoutError exceptions during the sstate-cache mirror fetch,
it appears that due to the number of (invalid) files requested the remote
side is eventually dropping the connection (not closing it) which can result
in a TimeoutError exception being sent, while rate it is different from the
urllib.error.URLError or ConnectionResetError.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Fix broken link "Obtaining bitbake".
Update documentation for the bitbake hello world example, the output was
outdated.
Fix LAYERSERIES_COMPAT warning by adding dunfell as default compatible release.
Add proper formating for base.bbclass command.
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake throws a warning if the layer compatibility is not defined since
cca81e33b58c390dcf5cc3a31555a43b79177166. The description of this variable
comes from the Yocto Project manual.
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3,
"process/cooker/command: Fix currentAsyncCommand locking/races", command.py
assumes it has access to the process server but the xmlrpc backend was
passing in the xmlrpc server object leading to errors like:
xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'">
Fixing to pass the process server to command.py resolves this issue.
[YOCTO #15008]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #11605] by:
- Adding definition of file-checksums to Variable Flags section.
- Describe data to add to list which adds external file dependencies.
- Write example on usage to prepend a value to file-checksums list.
Signed-off-by: Richard Elberger <rich@richelberger.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
>From the npm-install documentation [1] the CLI provides a set of
short forms when the install fetches from git. These include
"github:"
example: npm install github:mygithubuser/myproject
"gist:"
example: npm install gist:101a11beef
"gitlab:"
example: npm install gitlab:mygitlabuser/myproject
"bitbucket:"
example: npm install bitbucket:mybitbucketuser/myproject
Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in
npm shrinkwrap version] by Stefan Herbrechtsmeier added support for
the "github:" but the others would marked as 'Unsupported dependency'.
The other prefixes are added in this commit, along with extending the
tests to cover some of these.
However, there is one more short form for github which npm-install
allows which forgoes the prefix altogether.
example: npm install mygithubuser/myproject
Unfortunately this format is a bit problematic as it lacks any easily
identifiable 'marker' to match against, and it could be either the
github short form or install from folder format. Experimentation shows
that the folder format requires a leading './' or '/', so we use this
to rule out the ambiguity.
If this approach to folder and github formats disambiguation is
incorrect it won't matter anyways as the folder format is unrecognized
by the code as-is and thus with this change or without, things would
fail.
Since we have to be less strict in the check for git operations we
move it to be the last install format which we check, such that the
less ambiguous formats can be sorted out first.
[1] https://docs.npmjs.com/cli/v9/commands/npm-install
[Yocto #14236]
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The submodule repository URI contains a path to something not
necessarily on the local filesystem. This means that we can't use
realpath to normalise it without risking getting bad results if the path
happens to match something on the local filesystem. This situation can
cause very confusing errors if that matching local path happens to be a
symlink to somewhere else.
Using normpath rather than realpath means that the path simplification
follows simple rules on the string rather than looking at the local
filesystem and avoids problems.
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Co-authored-by: Dave Craig
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This import is no longer used anywhere so can be removed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These environment variables are needed by gclient and needed to be
passed into fetcher.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've been seeing weird PRServ failures on the autobuilder. These had
one failure always followed by a second. Whilst I can't reproduce the first,
if I made that test fail, I could reproduce the second with memory resident
bitbake. This was with the tests:
prservice.BitbakePrTests.test_import_export_replace_db
and then
prservice.BitbakePrTests.test_pr_service_deb_arch_dep
which was giving a strange looking error:
NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm)
NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa)
ERROR: No such task: do_package_write_rpm
ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1'
where the issue is that selftest.inc written by the test framework
and containing PACKAGE_CLASSES = "package_deb" was being ignored.
The issue is the cached_statements{} within BBHandler() is not being
invalidated at the right time.
This patch changes the code to ensure base configuration is not parsed
until inotify updates have been processed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake f68682 changed the logger's debug() method to be compatible with
logging.debug(), but this caller was still using the old API where you
can pass an integer as the first argument:
WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]')
Instead, call bbdebug() which has the priority argument.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake is now able to understand when a UI wants it to stop the current
processing. There are some long running functions which currently have no
mechanism to interrupt them however.
This patch adds a call, bb.event.check_for_interrupts(d) which can be
placed in long running code and allows an internal state flag within
bitbake to be checked. If set, that flag will trigger an exit.
This means that Ctrl+C can be made to work in a variety of places where
it currently would not.
Long running event handlers in OE-Core can also then benefit from this
new approach with the addition of the function call as well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Due to using mkdtemp instead of TemporaryDirectory we needed to
manually cleanup the directory in a try finally block.
With tempfile.TemporaryDirectory we can handle the cleanup with
a "with" statement and not need to manually clean up oursevels.
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
branchname was set but never used in the context of _contains_lfs
method.
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.
The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.
lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.
lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.
lfs=0 actually disables git-lfs functionality even if supported.
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Not restoring the mocked _find_git_lfs leads to other tests
failing.
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the inotifier code has an exception, bitbake currently hangs. Catch any
exception and exit if seen. Also check the idle thread is alive and exit
if it disappears. This should stop bitbake hanging if such a situation arises
in future such as this example:
3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
self.cooker.process_inotify_updates()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
n.read_events()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've seen a couple of cases which bitbake hangs due to an inotifer exception
such as:
3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
self.cooker.process_inotify_updates()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
n.read_events()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)
Ensure we don't destory the inotifier when the idle thread is reading is
by holding the lock during setup/teardown.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using bb.fatal for a fatal error message is the best practise, switch
the code to match other call sites.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This code appears to be dangerous, it swallows exceptions, turning them into
"handled" versions which then show no errors to the user. This is a pretty
poor user experience and I can't see why this code should be swallowing
such things. Drop the worst bits of code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Martin Jansa reported that if you put a syntax error into an imported
module such as qa.py in OE, no error is shown.
Part of the issue appears to be that the catch_parse_error() decorator only
catches certain exceptions and SyntaxError isn't one of them. As far as I can
tell we should remove all the special cases and use the more advanced code
in all cases, not just expansion errors.
I confirmed this now prints a proper error message for a qa.py syntax error.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By including the full path to bblayers.conf the remove-layer
command can be executed from any location, not only from the
build directory.
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By including the full path to bblayers.conf the add-layer
command can be executed from any location, not only from the
build directory.
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Assigning a return value which is potentially None to a tuple and
catching TypeError is pretty ugly. Rewrite the code to explicitly check
the value for clarity.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the export/unexport/network flags are acted on only by presence
or lack of in the data store. This is deliberate due to performance reasons
since a given recipe shell task might export ~80-90 variables and this means
expanding flags for every one of them.
This does catch users unaware since values which expand to nothing don't work
e.g. ${@""} and setting values to "0" wouldn't work either.
The downside to this patch is slow down in parsing speed of around 4%.
The network flag is easier to change and doesn't affect performance, it was
made to operate the same as export/unexport for consitency reasons.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you are behind a corporate proxy, the npm fetcher uses
the proxy IP address already passed in the list of exports.
However, it will error if the proxy injects its own root
CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment
variable must be passed so the user can include their
company's root CA as a trusted CA inside node's
certificate store.
Signed-off-by: George Kelly <george.kelly1097@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
|
|
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]
For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.
Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.
Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"
However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))
I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.
Co-authored-by: JJ Robertson <jrobertson@snap.com>
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Increase the `ljust` value for each column in show-layers
output. This improves readability when long layer paths are
included
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Replace the layer directory name with the name from BBFILE_COLLECTIONS
which is useful when assigning a LAYERDEPENDS value
Signed-off-by: Pedro Baptista <pedro.miguel.baptista@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.
This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.
The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.
The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Update to latest langdale 4.1.2 and kirkstone 4.0.6
* Re-instate dunfell and update to 3.1.22
- drop comments about bitbake crash
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
bitbake-getvar does not have a way to silence bitbake
server's logger and that makes the tool hard to use for
text processing. This is especially true when one wants to
get a bitbake value to be piped to some other utility and
instead we get uncontrolled logging messages or warnings
together with bitbake's variable value.
Example without quiet:
bitbake-getvar --value MACHINE
NOTE: Starting bitbake server...
qemux86-64
With quiet:
bitbake-getvar --value MACHINE --quiet
qemux86-64
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Deprecated in Python 3.10:
https://docs.python.org/3/whatsnew/3.10.html#deprecated
https://github.com/python/cpython/pull/25174
Fixes warnings like:
...bitbake/lib/bb/ui/uievent.py:68: DeprecationWarning: setDaemon() is
deprecated, set the daemon attribute instead
self.t.setDaemon(True)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some users of _findVar don't need the override data and even
getVarFlag doesn't need it in some common cases (parsing=True).
Rearrange the code as the current overridedata approach doesn't need
to be in the _findVar code anyway. This removes some search overhead
from a critical path.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the codeparser cache ends up being extended for every parse run
since there are values in the functions such as the result of os.getpid()
from LOGFIFO in OE-Core.
Digging into that issue, there are also lots of similar but different
functions being parsed where the change might just be a path to WORKDIR,
a change in PN or PV or something like DATE/TIME.
There is no reason we have to use these changing values when computing the
dependenies of the functions. Even with a small tweak like:
BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn"
the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run
to around 1.3MB and remaining static for oe-core and meta-oe. In my local
build, admittedly heavily experimented with, the cache had grown to 120MB.
The benefits of doing this are:
* faster load time for bitbake since the cache is smaller to read from disk
and load into memory
* being able to skip saving the cache upon shutdown
* lower memory footprint for bitbake
* faster codeparser data lookups (since there is less data to search)
We only use these special values when passing code fragments to the codeparser
to parse so the real variable values should otherwise be used in the hash data.
The overall effect of this change, combined with others to avoid saving unchanged
cache files can be ~2s on a ~16s parse on my local system and results in a more
responsive feeling bitbake. It also allows parsing performance to be investigated
more consistently.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When draining the result queue from the parsing processes, cache objects
can be created even if they are then immediately destroyed. The reset
in the sync code needs to happen after any objects have been created.
Change the ordering to fix this. This ordering has caused various
cache errors, particularly when interrupting parsing with Ctrl+C.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Alberto Pianon
Richard Purdie
Tim Orling
Mark Asselstine
Frederic Martinsons
Mark Hatle
Fawzi KHABER
Richard Elberger
Mike Crowe
Caner Altinbasak
Ross Burton
Paulo Neves
Pedro Baptista
George Kelly
Etienne Cordonnier
Michael Opdenacker
Paulo Neves