| Age | Commit message (Collapse) | Author |
|---|
|
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.
This also aligns the semantics with nproc.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid:
Exception: AttributeError: 'PosixPath' object has no attribute 'find'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some cases ${ may not be expanded in the WORKDIR (one of the cases is
undefined variable) and causes cryptic failures [1]. Guard this by
erroring out if directory name contains ${
Fixes: [Yocto #15255]
[1]
ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_hardcode_path(d)
0003:
File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path
0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd))
0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir)
0652:
0653: # If the fixmefn is empty, remove it..
*** 0654: if os.stat(fixmefn).st_size == 0:
0655: os.remove(fixmefn)
0656: else:
0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd))
0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath'
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Adds a utility that checks if one path is an descendant of another. This
check uses os.path.samestat() to make it immune to symlinks and bind
mounts.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some variables may be set as:
X = 1
as well the more usual
X = "1"
so add support to to_boolean to handle this case.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
|
|
It can be used to enable the loopback interface, typically after calling
disable_network().
Also correct a typo in a debug message.
Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
|
|
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
If the path to bitbake.lock is in a deep directory, bitbake will hang. The
reason was that the max file length limiting code (to 255 chars) was including
the directory name and it should only act on the filename within the directory.
Fix it to just use the base filename.
[YOCTO #14766]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
hashlib.md5() is not permitted on a FIPS enabled host system. This is due
to md5 not being an approved hash algorithm.
Instead use:
hashlib.new('MD5', usedforsecurity=False)
This is allowed, as it's clear the hash is used for a non-security purpose.
Note: utils.py version should never be used to verify file integrity, but
instead be used to identify if the file may have changed. sha256 should be
used for integrity purposes.
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH
BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
(Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.
The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus
with bb.utils.environment(**newenv):
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment
del os.environ[var]
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__
raise KeyError(key) from None
It is possible the wrapped function might change the environment which can lead
to KeyError issues like this. The key may no longer be in the original environment
so handle that case correctly and avoid the tracebacks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is deprecated in python 3.12 and Fedora 35 is throwing warnings so
move to the new functions.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The fetcher mirror code can go crazy creating lock filenames which exceed the
filesystem limits. When this happens, the code will loop/hang.
Handle the filename too long exception correctly but also truncate lockfile
lengths to under 256 since the worst case situation is lockfile overlap
and lack of parallelism.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bb.utils.environment() is a context manager to alter os.environ inside
a specific block, restoring it after the block is closed.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
os.rename can fail for example an incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.
[YOCTO #14301]
Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A list of functions that now has a docstring.
* vercmp_string
* explode_dep_versions
* prunedir
* prune_suffix
* to_boolean
* contains_any
* export_proxies
See [YOCTO #9725] for details.
Signed-off-by: Milan Shah <mshah@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a umask context manager which can be used to temporarily change the
umask in a 'with' block.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
PEP 3110 changed how exceptions work. 'e' is unbound
after the 'except' clause. See: https://www.python.org/dev/peps/pep-3110/#semantic-changes
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The following code costs a lot of time when there are lot of layers and recipes:
for collection in collections:
collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)
The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't
make sense to go on the loop when file is matched.
And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I strongly suspect this function doesn't work with modern python so
and its unused now, drop it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The value of TERM is leaking into OE-Core postinst-useradd-${PN} scripts,
which in turn can optionally be monitored by buildhistory. Prune the value in
order to make the OE-Core buildhistory output more deterministic.
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806
Uncomment lchown() to preserve ownership of symlink.
Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This function checks if a string is a semantic version:
https://semver.org/spec/v2.0.0.html
The npm fetcher needs this function to validate its version parameter.
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The npm fetcher needs these functions to support the subresource
integrity: https://www.w3.org/TR/SRI/
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Given a start expression, bb.utils.get_referenced_vars returns the
referenced variable names in a quasi-BFS order (variables within the
same level are ordered aribitrarily).
For example, given an empty data store:
bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d)
returns either ["A", "B", "C"], ["A", "C", "B"], or another
permutation.
If we then set A = "${F} ${G}", then the same call will return a
permutation of [A, B, C] concatenated with a permutation of [F, G].
This method is like a version of d.expandWithRefs().references that
gives some insight into the depth of variable references.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by
externalsrc.bbclass) was leaked, giving a warning like:
WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'>
exec_func(task, localdata)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
md5_file() uses a mmap() window to improve performance when hashing files, so
refactor the code and do the same for SHA1 and SHA256.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Autobuilder type infrastructure can benefit from deletion of certain files as
background IO due to the way Linux filesystem priority works.
We have problems where build directories as part of oe-selftest being
delete starves the running tasks of IO to the point builds take much
longer to compelte.
Having this option of running the deletion at "idle" helps a lot with
that.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* with python3 this fails with:
File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile
0795: try:
0796: os.rename(src, destpath)
0797: renamefailed = 0
0798: except Exception as e:
*** 0799: if e[0] != errno.EXDEV:
0800: # Some random error.
0801: print("movefile: Failed to move", src, "to", dest, e)
0802: return None
0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device
Exception: TypeError: 'OSError' object is not subscriptable
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its hard to see what this exception adds in the current codebase. The logfile
attribute is effectively ignored, the exception doesn't serve a defined
purpose and mostly seems to be worked around.
Remove it entirely. If this does cause output problems, we'll figure
out better ways to address those.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Felix Moessbauer
Michael Opdenacker
Richard Purdie
Pavel Zhukov
Julien Stephan
Joshua Watt
Ross Burton
Frank de Brabander
Mark Hatle
Mattias Jernberg
Jose Quaresma
Scott Murray
Zygmunt Krynicki
Robert Yang
Ross Burton
Devendra Tewari
Milan Shah
Chris Laplante
Jacob Kroon
Daisuke Yamane
Jean-Marie LEMETAYER
Chris Laplante via bitbake-devel
Frazer Clews
Ross Burton
Martin Jansa