| Age | Commit message (Collapse) | Author |
|---|
|
We want things to be reproduicble and the variable doesn't really change
much any more. Drop the remaining uses and make those code paths always
active.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reproducibility is here to stay and needs to be part of our default workflow.
Move the remaining code to base.bbclass so it is always a first class citizen
and it is clear people need to be mindful of it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To try and avoid parse/memory overhead of functions within bitbake,
move the bulk of the reproducibility functions to the function library.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow a user to validate sstate objects against a list of keys, instead
of just any known key in the user's keychain.
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, installed packages are listed for images in image-info.txt, but
not for SDKs in sdk-info.txt. Add TOOLCHAIN_HOST_TASK and
TOOLCHAIN_TARGET_TASK to the output variables in sdk-info.txt.
Moreover, package output files for the SDK host are empty because
PKGDATA_DIR defaults to the target directory. Fix this bug and create a new
variable called PKGDATA_DIR_SDK which stores the correct path for the SDK
host package data.
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Use regex strings (r’’) to silence below deprecation warning [1]:
$ cat tmp/work/intel_x86_64-wrs-linux/linux-yocto/5.10.x+gitAUTOINC+917c420111_373c02c3ca-r0/temp/log.do_deploy
[snip]
/build/layers/oe-core/meta/lib/oe/packagedata.py:22: DeprecationWarning: invalid escape sequence \s
r = re.compile("(^.+?):\s+(.*)")
[snip]
[1] https://docs.python.org/3/library/re.html
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 2a76082363d189880613765ad339718e3614049d.
We have an issue where x86 host builds are not matching hashes with
aarch64 host builds. We'd expect that for a given target, the target
artefacts should work regardless of the host architecture, compiler
version etc. but this isn't happening and the hashes are differing.
This is due to issues from hash equivalence.
I believe the commit being reverted was added as a test and there were
other fixes at the time which resolved these issues.
As illustration of that, different gcc versions are not cauing issues
with hash equivalence. That should be similar to the aarch64 case vs.
x86-64 and hence if we're not seeing gcc verison issues, we also don't
need this special case. As such, revert it as we don't need it and it
is in fact breaking sstate reuse cross platform.
[YOCTO #14578]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds several comments to this file to explain the intention and how it
should be used
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is added to allow the create-spdx code to create annotations
that store values properly according to the SPDX Specification.
Initialy they will be used to track if a recipe is a native type.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
collections.abc is available since 3.3 and doesn't need special handling.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The package task references WORKDIR at it's top level and we can't
easily make the timestamp for that determnistic due to writes to files
there and in other subdirs. We could try and force it to a specific value
but it is easier to just remove it from the package task, we don't need
it there or care about it in this case.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Repo-wide replacement to use newer variable to represent systemd
system unitdir directory.
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Licenses reported in the SPDX documents should be either:
A) A valid SPDX identifier cross referenced from the SPDX license
database
B) A "LicenseRef" to a license described in the SPDX document
The licensing code will now add a placeholder extracted license with
corresponding "LicenseRef" for any licenses that are not matched to the
SPDX database
Parenthesis in the license expression are now handled correctly
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds a class as a first attempt to create SPDX SBoM documents during the
build. This initial work was influenced by [meta-doubleopen][1],
although almost completely rewritten.
[1]: https://github.com/doubleopen-project/meta-doubleopen
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds extended package data which is encoded as JSON which allows it to
encode more structure than the "flat" package data files. The extended
data might be much larger than the standard package data, so it is not
read by default and instead requires
oe.packagedata.read_subpkgdata_extended() to be called
Currently, the file sizes and ELF debug sources are saved off into the
extended package data
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Reviewed-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, depends.dot includes per-file dependencies but not the packages
providing those files. This makes it hard to obtain all package
dependencies by just looking at depends.dot.
Parse the RPROVIDES and FILERPROVIDES fields from pkgdata to map each of
their values to the package providing the component. Include runtime
packages as dependencies in depends.dot, together with the component
provided by the package as a label.
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch is a follow on of the the PR server rework in bitbake to add
read-only support. The shift to using the bb.asyncrpc code in the PR
server and client brings issues with respect to reuse of the same
asyncio loop in different processes. This patch removes the PR service
connection caching to avoid one source of this problem. It is believed
that in practice this should have little impact on overall performance.
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add entries for Android to the mappings.
Signed-off-by: Hsia-Jun Li <randy.li@synaptics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix read_subpkgdata_dict to account for the new override operator being
":" instead of "_"
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Moving the function will allow other classes to capture which CVEs have
been patched, in particular SBoM generation.
Also add a function to capture the CPE ID from the CVE Product and
Version
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The automated conversion of OE-Core to use the new override sytax isn't
perfect. This patches some mis-converted lines and some lines which were missed
by the automation.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Removes the /var/cache/ldconfig auxiliary cache directory from
the rootfs when:
- read-only-rootfs is in IMAGE_FEATURES,
- ldconfig is not in DISTRO_FEATURES.
In both cases the /var/cache/ldconfig/aux-cache is useless.
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
install_complementary
When attempt_only flag is set to True all installation failures are
ignored. Disable attempt_only, that is the default in install function,
to raise error in install_complementary and to prevent do_populate_sdk
task to treat errors as warnings.
[RP: Historically some of our addtional packages were in bad shape and
wouldn't all install. This is no longer the case so we shouldn't need
a hack like this any longer.]
Signed-off-by: Fabio Berton <fabio.berton@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds an option to strip a prefix from the paths reported in
buildhistory. This makes it easier to compare task signatures in the
build history when the builds were done from different directories.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Parse systemctl error to provide more info
[Yocto #14395]
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* all known issues in this area were fixed, make it fatal that it
cannot be overlooked if someone triggers this issue again
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A stray space made it into the command for verifying gpg signatures.
This caused verification to fail, at least on my host. Removing the
space makes it work as expected.
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the scripts/postinst-intercepts is owned by root/root then the copyfile() calls
will fail due to chown issues. We don't care about ownership of these files so
use shutil.copy() instead which won't perform any chown.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.
[YOCTO #14301]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* with xz PACKAGECONFIG enabled in kmod and xz module compression enabled in kernel
the do_rootfs task doesn't run depmod in the image, because it thinks there are no modules:
NOTE: No Kernel Modules found, not running depmod
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Christophe Chapuis <chris.chapuis@gmail.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The new prserv connect() function decouples the code in oe-core from the
exact classes and implementation details used within bitbake. This
allows us to more easily switch over to a new asyncrpc based prservice.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If not postinstall applied, some nativesdk command could not be found
in sdk due to update-alternatives in postinst not be executed, such as chroot:
$ which chroot
/sbin/chroot
$ which chroot.coreutils
path-to-sdk/sysroots/x86_64-wrlinuxsdk-linux/usr/bin/chroot.coreutils
After applying the fix
$ which chroot
path-to-sdk/sysroots/x86_64-wrlinuxsdk-linux/usr/bin/chroot
$ which chroot.coreutils
path-to-sdk/sysroots/x86_64-wrlinuxsdk-linux/usr/bin/chroot.coreutils
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
`tmux new -c` fails on tmux older than 1.9, when that flag was added.
We can omit the flag for older versions of tmux, and the working
directory gets set even without it.
Signed-off-by: Peter Budny <pbbudny@amazon.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since pyc cache files are really system specific, no real reason to copy or
generate them during the eSDK build process. Also generating them has the
possibility of re-using inodes that pseudo may have been tracking, leading
a build failure.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The run-postinsts package runs post installation scripts
on target if packages request delayed post installations. When
no delayed post installations are found the sysV style scripts
are disabled for the package and hence it did not run on sysV
based systems. However, the package provides systemd service
as well which still ran on systems based on systemd even when
no post installations were found.
Rather than disabling/masking scripts for different initialization
managers we now simply remove/uninstall the run-postinsts package
when no post installations are found to be delayed till runtime.
This is also more aligned with the function (_uninstall_unneeded)
this functionality is triggered through.
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For the purpose of image construction using du on a rootfs directory isn't
entirely satisfactory. Bare "du" will report the actual disk usage so file
systems which can compress the data will report less than the actual space
required. Using "du --apparent-size" will report the actual space used, but as
this simply sums the bytes used for content across an entire file system can
result in significant under-reporting due to block size overhead.
Attempt to solve these problems by implementing our own function to calculate
how large a rootfs will be. This function handles hardlinks correctly but
rounds up all sizes to multiples of the block size (currently, 4KB is the
hard-coded block size).
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
CVE_VERSION_SUFFIX in "patch" to treat version string with suffix "pX"
or "patchX" as patched release.
also update testcases to cover this changes and set CVE_VERSION_SUFFIX
for sudo.
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* not sure if there are some valid use-cases for missing manifest, but
recently I'm seeing increasing number of build failures where something
from native recipe is missing (seen it with pseudo, autoconf, nodejs
recently) and the only indication that something is wrong (before showing
sometimes misleading error like:
recipe-sysroot-native/usr/bin/node: No such file or directory
is this warning:
NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
if I rebuild that native dependency, then it gets fixed and I don't
see these failures in clean builds (as without sstate and with empty
TMPDIR), only in incremental builds
* but if there isn't valid reason for missing manifest file, then I think
it would be better to error early (or even bb.fatal())
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Ensure the fallback value if used is written to the SDE file
and hence stored in sstate, reducing any confusion within the
code over '0' values.
Bump the HASHEQUIV_VERSION since we've had a ton of trouble
with ensuring this rolls out correctly on the autobuilder so
others may too, take a clean slate for it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
GPLv2 and GPLv2+ are two difference licenses with different meanings
and we can't just pretend they're the same thing. Change the code
to treat them separately.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
`devtool` uses `copy_recipe_files` for the upgrade operation when
creating the new, workspace recipe. Before handling the copy operations,
the function checks the entry in `SRC_URI` against `FILE` while in turn
uses absolute paths. When BBLAYERS contains entries that are not
normalised, this check will fail resulting in having the recipe in the
workspace without the initial patches.
Signef-off-by: Robert Drab <robert.drab@huawei.com>
Signed-off-by: Andrei Gherzan <andrei.gherzan@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When building SDKs with the deb backend you could see errors like:
Setting up nativesdk-python3-ndg-httpsclient (0.5.1-r0) ...
mkdir: cannot create directory ‘/usr/lib/opkg’: Permission denied
dpkg: error processing package nativesdk-python3-ndg-httpsclient (--configure):
which is due to environment misconfiguration when removing packages.
Fix this by setting the same environment variables as used for installation.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Daniel McGregor
Ross Burton
Andres Beltran
Mingli Yu
Joshua Watt
Saul Wold
Alexander Kanavin
Robert P. J. Day
Scott Murray
Hsia-Jun Li
Vyacheslav Yurkov
Damian Wrobel
Fabio Berton
Armin Kuster
Martin Jansa
Devendra Tewari
Christophe Chapuis
Paul Barker
hongxu
Peter Budny
Mark Hatle
Awais Belal
Ross Burton
Lee Chee Yang
Andrei Gherzan