| Age | Commit message (Collapse) | Author |
|---|
|
The PGO (Profile-guided Optimization) collect data about the typical execution of a program
and then use this data to inform optimizations such as inlining, machine-code layout,
register allocation, etc.
This optimization is by default disabled in rust sources but enabled in Yocto and causing
the reproducibility issue in rustdoc binary. To fix the issue this optimization is set to
it's default 'false'.
More about the optimization: https://doc.rust-lang.org/rustc/profile-guided-optimization.html
With the reproducibility issue fixed, we can enable the reproducibility tests again.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The 'rustdoc' is moved to a separate 'rust-rustdoc' package. This is a
workaround to test if the main rust binary is reproducible even if
rustdoc isn't.
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES can be used to prevent known
not-reproducible packages to make the reproducible test fail.
For example, in local.conf:
OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES = "example-not-reproducible"
To quote the original exclusion commit (4864764667097b32a3ee1935e2c52df732d55632)
> selftest/reproducible: add an exclusion list for items that are not yet reproducible
> [...]
> Non-reproducible excluded packages are not given to diffoscope and do not cause a
> failure, but still saved side-by-side with non-reproducible failing ones to make
> investigation easier.
NB: Patch was written by Richard with small fixes and a commit message
from Yoann.
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
No other change.
Just cosmetic to avoid tripping the maximum line length of patchtest
when the next element will be added to the list.
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Unfortunately there is still an issue with rustdoc not being reproducible
so part of the problem has been fixed but not everything.
Add the exclusion back until this has been addressed to avoid autobuilder
failures and long diffoscope analysis time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The '--remap-path-prefix' option removes all references to build directory
structure in the debug information within the compiled output for Cargo
dependencies and the project's binary.
However, some references to build directories remains in the final binary
in .rustc section in the form of compressed metadata and this makes the
build output dependent on the folder structure of the computer it's compiled on.
So, for reproducible builds, use the configuration option
'remap-debuginfo = true' along with the '--remap-path-prefix'.
[YOCTO# 14875]
Signed-off-by: Sundeep KOKKONDA <sundeep.kokkonda@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the bb_get_var calls trigger multiple "bitbake -e" executions
which slow the start of oe-selftest for any test. Rework the code to avoid
these delays.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We can't have systemd here any longer without usrmerge. We don't really
want to enable the latter since having separate usr will likely result in
a class of reproducibility and host contamination issues that enabling it
might hide.
Also drop INHIBIT_PACKAGE_STRIP since we generally don't build with that
and the debug binaries should be generated regardless. I suspect this
is legacy from older issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a follow-up of 76e5fcb2 that also allow users to chose
the package manager using OEQA_REPRODUCIBLE_TEST_PACKAGE
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow users to set different build reproducibility targets than
the defaults using OEQA_REPRODUCIBLE_TEST_TARGET and
OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS variables in local.conf.
Fixing all issues from "world" builds is not possible in some
complex build environments with lots of layers. Limiting the focus to
a smaller subset allows using this test to detect and fix build
reproduction issues incrementally.
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
We don't compare reproducibility of the native/cross components, only the
target ones. With the long build times of rust-native, the test now takes
crazy lengths of time so this tweak should allow us to reuse native/cross
artefacts from sstate whilst still testing the target output is
reproducible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
output
When we provide sstate_targets recipes they are reported as missing
which is not quite correct because they are proveided to cut build/debugging time.
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
There looks to be a reproducibility issue left in one of the rust
libraries. It doesn't appear to be a string issue but some binary
problem. Disable rust from the reproducibility testing until we can
get to the bottom of the issue (allowing wider testing of all the
other improvements).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The issue is reported upstream:
https://bugs.ruby-lang.org/issues/18456
Otherwise I do not feel that further investigation of rdoc's parser to find
out the source of non-determinism is worth the time.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Diffoscope can end up running for a very long time if there are a lot of
changes. To put a limit on how long it can run, cap the maximum report
size at 250 MB by default.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With this changeset, go becomes reproducible \0/
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reproducibility is here to stay and needs to be part of our default workflow.
Move the remaining code to base.bbclass so it is always a first class citizen
and it is clear people need to be mindful of it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
rust-llvm-liblto and rust-llvm-staticdev sometimes vary in contents. Exclude
them from the test for now until we can work on and resolve the issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The hardcoded path issue was fixed, drop the exclusion.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The meson determinism issue is the same frozenset issue we encountered with
python itself. Remove the problematic pyc file until upstream work out the
best way to address the issue (as with core python).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
go-dep was an effort for dependency management before go modules, which
since 2020 has been deprecated in favor of go modules. Since its not
developed any longer and go mdules is officially supported, this should
be retired from OE-core as well.
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Otavio Salvador <otavio.salvador@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have fixes for perf reproducibility, so we can drop it from the
exclusion list.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are build paths which are in generated sources injected
into ${PN}-src. These are in generated files and don't influence the
binary output. As such we can correct them at do_package time by
tweaking in an appropriately injected function. This fixes
the reproducbility of the recipe.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a patch adding sorting to a couple of points in the Makefiles
which removes most of the determinism issues in ltp.
Build swapon before the main build to ensure libswapon.o is built
deterministically as it races with swapoff.
All issues reported on the upstream mailing list.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This change adds core-image-weston to the reproducible build test.
For this we also need to update the static-group and static-passwd.
Besides the world package build which includes the wayland package,
this adds a wayland/weston-based image to the reproducible build test.
Signed-off-by: Jan-Simon Moeller <jsmoeller@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a hack to hardcode in specific rpaths which we then remove,
allowing the build to be reproducible.
Strip build patches out of one of the test scripts too.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a patch to address two determinism issues and allow reproducible
builds.
Also strip full paths we don't need out of some ptest files,
we can use the installed binaries.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When debugging why a single recipe doesn't reproduce, its a pain
to wait for the world to rebuild from scratch. Update the selftest
to allow this to be configured, for example you could set
targets as ['perf'] and sstate_targets as ['virtual/kernel']
and then it should only be rebuilding perf in the test rather than
things like the toolchain (parts of the kernel may be unavoiable
as they're not in sstate).
Can be run as:
OEQA_DEBUGGING_SAVED_OUTPUT=/tmp/perf-diffoscope oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
to save diffoscope output.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
LTO likely doesn't buy us much here, disable it to allow the binaries
to be reproducible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add sorting to the globbing within the Makefile to make the output
reproducible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove hardcoding the build configuration into the help/version output
from swig to make the binaries reproducible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add sorting to wildcard expansion in the makefile to make builds
reproducible.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The library hardcodes paths to the python library internally and currently
these are build paths. Fix this to use the correct target path and
fix reproducibility in the process.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to use CFLAGS with the correct WORKDIR in them, replace those
in the sysroot file with the ones appropriate to the current recipe.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Configure was swallowing our cflags meaning the resulting binaries were
not reproducible. Tweak configure not to do that and fix reproducibility.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a configuration option to pass in srcdir, removing hard coded
build paths from the binaries.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The compressed docs contained a timestamp of the original file which
meant the SDE clamping during package creation didn't work. The
benefits of compression are minor, decompress the files to avoid
the reproducibility issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Calling sync between each file compare is horrible performance wise
as we compare thousands of files. We don't care about IO latency here
so disable.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't use tbe BUILDINFO line of host information in the Makefile
so remove it for reproducibility.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't want to encide BUILD_ROOT into target packages. This is used
for build time tests but in our case those would be on target anyway
do use the target paths.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
config_data was including a build system path to pkg-config, fix
this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
configure inspects the host's /etc/group for these configuration
options, fix this to the correct values by using configure options.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Sundeep KOKKONDA
Richard Purdie
Yoann Congal
Jose Quaresma
Mikko Rapeli
Ross Burton
Saul Wold
Alexander Kanavin
Joshua Watt
Khem Raj
Bruce Ashfield
Jan-Simon Moeller