| Age | Commit message (Collapse) | Author |
|---|
|
If an existing source date epoch file was found during do_unpack, it was
deleted and a new one would be written in its place. This causes a race
with check-before-use code in get_source_date_epoch_value. Resolve the
problem by making do_unpack write the new source date epoch to a
temporary file, then do an atomic rename to ensure it's always present,
and change the check-before-use code to use a EAFP exception instead of
checking for file existence.
[YOCTO #14384]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0b5e3b33187bf78a2d62cc886463e4b27d6bd228)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
|
|
Ensure the fallback value if used is written to the SDE file
and hence stored in sstate, reducing any confusion within the
code over '0' values.
Bump the HASHEQUIV_VERSION since we've had a ton of trouble
with ensuring this rolls out correctly on the autobuilder so
others may too, take a clean slate for it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A SOURCE_DATE_EPOCH of 0 might be misinterpreted by namely rpm as no SDE.
e.g.:
char *srcdate = getenv(SOURCE_DATE_EPOCH);
if (srcdate && rpmExpandNumeric(%{?clamp_mtime_to_source_date_epoch}))
Solve this by:
- providing a SOURCE_DATE_EPOCH_FALLBACK variable != 0
- changing defaults to use it
- using SOURCE_DATE_EPOCH_FALLBACK also when reading 0 out of sstate-cache
Signed-off-by: Jan-Simon Möller <jsmoeller@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Moves most of the python code used for dealing with the source date
epoch to library code.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, if an existing TMPDIR is rebuilt, do_fetch/do_unpack can rerun
but SDE would remain unchanged. This leads to different results compared
to a fresh build. An example change which triggered this is:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cb4e69e6346a9fbeebf83a5d5397cacbd41d48b5
Instead, delete any existing SDE and recalculate if we're reunning.
Also rename and drop the do_ prefix since these are for tasks,
not functions.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Defers the resolution of the SOURCE_DATE_EPOCH until the variable needs
to be actually realized with a value. The previous method of loading the
value in anonymous python had issues because it could occur before other
anonymous python functions that affect the location of the epoch file,
such as when a recipe uses AUTOINC/AUTOREV or allarch.bbclass.
Also adds more logging to help diagnose issues in the future.
[YOCTO #13763]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Creates the source data epoch file destination directory when restoring
from sstate
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The deployment of the source date epoch file had a race condition where
any task attempting to read from the file would race with creation of
the sstate archive for the do_deploy_source_date_epoch task. The
creation of the sstate archive requires moving the directory to a
temporary location, then moving it back. This means that the file
disappears for a short period of time, which will cause a failure if any
other task is running and trying to open the file to get the current
source date epoch.
The solution is to copy the source date epoch file to a separate
directory when deploying so the file never disappears. When the file is
restored from sstate, it is moved to the correct location after being
extracted.
[YOCTO #13501]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some packages (eg. init-ifupdown) take their source files entirely from
openembedded-core, that is, they download no source tarball.
These recipes either don't use S at all (ie. it is empty at unpack time),
or they set S = WORKDIR (as in init-ifupdown).
Looking at the file timestamps in the WORKDIR causes a non-reproducible
SOURCE_DATE_EPOCH, as files taken from file:// URIs do not have
reproducible timestamps.
If S == WORKDIR, we are better to assume that there is no source tarball,
and to fall back to a fixed timestamp for the SOURCE_DATE_EPOCH.
This makes the init-ifupdown build reproducible.
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
If none of the existing mechanisms are suitable, replace the do_deploy_source_date_epoch task
with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Log both the source of the source_date_epoch and the resulting
source_date_epoch in all cases.
Also, now that we are determining the source_date_epoch successfully for both
yocto and non-yocto kernels, remove the inherits_class('kernel') exception.
We will log a failure to find a source_date_epoch for kernels as well.
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Unnecessary.
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Change the search regime for find_git_folder():
1. WORKDIR/git: This is the default git fetcher unpack path
2. ${S}
3. Go looking for .git/ under the WORKDIR as a last resort.
linux-yocto:
We had an existing (silent) defect. The linux-yocto recipes all specify
two git SRC_URIs, one for the kernel source itself, the other for the
kmeta data (config fragments and friends). find_git_folder() was finding
the git checkout for the kmeta data, but due to a typo in the git log -1
--pretty=%ct line, we were (silently) reading the source_date_epoch from
the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your
build/ happened to be inside a git checkout, git would walk up the
directory tree, and silently read the commit timestamp from this other
git checkout. The correct path to read the git commit timestamp from is
the "gitpath", being that found by find_git_folder(), though this
function was incorrectly finding the kmeta data checkout, not the kernel
source tree.
Non-kernel git recipes:
The default git fetcher clones and checks out the sources at
WORKDIR/git/ regardless of the setting of S (unless subpath or
destsuffix is set). find_git_folder() now looks for the
WORKDIR/git/.git/ directory first.
Non-yocto linux kernels:
Kernel recipes that don't inherit kernel-yocto should always set
S = ${WORKDIR}/git, so that when base_do_unpack_append() in
kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and
symlinks it as WORKDIR/git, the build can still work by following the
symlink. We were previously failing to follow the symlink in the
os.walk(), but we now look first for WORKDIR/git/.git/, and find it due
to the symlink.
If none of the above mechanisms work for finding the git checkout,
perhaps there was a subpath or destsuffix specified in the SRC_URI.
We go looking for the git checkout under the WORKDIR as a last resort.
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Tidy whitespace, delete commented-out lines
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Using os.path.getmtime() will dereference symbolic links in an attempt
to get the last modified time. This can cause errors if the target
doesn't exist, or worse map to some absolute build host path which would
make a build not reproducible.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Setup environment for builds requiring binary reproducibility.
Determine and export SOURCE_DATE_EPOCH per each recipe.
This is a crucial step to achieve binary reproducibility.
The value for this variable (timestamp) is obtained after source code for
a recipe has been unpacked, but before it is patched. If the code sources
come from a GIT repo, we get the timestamp from the top commit. (GIT repo
does not preserve file mktime timestamps). Otherwise, if GIT repo is not
present, we try to get mtime from known files such as NEWS, ChangeLog, etc.
If this also fails, we go through all files and get the timestamp from the
youngest one. We create an individual timestamp for each recipe.
The timestamp is stored in the file '__source_date_epoch.txt' (in the folder
source-date-epoch_). Later on, each task reads this file and sets
the exported value of SOURCE_DATE_EPOCH to the value found in the file.
Uasge:
INHERIT += "reproducible_build"
[YOCTO#11178]
[YOCTO#11179]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
Joshua Watt
Richard Purdie
Jan-Simon Möller
Douglas Royds
Juro Bystricky