| Age | Commit message (Collapse) | Author |
|---|
|
Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This
variable is redundant now that BB_HASHSERVE is present.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION,
adapt to this cleanup. This is an API breaking change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we write to the sstate directory we try to do so atomically so
consumers either see one version or another but never an imcomplete file.
Unfortunately this is reliant on filesystem support and with some NFS
configurations a replaced file would be lost from memory even if users held
open descriptors.
It makes sense to try and avoid replacing existing files where we can.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This change ensures that the task signatures changes, and therefore
sstate tasks are rerun, when signing is enabled. This has the
positive outcome that if signing is enabled new signed shared state
objects will be produced, rather than just signing shared state
objects for tasks where no work has been performed yet.
The downside of this change is that enabling/disabling sstate object
signing alters the taskhash and results in rebuilding the world.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When signature verification of shared state objects is enabled
(SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a
warning, seems unexpected. Instead skip unsigned objects and force the
non-accelerated task to be run.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Referring to the sstate object as a staging package is an artefact of the
code's origins. Switch to referring to an "Sstate package" in order to be more
accurate and consistent with the rest of the file.
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After the recent changes in bitbake to runqueue, we need to recheck sstate validity,
particularly in multiconfig builds where tasks have the same checksum.
Avoid printing summary messages in this case. Also avoid multiple events to toaster
which may not be expecting that at later points in the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Make it consistent with bitbake
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild,
so if there's a sstate-cache available with "bad" sstate data in it that will still
be used even though the recipe is updated to address this.
[YOCTO #13144]
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Updates the output hash calculation for determining if tasks are
equivalent. The new algorithm does the following based on feedback:
1) The output hash function was moved to the OE library.
2) All files are printed in a single line tabular format
3) Prints the file type and mode in a user-friendly ls-like format
4) Includes the file owner and group (by name, not ID). These are only
included if the task is run under pseudo since that is the only time
they can be consistently determined.
5) File size is included for regular files
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There might be do_cleansstate errors sometimes:
ERROR: When reparsing
/path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate,
the basehash value changed from
b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to
c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata
is not deterministic and this needs to be fixed.
The stable reproducer is:
- Initial a fresh build, this is a must, otherwise we may can't reproduce it
$ . oe-init-build-env build
$ bitbake quilt-native -ccleansstate
This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g.,
ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as
SSTATE_EXTRAPATH did can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A couple have still been missed in the past despite multiple
attempts at doing so (or simply have re-appeared?).
Search & replace made using the following command:
sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
-i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
| cut -d':' -f1 \
| sort -u)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Converts sstate so that it can use a hash equivalence server to
determine if a task really needs to be rebuilt, or if it can be restored
from a different (equivalent) sstate object.
The unique hashes are cached persistently using persist_data. This has
a number of advantages:
1) Unique hashes can be cached between invocations of bitbake to
prevent needing to contact the server every time (which is slow)
2) The value of each tasks unique hash can easily be synchronized
between different threads, which will be useful if bitbake is
updated to do on the fly task re-hashing.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Handles the argument that passes task unique hash in the hash check
function, as it is now required by bitbake
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sstate bbclass uses workdir as a hardcoded string in path
manipulations. This means that the sstate caching mechanism does
not work for the work-shared directory which the kernel uses to
share its build configuration and source files for out of tree
kernel modules.
This commit modifies the path manipulation mechanism to use the
work-shared directory if detected in the paths when handling the
sstate cache packages.
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* in some rare cases the file format might be broken, e.g. when you run
out of disk space while writing to it (hint: make sure to use
BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop
gracefully before this happens).
* will show error like this:
ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch'
when the file isn't complete, like in my case:
$ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch
/OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0
/OE/build/luneos-thud/webos-ports
* instead of much longer exception which doesn't really show what's
wrong to selectively fix that (other than removing while TMPDIR):
ERROR: Command execution failed: Traceback (most recent call last):
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand
self.cooker.updateCache()
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache
bb.event.fire(event, self.databuilder.mcdata[mc])
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire
fire_class_handlers(event, d)
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler
ret = handler(event)
File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2
(stamp, manifest, workdir) = l.split()
ValueError: not enough values to unpack (expected 3, got 1)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk>
-ccleansstate" when there are a lot of sstate files.
For example:
* Before
$ bitbake quilt-native -ccleansstate
- Check log.do_cleansstate:
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz*
Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
There are no package tasks for quilt-native, so the first 4 lines doesn't
make any sense, but the glob pattern "sstate-cache/*/*" is very time
consuming when there are no disk caches. E.g., I have more than 600,000
sstate files:
- Without disk caches
# echo 3 >/proc/sys/vm/drop_caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s
- With disk caches (e.g., run it in the second time)
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 0m5.128s
user 0m2.772s
sys 0m2.308s
So the 4 removing *package* commands cost more than 20s or 272s in theory.
* After
$ bitbake quilt-native -ccleansstate
- Check log.do_cleansstate:
Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
We can see that it saved 20s or 272s in theory.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The glob.glob("/sstate/*/*/") is very time consuming, set
SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time
when there are many sstate files.
For example, I have more than 600,000 sstate files:
* Before
- Without disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s
- With disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 0m4.111s
user 0m2.348s
sys 0m1.756s
* After
$ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
- Without disk caches:
real 0m7.928s
user 0m0.172s
sys 0m0.124s
- With disk caches:
real 0m0.131s
user 0m0.088s
sys 0m0.044s
We can see that it saves about 3.8s with disk caches, and saves about 264s
without disk caches.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update SSTATE_DUPWHITELIST in sstate.bbclass.
* remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more
* add directories for package target-sdk-provides-dummy
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass,
then it could be overwritten by user configuration.
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
intel-microcode multilib recipes can generate identical overlapping
files: microcode.cpio.
Signed-off-by: Yongxin Liu <yongxin.liu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using re.compile() is around six times faster than recompiling the regexp
each time so maintain a cache.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
base-passwd|shadow-sysroot recipes
Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot.
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the *-initial recipes to be excluded from a recipe sysroot.
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, a dependency on any -native recipe will pull in
all dependencies of that -native recipe in the recipe
sysroot. This behaviour might not always be wanted, e.g.
when that -native recipe depends on build-tools that are
not relevant for the current recipe.
This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
which will be evaluated for such recursive dependencies to
be excluded. The idea is similar to
http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html
except that the list is not hard-coded anymore.
SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
expressions of recipe and dependency to ignore, e.g. in
the above flex-native / bison-native use-case, one would
specify
SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"
in layer.conf.
The existing special handling of "-initial" as well as
"base-passwd" and "shadow-sysroot" could also be
streamlined:
SSTATE_EXCLUDEDEPS_SYSROOT += "\
.*->.*-initial.* \
.*(base-passwd|shadow-sysroot)->.* \
"
Another anticipated user is meta-java, where certain newer
JDKs can only be bootstrapped (built) using older JDKs,
but it doesn't make much sense to copy all those older
JDKs and their own build tools (ant, etc.) into the
sysroot of recipes wanting to be built using the newer JDK
(only), e.g.:
SSTATE_EXCLUDEDEPS_SYSROOT += "\
openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \
"
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Files when restored from sstate were missing their sparseness. Fix up various
functions to preserve this and make things more deterministic.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if you build qemux86 and then generic86, the latter will
remove all of the former from deploy and workdir. This is because
qemux86 is i586, genericx86 is i686 and the architctures are compatible
therefore the sstate 'cleaup' code kicks in.
There was a valid reason for this to ensure i586 packages didn't get into
an i686 rootfs for example. With the rootfs creation being filtered now, this
is no longer necessary.
Instead, save out a list of stamps which a give machine has ever seen in
a given build and only clean up these things if they're no longer
"reachable".
In particular this means the autobuilder should no longer spend a load of time
deleting files when switching MACHINE, improving build times.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Replace the generic whitelist entry with entries for the three specific
'problem' cases in OE-Core. This means the general DEPLOY_DIR_IMAGE entry
doesn't mask problems for others as was recently encoutered by users
reported on irc. In the whitelisted cases they occur only in multilib builds
and the files are identical.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the user has no indication of how much sstate was already present
or that would be used by the build. This change adds some summary information
so that the user can see how much reuse is occurring. To fully work it
needs some extra information from a recent bitbake commit but this is
optional.
When combined with bitbake --dry-run this feature can be used to check
if sstate would be reused in a build.
[YOCTO #12749]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the compression of sstate objects is single threaded. In the case
of ltp, this takes around 33s. If we add pigz into the list of non-fatal
HOSTTOOLS and then use if it available when building the sstate object,
this time drops to around 6s.
Since pigz is now widely available this is an optimisation we should
utilise.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Removes an unused argument to pstaging_fetch()
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The SSTATECLEANFUNCS mechanism was introduced to solve user/group
deletion problem. After RSS mechanism was introduced, there's no
need to do so.
There was a patch to remove these obsolete codes for useradd.bbclass,
but the codes in sstate.bbclass were not removed. So clean it up.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This makes sure files provided by packages that use any of the extra
architectures defined using ${PACKAGE_EXTRA_ARCHS} are cleaned from
tmp/sysroot-components when sstate_eventhandler2() executes.
Without this, changing a package from using one of the extra
architectures to another architecture would lead to files being
leftover in tmp/sysroot-components, which could later be picked up
when creating the RSS for another package rather than the files that
belonged to the updated package.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It only substituted staging_target for target recipe which didn't work
for multilib, for example, postinst-useradd-lib32-polkit:
* No multilib:
PATH=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot-native/bin
staging_target=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot
The PATH would be substituted to:
FIXMESTAGINGDIRTARGET-native/bin
Not the funny "-native/bin", this works well.
* When multilib:
PATH=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/recipe-sysroot-native/bin
staging_target=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/lib32-recipe-sysroot
Now staging_target endswith "/lib32-recipe-sysroot", so it can't
replace '/recipe-sysroot-native' in PATH , so PATH can't be fixed, and
there would be build errors when building multilib + rm_work, for
example:
chown: invalid user: ‘polkitd:root’
Substitute staging_host for target recipe can fix the problem, now all
of native, cross and target need substitute staging_host, so we can
simply the code a little.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we migrated rpm v5 -> v4, we lost the ability to drop "per file"
dependencies from the rpm backend for things like "/bin/bash" and
"/usr/bin/env" which meant the sdks were becomming 'bloated'.
This restores the functionality using a dummy package, similarly to
the way the buildtools perl issue was addressed. It also removes
the non-functional old code so as not to confuse people in future.
I ran into this problem trying to filter dependencies to only rpms
a build directly depends upon and it turns out we have some determinism issues
in this area so this is something key to fix.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If there are many recipes that should be cleaned out from the
sysroots, it can actually take some time. This adds a progress bar to
give a visual clue that the clean up is actually progressing.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Packages, which depend on users/groups created from other packages,
needs "shadow-native" as a build time dependency. So, add an exception
to the "shadow-native" from otherwise discarded native/cross tools
dependency.
Fixes [YOCTO #11960]
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Since it was first implemented, devtool's source extraction (as used by
the devtool modify, extract and upgrade subcommands) ignored other recipe
dependencies - so for example if you ran devtool modify on a recipe that
fetches from svn or is compressed using xz then it would fail if those
dependencies hadn't been built first. Now that we can execute tasks in
the normal way (i.e. tinfoil.build_targets()) then we can rework it to
use that. This is slightly tricky in that the source extraction needs to
insert some logic in between tasks; luckily we can use a helper class
that conditionally adds prefuncs to make that possible.
Some side-effects / aspects of this change worth noting:
* Operations are a little slower because we have to go through the task
dependency graph generation and other startup processing. There's not
really any way to avoid this though.
* devtool extract didn't used to require a workspace, now it does
because it needs to create a temporary bbappend for the recipe. (As
with other commands the workspace be created on the fly if it doesn't
already exist.)
* I want any existing sysroot files and stamps to be left alone during
extraction since we are running the tasks off to the side, and
especially devtool extract should be able to be used without touching
these. However, this was hampered by the automatic removal process in
sstate.bbclass triggered by bb.event.ReachableStamps when the task
signatures change, thus I had to introduce a way to disable this
removal on a per-recipe basis (we still want it to function for any
dependencies that we aren't working on). To implement this I elected
to use a file written to tmp/sstate-control which gets deleted
automatically after reading so that there's less chance of stale files
affecting future sessions. I could have used a variable but this would
have needed to be whitelisted and I'd have to have poked its value in
using the setVariable command.
Fixes [YOCTO #11198].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Currently, dummy archives are created for skipped sstate tasks. We relied
on these never being installed (the setscene task is missing) however this
leads to odd behaviour as for example the setscene stamps are never looked
at.
Rather than trying to continue with the two separate behaviours, really
skip package creation. We do need the file manipulation steps to install files
under sstate control from a manifest perspective so we only skip at the final
creation step.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python function subprocess.call() returns the return value of the
executed process. If return values are not checked, errors may
go unnoticed and bad things can happen.
Change all callers of subprocess.call() which do not check for
the return value to use subprocess.check_call() which raises
CalledProcessError if the subprocess returns with non-zero value.
https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module
All users of the function were found with:
$ git grep "subprocess\.call" | \
egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call'
Tested similar patch on top of yocto jethro. Only compile tested
core-image-minimal on poky master branch.
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The previous change to relocate HOSTTOOLS wasn't complete as some files,
particularly in gcc stashed build directories were not being correctly
relocated. This patch addresses the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Paths to host tools that have been copied to ${HOSTTOOLS_DIR} may end
up in the sstate cache. They thus need to be corrected when restoring
from the sstate cache.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently there are multiple issues with useradd:
* If base-passwd rebuilds, it wipes out recipe specific user/group additions
to sysroots and causes errors
* If recipe A adds a user and recipe B depends on A, it can't see any of the
users/groups A adds.
This patch changes base-passwd so it always works as a postinst script
within the sysroot and copies in the master files, then runs any
postinst-useradd-* scripts afterwards to add additional user/groups.
The postinst-useradd-* scripts are tweaked so that if /etc/passwd doesn't exist
they just exit, knowning they'll be executed later. We also add a dummy entry to
the dummy passwd file from pseudo so we can avoid this too.
There is a problem where if recipe A adds a user and recipe B depends on A but
doesn't care about users, it may not have a dependency on the useradd/groupadd
tools which would therefore not be available in B's sysroot. We therefore also
tweak postinst-useradd-* scripts so that if the tools aren't present we simply
don't add users. If you need the users, you add a dependency on the tools in the
recipe and they'll be added.
We add postinst-* to SSTATE_SCAN_FILES since almost any postinst script of this
kind is going to need relocation help.
We also ensure that the postinst-useradd script is written into the sstate
object as the current script was only being added in a recipe local way.
Thanks to Peter Kjellerstedt <pkj@axis.com> and Patrick Ohly for some pieces
of this patch.
[Yocto #11124]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Commit 51edde653707e7a3cd2186082458f01f32cd1996 makes a wrong assumption
that SSTATE_MIRRORS have write permissions.
A mirror is by definition outside of it's user control. In my use case
it happens I does not have permissions to update the access time of the
dereferenced symbolic-link file.
Checked if file is writable before changing its atime.
Thanks to Paulo Neves for the patch.
[YOCTO #11307]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We never need these tasks as dependencies of other sstate tasks since
they're only ever needed to build artefacts so we can always skip them
and save some time/space.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When you enable the systemd DISTRO_FEATURE, opkg-native contains systemd units
which have a relocation fixme list. When systemd isn't in DISTRO_FEATURES, there
are no fixmes required. Unfortunately as sstate isn't cleaning up its installation
directory before use, if you install the systemd version, then install the
non-systemd version from sstate, it would leave behind the fixme file from the
systemd version and breakage results as it would try and fixup files which don't
exist.
The solution is to ensure the unpack/install directory is clean before use. It
does raise other questions about opkg-native, systemd and DISTRO_FEATURES but there
is an underlying sstate issue here too which would cause missing file failures.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
instead of "all"
Too many places in dnf/rpm4 stack make that assumption; let's not fight against it.
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
|
|
.siginfo files are not being accessed from local or NFS-mounted
sstate mirrors when sstate package is installed, so their atime
is not updated. If sstate mirror is cleaned based on access time,
they get deleted, even though they are still being used.
Updated atime of .siginfo symlinks with 'touch -a'. This command
dereferences symlinks pointing to the local mirror and updates
atime of the .siginfo file on the mirror.
[YOCTO #10857]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
It isn't clear that the README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
file in the deploy directory warrants the complexity it brings elsewhere.
Let's just remove it entirely.
In particular, if two do_image_complete tasks run in parallel they risk
both trying to put their image into ${DEPLOY_DIR_IMAGE} at the same time.
Both will contain a README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
file. In theory this should be safe because "cp -alf" will just cause one
to overwrite the other. Unfortunately, coreutils cp also has a race[1]
which means that if one copy creates the file at just the wrong point the
other will fail with:
cp: cannot create hard link ‘..../tmp-glibc/deploy/images/pantera/README_-_DO_NOT_DELETE_FILES_IN_THIS_D.txt’ to
+‘..../tmp-glibc/work/rage_against-oe-linux-gnueabi/my-own-image/1.0-r0/deploy-my-own-image-complete/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt’: File exists
[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25680
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
Joshua Watt
Richard Purdie
Joshua Lock via Openembedded-core
Robert Yang
Alexander Kanavin
André Draszik
Michael Ho
Martin Jansa
Kai Kang
Yongxin Liu
Chen Qi
Peter Kjellerstedt
Maxin B. John
Paul Eggleton
Mikko Rapeli
Ed Bartosh
Alexander Kanavin
Mike Crowe