| Age | Commit message (Collapse) | Author |
|---|
|
* without this patch:
Python 2.7.3
>>> sstate_search_cmd = "grep -rl /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime-reverse/vim-common /OE/jansa-test/shr-core/tmp-eglibc/sstate-control --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'"
>>> cmd_array = sstate_search_cmd.split(' ')
>>> search_output = subprocess.Popen(cmd_array, stdout=subprocess.PIPE).communicate()[0]
grep: |: No such file or directory
grep: sed: No such file or directory
* Adding shell=True and using cmd string instead of array makes it work:
>>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
>>> print search_output
manifest-armv7a-vfp-neon-gvim.package
manifest-armv7a-vfp-neon-vim-tiny.package
manifest-armv7a-vfp-neon-vim.package
But still isn't 100% reliable, I guess it's caused by some other package
being removed from sstate while grep is already running.
So sometimes grep can show error on STDERR
>>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
grep: /OE/jansa-test/shr-core/tmp-eglibc/sstate-control/manifest-armv7a-vfp-neon-systemtap.package: No such file or directory
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The value of subprocess.Popen().communicate()[0] is a string.
Checking for '!= None' will always match causing bogus warnings
regarding already staged files.
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently there is some odd behaviour of the packagegroup class in relation
to sstate since it sets PACKAGE_ARCH = "all" but does not use the allarch class
leading to it being undetected by sstate.
Previously it was not possible to use allarch as the recipe couldn't "undo"
settings made by the allarch class. Since this no longer happens when
PACKAGE_ARCH != all, we can use the allarch class.
This patch also fixes up one case we need to preserve TRANSLATED_TARGET_ARCH
and ensures sstate only assumes allarch when PACKAGE_ARCH is "all".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Clean up and clarify the populate_sysroot task dependencies. Target sysroot
packages do need their dependencies installed, as do some target/cross
relationships. We can whitelist the *-initial dependencies as these are
never needed indirectly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For example gcc-cross depends on linux-libc-headers and needs it to be present
to build/work correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
of some sstate installation
This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the
side of caution, as it gets wider testing, we can refine the logic as needed.
This code should allow a significant performance speedup to certain workflows, for
example "bitbake xxx-image -c rootfs" will not populate the target sysroot.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The recent preveeding sstate directory layout fixes made the code do what it
was originally intended to do, as can be clearly seen from the code.
Unfortunately this changed the contents and layout of the sstate files
themselves since the bug was leading to a directory prefix being missing.
This is now resulting in chaotic messages on the console since things
are getting confused with the two different layouts. The simplest way to
resolve this is to bump the version number, hence moving the new layout
into its own new namespace.
Its worth noting that whilst the failure messages are scary, the failure
mode is relatively harmless since it will just fall back to building the
data rather than installing from sstate.
Usually I'd give more notice of a change like this but under the
circumstances, I'm just going to push this in to resolve the failures
people are seeing. Initially I thought the problem was limited to
some of the -cross packages and therefore of low impact but that is
clearly not the case.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix missing parameter to endswith and pass paths through normpath to remove
any duplicate "/" characters which would corrupt other calls like basename.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The manifest file can become corrupted if sstate-inputdirs and sstate-outputdirs
don't have matching endings. This patch ensures that even if set incorrectly,
the code functions as intended, thereby handling manifest corruption safely.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Present the manifest file that contains the matches for
files being installed to a location that already contains
that file. This will help to determine which is the correct
recipe to fix when this occurs.
[YOCTO #3191]
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
Now do_package isn't machine specific, we're only left with do_populate_sysroot as a
machine specific task. This change marks only the machine specific manifests as machine
specific, defaulting to PACKAGE_ARCH for everything else.
This means we do less work where there are multiple machines using the same
core package architecture and we can start to clean up the sstate duplicate files
whitelist.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've noticed failures on the project autobuilders where a shared sstate
directory is used across multiple builders and the clocks become skewed.
Most of the time this causes harmless building but if this happens where
an environment is changed (make install vs make in qt4-x11-free for example),
the build can fail.
This avoids modification times in the future and should make builds safer
in shared environments sstate was designed for.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids errors where gcc/binutils get installed to the native sysroot
in the same location for multiple package architectures. Ultimately making
these native recipes with ${PACKAGE_ARCH} appended to PN will resolve this
but hide the warnings until this gets sorted out.
Also hide the python and docbook catalog warnings since they're known about,
nothing to worry about and we'll aim to clean them up properly in the 1.4 cycle.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this, path components like // could break comparisions with the whitelist leading
to warnings being displayed to the user unintentionally.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
do_package is a machine specific task at the moment due to packagedata. This means
do_package tasks and their dependencies rerun between different machines
with various duplicate file installations. There are plans to fix this but they're
too invasive before release.
This patch relaxes the whitelist for sstate duplicate file detection to account
for this. Post-release, we re-enable stricter settings once do_package is not
machine specific.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
This was added to allow detection of duplicate files being installed by sstate.
There is a much simpler way, just check if the file already exists. This
effectively uses the kernel VFS as the cache which is much more efficient.
This resolves a significant performance bottleneck (lock contention on a
single file) when running builds that are just being generated from sstate
cache files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to split this variable before using it. Otherwise a single "/"
character in the list whitelists every overlapping sysroot file which
was not the intention making the whole thing useless.
We'll start seeing warnings about overlapping files now this is working
correctly after this patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Appending to EXTRA_SSTATEMAPS is better than just hardcoding a value. Also
add a comment about why this is necessary.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I've not idea why this got left in but as per the comment, it needs fixing
and we shouldn't have hardcoded mappings like this. Lets remove it
and dynamically generate the data instead.
[YOCTO #3039]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes builds where master.list doesn't already exist. This
change was meant to be part of the previous sstate commit but ended
up separated.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have a potential problem where two sstate packages try and touch the same
file. This adds code which will print a warning whenever this happens.
The implementation does but by maintaining a master file list and comparing
file accesses against this. There are a number of places we have duplicate
accesses which are harmless, mostly in the deploy directory so these
are whitelisted.
For now the code prints warnings, this could be strengthened in future to
become error messages. Whilst working on this code on and off over the past
few months various issues were uncovered, some serious.
[YOCTO #238]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was pointed out we have a number of weird indentations in the python functions.
This patch cleans up 3, 7 and other weird indentations for the core bbclass files.
It also fixes some wierd (odd) shell function indentation which my searches picked up.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds the lsb name and revision to the path used for sstate files.
This means that reuse of sstate files between different distributions is restricted
by default. The behaviour can be configured using mirror urls, for example:
SSTATE_MIRRORS = "file://Ubuntu-11.10/(.*) file://Ubuntu/\1 \n"
would map Ubuntu 11.10 to a more generic "Ubuntu" named sstate feed.
Usually, more modern distros have increased libc versions for example
so whilst more older native/cross sstate packages will usually work on newer
distros, the opposite is not true. This patch allows development of policy
to better handle this although no default policy is currently being used.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently all sstate files are placed into one directory. This does not scale and
causes a variety of filesystem issues. This patch adds a two character subdirectory
to the layout (based on the first two characters of the hash) so that files
can be split into several directories.
This should help performance of sstate in most cases.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
allarch sstate packages could be marked as machine or package_arch specific. This
change ensures they are not.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There was a bug showing up where the crosssdk recipes were being installed as machine
specific in the sstate-control directory. This turned out to be due to the architecture
fields used by sstate being set incorrectly. The problem is that the crosssdk inherits
the cross class. It therefore needs to be listed in this if statement block before
the cross check, not after.
This should resolve some package architecture issue of crosssdk sstate files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
Replace os.system with subprocess.call since the older function would
fail (more or less) silently if the executed program cannot be found
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2454]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Jason Wessel noticed that a package without any fixmepath entries would
generate a sed warning about no input files. This patch resolves that
by ensuring that an empty fixmepath file never gets written into the
sstate archive. Also we avoid a second message by only doing xargs if
we got input.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In order to make sharing sstate files easier chmod them with 0664 permissions so that they are readable and writable by the user and any other members of their group.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
|
|
The fixmepath file that is generated contains a list of all of the files
that need their paths fixed. In the previous version the fixmepath was
generated to include all of the files that sed may have changed. In the
new version, we first grep the files to see if they contain a path that
needs to be changed, only then do we perform the sed operation on those files.
This results in a modest performance increate in the creation of the sstate
file. The following numbers include the do_package and do_populate_sysroot
tasks on the perl recipe.
Before the change:
real 4m23.018s
user 1m57.067s
sys 1m33.327s
After the change:
real 4m13.083s
user 1m54.062s
sys 1m26.064s
However, a more significnt performance gain is felt during the
extraction/install of sstate cache files, as the fixmepaths file now has a
significantly smaller list of files to modify.
Before the change:
real 0m39.798s
user 0m11.158s
sys 0m12.642s
After the change:
real 0m25.511s
user 0m8.408s
sys 0m5.077s
(All numbers above were recorded with a cold filesystem cache on a machine
with 12 GB of ram.)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In a pathological case, lots of files to process, the sstate_installpkg
performance was very poor. It interated over each file and ran 3
individual sed commands per file. Changing this to keep iterating
but running only a single command took about 1/3 time time.
However, when looking at the corresponding sstate_hardcode_path
function, it was clear we could optimize this further.
Using the same encoding logic to specify only the minimumal sed
operation necessary, and using xargs to avoid the os.system call the
install step was able to be performed in 13% of the original time.
Example timing numbers for perl:
3m7s original code
1m20s single sed, but interating
0m26s using xargs and limited sed
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
files to avoid issues"
Reluctantly revert this since it breaks the tar-native workaround we have
for old versions of tar :(
This reverts commit 01218e29f963120c77885964702198d9bce77e11.
|
|
writing atomic
There is currently a race window when creating sstate packages since we don't
atomically write the files to SSTATE_DIR. This change ensures we do so by writing
to a temporary file and then doing an atomic move.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
avoid issues
This resolves issues related to pigz-native when installing from sstate that people
have been seeing. It also gives us a way to solve issues like the gzip-native race
during sstate package creation covered in Yocto #1774.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
sed \
-e 's:bb.data.\(expand([^,()]*\), *\([^) ]*\) *):\2.\1):g' \
-i `grep -ril bb.data.expand *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* also fixes replacing paths for perl where cmd line was probably
too long for os.system(cmd) (it had 560410 characters because a lot of
files from sstate_scan_cmd).
* also print those 2 commands so we can find them in log.do_package
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
* e.g. mysql5 has mysql_config not mysql-config
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sstate_hardcode_path() function triggered large numbers of exec()
calls when processing packages with large numbers of file relocations
(e.g. perl). This patch optimises those calls into longer single commands
which make the code significantly more efficient.
This reduced the do_package time for perl by 2 minutes (from 4.75 minutes)
for me.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We process all files in the native/cross cases for finding and
fixing relocation issues. In the target case we've only processed
.la and binconfig files. Since there are other files which are
in need of this processing, this change allows recipes to specify
files that may be outside the normal set. This means hardcoded
paths that need to be fixmepathed to work correctly are handled
and addresses some sstate relocation bugs that have been seen.
Based on a patch from Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Complete the bb.data.getVar/setVar replacements with accesses
directly to the data store object.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this change we can end up looking for <stamp>.${MACHINE}
instead of the expected expanded value.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of running the following over the metadata:
sed \
-e 's:bb.data.\(setVar([^,()]*,[^,()]*\), *\([^ )]*\) *):\2.\1):g' \
-e 's:bb.data.\(setVarFlag([^,()]*,[^,()]*,[^,()]*\), *\([^) ]*\) *):\2.\1):g' \
-e 's:bb.data.\(getVar([^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \
-e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \
-e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^() ]*\) *):\2.\1):g' \
-e 's:bb.data.\(getVar([^,()]*\), *\([^) ]*\) *):\2.\1):g' \
-i `grep -ril bb.data *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
current task
sstate was being a little too ethusiastic about removing stamp files and
was removing stamp files for other machines when it shouldn't have been.
This patch teaches sstate about machine specific stamp extensions and
allows it to only remove the current task's stampfiles.
Based on a patch from Phil Blundell <philb@gnu.org> with some tweaks
from me.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Martin Jansa
Enrico Scholz
Richard Purdie
Saul Wold
Ross Burton
Robert Yang
Mark Hatle
Joshua Lock