summaryrefslogtreecommitdiffstats
path: root/meta/classes/sstate.bbclass
AgeCommit message (Collapse)Author
2012-12-05sstate.bbclass: fix detection of owners matching sstate filesMartin Jansa
* without this patch: Python 2.7.3 >>> sstate_search_cmd = "grep -rl /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime-reverse/vim-common /OE/jansa-test/shr-core/tmp-eglibc/sstate-control --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" >>> cmd_array = sstate_search_cmd.split(' ') >>> search_output = subprocess.Popen(cmd_array, stdout=subprocess.PIPE).communicate()[0] grep: |: No such file or directory grep: sed: No such file or directory * Adding shell=True and using cmd string instead of array makes it work: >>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] >>> print search_output manifest-armv7a-vfp-neon-gvim.package manifest-armv7a-vfp-neon-vim-tiny.package manifest-armv7a-vfp-neon-vim.package But still isn't 100% reliable, I guess it's caused by some other package being removed from sstate while grep is already running. So sometimes grep can show error on STDERR >>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0] grep: /OE/jansa-test/shr-core/tmp-eglibc/sstate-control/manifest-armv7a-vfp-neon-systemtap.package: No such file or directory Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-12-05sstate.bbclass: fixed file-conflict checkEnrico Scholz
The value of subprocess.Popen().communicate()[0] is a string. Checking for '!= None' will always match causing bogus warnings regarding already staged files. Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-26packagegroup/allarch: Convert to use allarch classRichard Purdie
Currently there is some odd behaviour of the packagegroup class in relation to sstate since it sets PACKAGE_ARCH = "all" but does not use the allarch class leading to it being undetected by sstate. Previously it was not possible to use allarch as the recipe couldn't "undo" settings made by the allarch class. Since this no longer happens when PACKAGE_ARCH != all, we can use the allarch class. This patch also fixes up one case we need to preserve TRANSLATED_TARGET_ARCH and ensures sstate only assumes allarch when PACKAGE_ARCH is "all". Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-24sstate: Explicitly define populate_sysroot task relationshipsRichard Purdie
Clean up and clarify the populate_sysroot task dependencies. Target sysroot packages do need their dependencies installed, as do some target/cross relationships. We can whitelist the *-initial dependencies as these are never needed indirectly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-24sstate: Add a rule for target sysroot requirements from cross dependenciesRichard Purdie
For example gcc-cross depends on linux-libc-headers and needs it to be present to build/work correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-19sstate: Implement a setscene dependency validation routine to allow skipping ↵Richard Purdie
of some sstate installation This is a first attempt at logic to determine when a sstate dependency needs to be installed and when it does not. Its a start at the logic and errs on the side of caution, as it gets wider testing, we can refine the logic as needed. This code should allow a significant performance speedup to certain workflows, for example "bitbake xxx-image -c rootfs" will not populate the target sysroot. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-18sstate: Drop now unneeded python whitelist entriesRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-14sstate: Bump version number to deal with layout fixesRichard Purdie
The recent preveeding sstate directory layout fixes made the code do what it was originally intended to do, as can be clearly seen from the code. Unfortunately this changed the contents and layout of the sstate files themselves since the bug was leading to a directory prefix being missing. This is now resulting in chaotic messages on the console since things are getting confused with the two different layouts. The simplest way to resolve this is to bump the version number, hence moving the new layout into its own new namespace. Its worth noting that whilst the failure messages are scary, the failure mode is relatively harmless since it will just fall back to building the data rather than installing from sstate. Usually I'd give more notice of a change like this but under the circumstances, I'm just going to push this in to resolve the failures people are seeing. Initially I thought the problem was limited to some of the -cross packages and therefore of low impact but that is clearly not the case. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-14sstate: Fix various path manipulation issuesRichard Purdie
Fix missing parameter to endswith and pass paths through normpath to remove any duplicate "/" characters which would corrupt other calls like basename. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-14sstate: Be consistent about sstate-inputdirs/outputdirs ending with '/'Richard Purdie
The manifest file can become corrupted if sstate-inputdirs and sstate-outputdirs don't have matching endings. This patch ensures that even if set incorrectly, the code functions as intended, thereby handling manifest corruption safely. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-27sstate: add manifest info for shared file matchesSaul Wold
Present the manifest file that contains the matches for files being installed to a location that already contains that file. This will help to determine which is the correct recipe to fix when this occurs. [YOCTO #3191] Signed-off-by: Saul Wold <sgw@linux.intel.com>
2012-10-22sstate: Improve handling of machine specific manifestsRichard Purdie
Now do_package isn't machine specific, we're only left with do_populate_sysroot as a machine specific task. This change marks only the machine specific manifests as machine specific, defaulting to PACKAGE_ARCH for everything else. This means we do less work where there are multiple machines using the same core package architecture and we can start to clean up the sstate duplicate files whitelist. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-18sstate: when warnings about sysroot overwrites, say what the recipe wasRoss Burton
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-18sstate: Use -m option to tar when unpacking sstateRichard Purdie
We've noticed failures on the project autobuilders where a shared sstate directory is used across multiple builders and the clocks become skewed. Most of the time this causes harmless building but if this happens where an environment is changed (make install vs make in qt4-x11-free for example), the build can fail. This avoids modification times in the future and should make builds safer in shared environments sstate was designed for. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-08sstate: Also add datadir/sgl to sstate whitelist to avoid openjade warningRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-05sstate: Add extra entries to the sstate duplicate files whitelistRichard Purdie
This avoids errors where gcc/binutils get installed to the native sysroot in the same location for multiple package architectures. Ultimately making these native recipes with ${PACKAGE_ARCH} appended to PN will resolve this but hide the warnings until this gets sorted out. Also hide the python and docbook catalog warnings since they're known about, nothing to worry about and we'll aim to clean them up properly in the 1.4 cycle. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-05sstate: Normalise paths before comparing with the whitelistRichard Purdie
Without this, path components like // could break comparisions with the whitelist leading to warnings being displayed to the user unintentionally. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-28sstate: Relax the duplicate file whitelist for nowRichard Purdie
do_package is a machine specific task at the moment due to packagedata. This means do_package tasks and their dependencies rerun between different machines with various duplicate file installations. There are plans to fix this but they're too invasive before release. This patch relaxes the whitelist for sstate duplicate file detection to account for this. Post-release, we re-enable stricter settings once do_package is not machine specific. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Saul Wold <sgw@linux.intel.com>
2012-09-26sstate: Remove master manifest usageRichard Purdie
This was added to allow detection of duplicate files being installed by sstate. There is a much simpler way, just check if the file already exists. This effectively uses the kernel VFS as the cache which is much more efficient. This resolves a significant performance bottleneck (lock contention on a single file) when running builds that are just being generated from sstate cache files. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-26sstate: Fix SSTATE_DUPWHITELIST variable usageRichard Purdie
We need to split this variable before using it. Otherwise a single "/" character in the list whitelists every overlapping sysroot file which was not the intention making the whole thing useless. We'll start seeing warnings about overlapping files now this is working correctly after this patch. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-12sstate: Append to EXTRASSTATEMAPS and add commentRichard Purdie
Appending to EXTRA_SSTATEMAPS is better than just hardcoding a value. Also add a comment about why this is necessary. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-31sstate.bbclass: Remove hardcoded task mappingsRichard Purdie
I've not idea why this got left in but as per the comment, it needs fixing and we shouldn't have hardcoded mappings like this. Lets remove it and dynamically generate the data instead. [YOCTO #3039] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-30sstate: Ensure master.list exists if it doesn't alreadyRichard Purdie
This fixes builds where master.list doesn't already exist. This change was meant to be part of the previous sstate commit but ended up separated. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-30sstate.bbclass: Detect overwriting files in sstate controlled directoriesRichard Purdie
We have a potential problem where two sstate packages try and touch the same file. This adds code which will print a warning whenever this happens. The implementation does but by maintaining a master file list and comparing file accesses against this. There are a number of places we have duplicate accesses which are harmless, mostly in the deploy directory so these are whitelisted. For now the code prints warnings, this could be strengthened in future to become error messages. Whilst working on this code on and off over the past few months various issues were uncovered, some serious. [YOCTO #238] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-21meta/classes: Various python whitespace fixesRichard Purdie
It was pointed out we have a number of weird indentations in the python functions. This patch cleans up 3, 7 and other weird indentations for the core bbclass files. It also fixes some wierd (odd) shell function indentation which my searches picked up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-27sstate.bbclass: Add a string representing the lsb release to native/cross sstateRichard Purdie
This patch adds the lsb name and revision to the path used for sstate files. This means that reuse of sstate files between different distributions is restricted by default. The behaviour can be configured using mirror urls, for example: SSTATE_MIRRORS = "file://Ubuntu-11.10/(.*) file://Ubuntu/\1 \n" would map Ubuntu 11.10 to a more generic "Ubuntu" named sstate feed. Usually, more modern distros have increased libc versions for example so whilst more older native/cross sstate packages will usually work on newer distros, the opposite is not true. This patch allows development of policy to better handle this although no default policy is currently being used. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-27sstate: Add a two character subdirectory to the sstate directory layoutRichard Purdie
Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories. This should help performance of sstate in most cases. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-22sstate: Set SSTAGE_PKGARCH for allarch packages to allow reuseRichard Purdie
allarch sstate packages could be marked as machine or package_arch specific. This change ensures they are not. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-19Convert tab indentation in python functions into four-spaceRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-16sstate.bbclass: Move crosssdk before cross to ensure correct variables are setRichard Purdie
There was a bug showing up where the crosssdk recipes were being installed as machine specific in the sstate-control directory. This turned out to be due to the architecture fields used by sstate being set incorrectly. The problem is that the crosssdk inherits the cross class. It therefore needs to be listed in this if statement block before the cross check, not after. This should resolve some package architecture issue of crosssdk sstate files. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Saul Wold <sgw@linux.intel.com>
2012-05-30meta: replace os.system with subprocess.callRobert Yang
Replace os.system with subprocess.call since the older function would fail (more or less) silently if the executed program cannot be found More info: http://docs.python.org/library/subprocess.html#subprocess-replacements [YOCTO #2454] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-18sstate.bbclass: Make sure we don't have an empty fixmepath fileMark Hatle
Jason Wessel noticed that a package without any fixmepath entries would generate a sed warning about no input files. This patch resolves that by ensuring that an empty fixmepath file never gets written into the sstate archive. Also we avoid a second message by only doing xargs if we got input. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-17meta/classes/sstate.bbclass: ensure sstate files have group R/WJoshua Lock
In order to make sharing sstate files easier chmod them with 0664 permissions so that they are readable and writable by the user and any other members of their group. Signed-off-by: Joshua Lock <josh@linux.intel.com>
2012-05-15sstate.bbclass: Optimize the generation and install path fixupsMark Hatle
The fixmepath file that is generated contains a list of all of the files that need their paths fixed. In the previous version the fixmepath was generated to include all of the files that sed may have changed. In the new version, we first grep the files to see if they contain a path that needs to be changed, only then do we perform the sed operation on those files. This results in a modest performance increate in the creation of the sstate file. The following numbers include the do_package and do_populate_sysroot tasks on the perl recipe. Before the change: real 4m23.018s user 1m57.067s sys 1m33.327s After the change: real 4m13.083s user 1m54.062s sys 1m26.064s However, a more significnt performance gain is felt during the extraction/install of sstate cache files, as the fixmepaths file now has a significantly smaller list of files to modify. Before the change: real 0m39.798s user 0m11.158s sys 0m12.642s After the change: real 0m25.511s user 0m8.408s sys 0m5.077s (All numbers above were recorded with a cold filesystem cache on a machine with 12 GB of ram.) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-11sstate.bbclass: Improve sstate_installpkg performanceMark Hatle
In a pathological case, lots of files to process, the sstate_installpkg performance was very poor. It interated over each file and ran 3 individual sed commands per file. Changing this to keep iterating but running only a single command took about 1/3 time time. However, when looking at the corresponding sstate_hardcode_path function, it was clear we could optimize this further. Using the same encoding logic to specify only the minimumal sed operation necessary, and using xargs to avoid the os.system call the install step was able to be performed in 13% of the original time. Example timing numbers for perl: 3m7s original code 1m20s single sed, but interating 0m26s using xargs and limited sed Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-23Revert "bitbake.conf/sstate.bbclass: Change PATH when installing sstate ↵Richard Purdie
files to avoid issues" Reluctantly revert this since it breaks the tar-native workaround we have for old versions of tar :( This reverts commit 01218e29f963120c77885964702198d9bce77e11.
2012-03-23sstate.bbclass: Remove possibility of file corruption and make package ↵Richard Purdie
writing atomic There is currently a race window when creating sstate packages since we don't atomically write the files to SSTATE_DIR. This change ensures we do so by writing to a temporary file and then doing an atomic move. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-21bitbake.conf/sstate.bbclass: Change PATH when installing sstate files to ↵Richard Purdie
avoid issues This resolves issues related to pigz-native when installing from sstate that people have been seeing. It also gives us a way to solve issues like the gzip-native race during sstate package creation covered in Yocto #1774. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05meta: Replace bb.data.expand(xxx, d) -> d.expand(xxx)Richard Purdie
sed \ -e 's:bb.data.\(expand([^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -i `grep -ril bb.data.expand *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05sstate: Remove unecessary bb.data.expand()Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05meta/classes: Convert to use appendVar and appendVarFlagsRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-24sstate.bbclass: improve performance of sstate package creationMartin Jansa
* also fixes replacing paths for perl where cmd line was probably too long for os.system(cmd) (it had 560410 characters because a lot of files from sstate_scan_cmd). * also print those 2 commands so we can find them in log.do_package Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
2012-02-23sstate.bbclass: add *_config to SSTATE_SCAN_FILESMartin Jansa
* e.g. mysql5 has mysql_config not mysql-config Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-10sstate.bbclass: Optimise sstate_hardcode_pathRichard Purdie
The sstate_hardcode_path() function triggered large numbers of exec() calls when processing packages with large numbers of file relocations (e.g. perl). This patch optimises those calls into longer single commands which make the code significantly more efficient. This reduced the do_package time for perl by 2 minutes (from 4.75 minutes) for me. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-27sstate.bbclass: Add support for sstate preinst functionsRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-12-15sstate: Add SSTATE_SCAN_FILESRichard Purdie
We process all files in the native/cross cases for finding and fixing relocation issues. In the target case we've only processed .la and binconfig files. Since there are other files which are in need of this processing, this change allows recipes to specify files that may be outside the normal set. This means hardcoded paths that need to be fixmepathed to work correctly are handled and addresses some sstate relocation bugs that have been seen. Based on a patch from Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-26getVar/setVar cleanupsRichard Purdie
Complete the bb.data.getVar/setVar replacements with accesses directly to the data store object. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-21sstate.bbclass: Ensure we expand stamp-extra-infoRichard Purdie
Without this change we can end up looking for <stamp>.${MACHINE} instead of the expected expanded value. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-10Convert to use direct access to the data store (instead of bb.data.*Var*())Richard Purdie
This is the result of running the following over the metadata: sed \ -e 's:bb.data.\(setVar([^,()]*,[^,()]*\), *\([^ )]*\) *):\2.\1):g' \ -e 's:bb.data.\(setVarFlag([^,()]*,[^,()]*,[^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -e 's:bb.data.\(getVar([^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \ -e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \ -e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^() ]*\) *):\2.\1):g' \ -e 's:bb.data.\(getVar([^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -i `grep -ril bb.data *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-10-06sstate.bbclass: Ensure machine specific stamps are only wiped for the ↵Richard Purdie
current task sstate was being a little too ethusiastic about removing stamp files and was removing stamp files for other machines when it shouldn't have been. This patch teaches sstate about machine specific stamp extensions and allows it to only remove the current task's stampfiles. Based on a patch from Phil Blundell <philb@gnu.org> with some tweaks from me. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>