| Age | Commit message (Collapse) | Author |
|---|
|
buildhistory.bbclass by design is incremental: each build adds or
updates information. Information is never removed.
Sometimes it can be useful to reduce the information only to those
recipes that were build during a specific bitbake invocation, for
example when the invocation does a full world build.
This is now possible by setting BUILDHISTORY_RESET as explained in the
modified class. The comment on the variable also mentions the caveats
associated with using this mode.
In this mode, buildhistory.bbclass first moves all existing
information into a temporary directory called "old" inside the build
history directory. There the information is used for the "version
going backwards" QA check. Then when the build is complete and before
(potentially) committing to git, the temporary directory gets deleted.
Because information that has not changed during the build will be
reconstructed during full world rebuilds, a git log will then only
show real updates, additions and removals.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Variable expressions are no longer expanded in python functions as of
BitBake commit 8bf33a8e92c0e188fa392030025756196c96fcbb, so we've now
got to do this explicitly here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its valid to have dependencies like XXX (=2.1) without spaces, as injected
by debian.bbclass. The code was breaking these into separate components
and destroying them so improve the regex to handle them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously, in order to determine the existence of an already
initialized Git repository we checked if a directory named '.git' was
present in the buildhistory dir. However, e.g. in the case of git
submodules '.git' may also be a regular file referencing some other
location which was causing unwanted behavior. This patch changes
buildhistory.bbclass to check for any file named '.git' which fixes
these problems.
[YOCTO #8911]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This patch changes the use list_pkgs() instead of list()
from class RpmPkgsList. The change is in two functions,
image_list_installed_packages from rootfs.py and
sdk_list_installed_packages from sdk.py.
With this change the functions calling the functions
listed above, must format the output as they required.
The formatting can be done using format_pkg_list() from
oe.utils.
The classes calling the afected functions are changed too
with this patch, to keep the same functionality using the
new data structure.
[YOCTO #7427]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
These form part of the configuration for the extensible SDK, we should
really be recording what goes into them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I was using bytes here for the sizes and writing out KiB as the suffix.
Change it to actually write out size values in KiB.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* Ensure extensible SDK and standard SDK go into their own directories
* Record extra extensible SDK variables
* Write sstate-package-sizes.txt and sstate-task-sizes.txt files so you
can analyse the size of the contents
* Add BUILDHISTORY_SDK_FILES (similar to BUILDHISTORY_IMAGE_FILES) and
default it to pick up config files installed in the extensible SDK
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
After OE-Core revision baa4e43a29e45df17eaa3456acc179b08d571db6 we lost
recording SDK the contents in buildhistory. This was due to the
SDK_POSTPROCESS_COMMAND variable being set with = in
populate_sdk_base.bbclass which overwrote any value set with += in
buildhistory.bbclass; to fix it, use _append in buildhistory.bbclass
instead.
Fixes [YOCTO #8839].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The idea behind the implementation of Yocto #8138 was that an
additional class can write additional files in the recipe directories,
for example by hooking into the functions of buildhistory.bbclass or
by implementing its own SSTATEPOSTINSTFUNCS function.
However, when these additional files get created before
buildhistory_emit_pkghistory(), they get removed again by that
function because it contains code which removes everything it does
not know about. The reason for that is that these unknown items
are probably obsolete.
This logic is the reason why the additional "kconfig" file from
buildhistory-extra.bbclass never showed up in the final build history.
To fix this, the hard-coded list of known files in
buildhistory_emit_pkghistory() must be turned into a variable which
derived classes can extend.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Fix some spelling mistakes in bbclass files
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We don't actually use values we read in here that are likely to
contain = characters but we might as well split the value properly in
case we do in future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We have the command in the commit message, we might as well have the
build result as well (succeeded/failed and whether or not it was
interrupted by the user). The interrupted part relies upon a change to
BitBake to extend the BuildCompleted event to include an attribute for
that, but will not fail if the attribute is not present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We do already commit these into the repository itself, but have them in
the commit message as well as a reference. As part of this, refactor out
running "git commit" into a separate function so we don't have to
duplicate the code in the two places we call it.
Implements [YOCTO #7966].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Avoid an error when attempting to remove previous data if it's not a
subdirectory - we were assuming that anything that wasn't named "latest"
or "latest_srcrev" had to be a directory. This makes it possible to have
a buildhistory_emit_pkghistory_append which writes additional files at
the recipe level.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If we want the correct file permissions to show up here as they would on
the target, we need to run the command under pseudo. Normally we'd set
the fakeroot varflag on the function and that would be enough, but it
turns out that setting fakeroot on a non-task function that you execute
using bb.build.exec_func() isn't working at the moment. Work around this
by simply using FAKEROOTENV and FAKEROOTCMD. Unfortunately that means we
have to duplicate the command for the two cases but I couldn't find a
better means of doing that that actually works.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* Avoid using ${...} for shell variables (since they could be
expanded as bitbake variables if present)
* Use files-in-package.txt rather than files-in-<packagename>.txt; the
file is already in a subdirectory named with the package name and this
naming is consistent with that of files-in-image.txt.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Shell functions use tabs in this file.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We don't care about the permissions on the top-level directory in which
the files are contained, just everything under it; this also avoids
lists with just this entry in it for empty packages. Affects file
listings for both images and packages.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If there aren't any changes, we still make a commit to the buildhistory
repo, but this wasn't being pushed if BUILDHISTORY_PUSH_REPO is set.
Move the push to the end to make it unconditional.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
files-in-$pkg.txt
This is just a safe check to make sure the output folder is present, before
creating the files-in-$pkg.txt file.
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
A file similar (in format) to files-in-image.txt is created for each
recipe's build package. This would allow to easily see what the
recipe's package may install when including it into an image.
These files reside on the specific buildhistory recipe package (i.e.
buildhistory/packages/i586-poky-linux/busybox/busybox-dev/files-in-busybox-dev.txt
for busybox-dev package)
[YOCTO #5870]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The git user.email and user.name settings must be set or the commit to
the buildhistory repo (when BUILDHISTORY_COMMIT is set to "1") will
fail. If the user hasn't set these, set some dummy values since the
actual value is unlikely to be particularly useful and it's far worse
for the commit not to be done leaving the repository in a partially set
up state.
Prompted by [YOCTO #6973].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* Use argparse instead of optparse for standardised help output, options
and a much cleaner code structure
* Look up pkgdata directory automatically so the user doesn't have to
specify it
* Use standard logging
NOTE: this does mean a slight change in syntax - if you do want to
specify the pkgdata directory (usually only necessary if you're calling
it from within the build process) you need to use the parameter -p (or
--pkgdata-dir) and specify this before the command, not after it.
Examples:
oe-pkgdata-util find-path /sbin/mke2fs
oe-pkgdata-util lookup-recipe libelf1
oe-pkgdata-util read-value PKGSIZE libc6
oe-pkgdata-util -p /home/user/oe/build/tmp/sysroots/qemux86-64/pkgdata read-value PKGSIZE libc6
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Since the rewrite of the image construction code in python a few
releases ago, we remove a couple of packages from the image as one of
the final steps when constructing the image (notably update-rc.d and
run-postinsts). However, because of the order of operations, these
packages are still listed both in the buildhistory
installed_package*.txt files and in the manifest file created next to
the image, which is wrong.
There were two possible solutions to this: (1) change the order such
that the uninstallation occurs before calling ROOTFS_POSTPROCESS_COMMAND
or (2) add another hook variable in such that we can have the
package list collection code run at the right time. Because it's
currently possible (but very much not recommended) to install additional
packages within ROOTFS_POSTPROCESS_COMMAND, which may have postinstall
scripts and thus require the packages we would otherwise uninstall if we
were to take option 1, option 2 is really the least likely to cause
problems. Therefore, add ROOTFS_POSTUNINSTALL_COMMAND and make the image
and buildhistory classes use it.
Fixes [YOCTO #6479].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Writes build information to target filesystem on /etc/build such as enabled
layers, their current status and commit.
squashspaces was moved to oe/utils.py to make it available to different classes
and avoid code duplication.
[YOCTO #6770]
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
SSTATEPOSTINSTFUNCS is now set with = in sstate.bbclass, and because
the line here in buildhistory.bbclass ends up being parsed before that
we now need to use _append or the value we are adding will be wiped out.
This fixes buildhistory no longer recording package information since
OE-Core revision 9d659c6f20fa4a141b491c62a3ef0dfb1f896d9c.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* it was added in:
http://git.openembedded.org/openembedded-core/commit/meta/classes/buildhistory.bbclass?id=540a2a30be21c3eca4323efbe91e7dcfc31a4c97
* then accidentally removed in:
http://git.openembedded.org/openembedded-core/commit/meta/classes/buildhistory.bbclass?id=6db6c74f2876df7a183990426bb4262972966441
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* use .txt extension like other reports (e.g. image-info.txt) it's
also useful when looking at the file from web server which doesn't
have text/plain as default content-type
* include whole BUILDCFG_HEADER/BUILDCFG_FUNCS instead of just layer
info
* this makes it easier for distro to include more fields without the
need to modify buildhistory.bbclass
* currently it adds following fields:
Build Configuration:
BB_VERSION = "1.23.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Gentoo-2.2"
TARGET_SYS = "x86_64-oe-linux"
MACHINE = "qemux86-64"
DISTRO = "shr"
DISTRO_VERSION = "2013.07-next-20140725"
TUNE_FEATURES = "m64 core2"
TARGET_FPU = ""
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Replace:
cat <file> | sed -e xxx
By:
sed -e xxx <file>
Signed-off-by: Matthieu Crapet <Matthieu.Crapet@ingenico.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
The base_contains is kept as a compatibility method and we ought to
not use it in OE-Core so we can remove it from base metadata in
future.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The old functions were calling the list_installed_packages() wrapper
function that only listed the packages in an image rootfs. Even for
target/host SDK. Also, a python crash was possible if 'bitbake -c
populate_sdk core-image-*' was called without calling 'bitbake
core-image-*' first. That's because the wrapper was always looking into
the image rootfs...
This commit fixes the problem and calls the right wrapper for image/sdk.
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Call the new python routines.
[YOCTO #5904]
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix task signatures for recipes that add to SSTATEPOSTINSTFUNCS changing
when adding and removing INHERIT += "buildhistory" (really this time!)
This relies on the BitBake vardepvalueexclude feature, however it will
not fail without it - signatures will be changed in that case though.
Part of the fix for [YOCTO #5897].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
Call the new python routines to create the packages list.
[YOCTO #5831]
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
buildhistory_get_sdk_installed
bb.build.exec_func() does not allow passing arguments to the executed
scripts. Use wrappers instead.
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Postfuncs are now incorporated into task signatures as of BitBake
rev b84d010144de687667cf855ddcb41c9b863c236e, so we need to exclude
the one we're adding to do_fetch. The change to the SSTATEPOSTINSTFUNCS
variable value was also entering the signatures of do_populate_sysroot
and probably other tasks, so we need to use a slightly hacky trick to
avoid that as well.
Apart from the final do_rootfs / do_populate_sysroot, this now means
that adding and removing INHERIT += "buildhistory" will not cause tasks
to be re-run.
Also update the copyright date, properly this time.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the package information contained unicode characters (for example, in
the package file listing) then writing them out as ASCII would fail, so
write them out using codecs.open() instead of open() using UTF-8. This
fixes ca-certificates failing in do_packagedata when buildhistory is
enabled.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
OE-Core commit 259b8718a31b886f8a158aeb5de164840c9a28b2 fixed UTF-8
errors but broke decoding of escape sequences in strings (e.g.
pkg_postinst scripts had \n \t in them instead of newlines and tabs.)
We need a second call to decode() here as specifying 'string_escape' as
the second parameter won't do anything.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The filenames sometimes may have strange names. With the 'awk' script
it handled a limited number of spaces in the filename and a package
installing a file named "test file with spaces" would have its name
truncated.
This patch uses the find's printf formating to simplify the code and
properly handle this case. From a testing image, the only diff produced
is:
,----[ files-in-image.txt diff ]
| --rwxr-xr-x root root 0 ./usr/bin/test\ file\ with\
| +-rwxr-xr-x root root 0 ./usr/bin/test file with spaces
`----
The options used are available since findutils 4.2.5, released in 19
Nov 2004, making it available in all supported host distributions.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
On systems where default locale is utf-8 we get errors like
File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts)
0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist)
0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character
u'\xed' in position 337: ordinal not in range(128)
This patch specifies decode to use utf-8 so ascii and utf-8 based
locales both work
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After 1b8e4abd2d9c0 [bitbake.conf/package: Collapse PKGDATA_DIR into
a single machine specific directory], oe-pkgdata-util does not use
target_suffix parameter, so do not need to loop the vendor
Signed-off-by: Roy Li <rongqing.li@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
When writing out variable values to pkgdata, if the value has been set
in the datastore with an override for the package, we use the package
name override in the pkgdata key as well; however the recently added
code to read pkgdata in buildhistory.bbclass was just using the override
where we normally expect to have it. However, if a recipe overrides one
of the values that is normally set for the recipe on a per-package basis
(e.g. the external-sourcery-toolchain recipe sets PKGV this way) then
this led to KeyErrors. Re-write the pkgdata loading code to always strip
off the package name override if it is present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't normally perform any operations (such as "git pull") that
trigger "git gc --auto", thus garbage collection never happens which
means performance of accessing the repository degrades noticeably over
time. Add an explicit "git gc --auto" to clean things up when needed.
Thanks to Elijah Newren and Ross Burton for suggesting this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Disable several python functions if not parsing within the worker
context. This avoids executing expensive operations while parsing
recipes (which is unnecessary).
(Thanks to Richard Purdie for pointing out the issue and suggesting the
workaround.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use a function added to SSTATEPOSTINSTFUNCS and read the necessary
information out of pkgdata, instead of using a function executed
during do_package that reads the data directly. This has two benefits:
* The package info collection will now work when the package content is
restored from shared state
* Adding/removing the inherit of buildhistory will no longer change the
do_package signatures and force re-execution of that function for
every recipe.
Fixes [YOCTO #5358]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add PACKAGE_EXCLUDE and NO_RECOMMENDATIONS to the info we track for
images, since these can change what ends up in the image.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This should have been removed when the implementation was rewritten in
OE-Core commit 2179db89436d719635f858c87d1e098696bead2a. The collected
values weren't being used anywhere since then.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The buildhistory code has fallback paths for older bitbakes for now. The
distrodata class is much less used and it can be assumed a recent bitbake
is used in that case rather than adding fallback code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Patrick Ohly
Paul Eggleton
Richard Purdie
Markus Lehtonen
Mariano Lopez
Maxin B. John
Leonardo Sandoval
Alejandro Hernandez
Martin Jansa
Matthieu Crapet
Otavio Salvador
Laurentiu Palcu
Ross Burton
Khem Raj
Roy Li