| Age | Commit message (Collapse) | Author |
|---|
|
Tasks for image recipes cannot be locked - there's nothing to restore
from shared state to cover them and as a result, if you had "live" in
IMAGE_FSTYPES the build would fail with "taskhash mismatch" errors for
do_rootfs and do_image_complete for the initramfs image recipe, since it
had to try to run those. We should probably catch that issue earlier in
the build and produce a proper error, but for now at least exclude these
signatures from the locked-sigs.inc file so that extensible SDK
installers built when IMAGE_FSTYPES includes "live". (It turned out we
already had code to find other image tasks in the task list in order to
generate the list of install targets.)
Follow-up fix for [YOCTO #9826].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If you build an extensible SDK for an image and IMAGE_FSTYPES includes
"live" then the extensible SDK will fail to install with a bunch of
unexpected task execution errors, matching the missing items required to
build the live image. The issue was we were still depending on do_rootfs
rather than do_image_complete. The fix was slightly more complicated
than just changing the task name as do_image_complete's dependencies are
in the form of dependencies on tasks within the same recipe (represented
in the "deps" varflag rather than the "depends" varflag).
Fixes [YOCTO #9826].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Use the new BB_SETSCENE_ENFORCE functionality to avoid having to run
bitbake twice on installing the extensible SDK - we can now do it all in
one invocation which not only takes less time, but we should also get
more meaningful errors for some types of failure, in particular where
downloading from an sstate mirror fails.
One result of this change is that you get the errors printed on the
console during normal output rather than this going to the
preparing_build_system.log file first. In OE-Core revision
227d2cbf9e0b8c35fa6644e3d72e0699db9607fa, we changed to always print the
contents of preparing_build_system.log on failure, but now at least the
error contents of that log is duplicated. Besides, I intentionally
didn't print out the contents of that log during normal usage because
it's quite verbose - the bug that we were attempting to fix was about
not getting this information when seeing failures in the automated
tests, thus I've moved printing the log to the test handling code
instead.
Part of the implementation of [YOCTO #9367].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Blacklist SSTATE_DIR, DL_DIR and TMPDIR by default to avoid problems
such as the one mentioned in [YOCTO #9605].
* Blacklist BB_NUMBER_PARSE_THREADS since we already blacklist
BB_NUMBER_THREADS and they may be set separately.
Fixes [YOCTO #9605].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
It has been determined that it is highly likely that users might get
signatures that don't match in an extensible sdk. This doesn't
necessarily happen with oe-core, so we can set the mismatch to an error
during testing if we like.
However, for the case where users are creating their own sdks, we don't
need an error halting their progress. locked-sigs will still function as
it should.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
To avoid lots of output in the SDK EXT installation phase, system redirects
it to a logfile ($target_sdk_dir/preparing_build_system.log) but in case of error,
the contents should be shown so debugging could be faster.
[YOCTO #9576]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When a recipe is added to the workspace, the signatures for the tasks
will change. This means that bitbake must be told to allow the
signatures to be different if they are in locked-sigs.inc.
This is done by creating an unlocked-sigs.inc file which contains all
the recipes in the workspace each time devtool reads the workspace.
So not only will necessary things get added, previously added items will
be removed by virtue of them no longer being in the workspace.
This also makes sure that the extensible sdk picks up unlocked-sigs.inc
as part of the configuration.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the extensible sdk we want there to be an error if a task tries to
run without signatures that match locked-sigs.inc. This patch enables
that error.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Otherwise the task hash changes between server and worker context
leading to changing task checksums. The dependency data here is
tracked by other pieces of the signature.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If SDK_EXT_TYPE is set to "minimal" then the SDK won't contain many
sstate artifacts, and you're required to set up an sstate mirror in this
case anyway so there's no point publishing the "stub" sstate-cache
directory from within the SDK since it won't be useful for update
purposes and may be confused with the real sstate-cache.
There is however a possibility that people might publish the real
sstate-cache directory under the same output directory provided to
oe-publish-sdk, thus deleting it after extracting (as we were doing with
other files we wanted to clean up at the end) would be problematic,
besides which extracting it and then deleting it is wasteful. Thus,
introduce a "-p" command line option to the SDK installer that we can
use to tell tar not to extract the items we don't want when publishing.
This has the added benefit of mostly keeping references to these in the
place they belong i.e. in populate_sdk_ext.bbclass.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When running bitbake you may pass in values of variables from the
external environment (making use of BB_ENV_EXTRAWHITE), and you may
choose to do this when building the extensible SDK, for example:
MACHINE=qemuarm bitbake -c populate_sdk_ext core-image-minimal
You would naturally expect those settings to be reflected in the
extensible SDK itself; however they were not, since we were only
considering local.conf and auto.conf. Check the variables mentioned in
BB_ENV_EXTRAWHITE to see if any are different than the values set in
local.conf/auto.conf and add lines setting them in the SDK's local.conf
if so.
Fixes [YOCTO #9339].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are various concerns about md5 so use sha256 instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of skipping the build system preparation step within the
extensible SDK install process when SDK_EXT_TYPE is "minimal", run
bitbake -p so that the cache is populated ready for the first time
devtool is run.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Commit 8b81bb56c69aabdea984352f8e267a9783c0bdbc was accidentally merged.
The DL_DIR piece was simply incorrect and should be removed.
The patch commit message should have mentioned that the changes were
to update populate_sdk_ext after the changes to uninative now the
download is placed into a specific directory in DL_DIR. We also
need to specify the uninative tarball checksum.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since the metadata for multiple layers will be put into the sdk, anytime
those layers change the sdk needs to be rebuilt. Ideally, this would use
checksums to only rebuild when necessary.
However, since this feature needs to be in the release, opt for a less
intrusive change by setting the task to nostamp. Unnecessary rebuilds
may occur, but it's better than a user wondering why their changes
didn't get added.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a build-sdk command which is only available within the extensible
SDK that builds a derivative extensible SDK. The idea is recipes in the
workspace become a part of the new SDK - for example, this allows taking
a vendor provided SDK, adding a few libs and then producing a new SDK
with those included.
When normally building the extensible SDK, the workspace is excluded;
here we need to copy into the new SDK (renaming it in the process); the
recipes' task signatures become locked and thus the sources are no
longer needed, so they are removed along with the workspace bbappends
which would interfere with the locked signatures. Additionally we need
to just copy the configuration files (i.e. local.conf and auto.conf)
rather than filtering and appending to them since that work has already
been done when constructing the original SDK. The extra sstate artifacts
from workspace recipes are also determined and copied into the new SDK
in minimal mode (on the assumption that you won't set up a new sstate
mirror).
This reuses some code from build-image, so that needed to be
generalised to allow that.
Implements [YOCTO #8892].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is possible that LCONF_VERSION won't be set, such as if meta-poky is
used. Without this change, bblayers.conf would have LCONF_VERSION =
"None" if LCONF_VERSION wasn't set, which would cause a sanity check
failure.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When installing the ext sdk, buildtools is extracted and installed as
well. The tar file containing buildtools isn't used after installation
so was wasted space and clutter.
[YOCTO #9172]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The rest of the environment setup script prepends to PATH, so when we
add the path to run devtool we should be prepending as well. This
also ensures that when you run the environment setup script from
extensible SDK installation A and then in the same shell session run the
environment setup script from installation B, and then run devtool, that
you're running B's devtool and not A's.
Fixes [YOCTO #9046].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When running ext-sdk-prepare.py during sdk installation a check is done to
make sure no tasks would run that aren't provided by the "leaf" recipes
specified in SDK_INSTALL_TARGETS.
However sometimes an image recipe can cause other images to be created
such as an initramfs. So make sure those additional images are
recognized by ext-sdk-prepare.py and don't flag an error.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The sdk_extraconf() method of setting the configuration was awkward
since you needed to set it in a class and then inherit that class since
function definitions aren't allowed in conf files. It seemed to me the
a neater way to do this was to read the extra lines from an additional
conf file sdk-extra.conf (which can be located in a conf/ directory
anywhere along BBPATH as with other configuration files).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently there isn't a way for the extensible sdk to know all the tasks
that will need sstate for an image. This is because a layer can add it's
on custom tasks that are required for an image to be generated.
The extensible sdk solved this for poky by using recrdeptask and
specifying the tasks known to be required for the image as well as for
building new recipes.
So the SDK_RECRDEP_TASKS variable allows a user to specify additional
tasks that need to be pulled in.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes a problem where SDK_INSTALL_TARGETS wouldn't pick up the
value in SDK_TARGETS. It also removes the inline python to make the
code more readable.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Extensible SDK needs to point to the correct manifest so add
SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST variables.
oeqa/oetest.py: Fix SDKExtTestContext for load the correct manifests.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This variable is needed by testextsdk to known the name of extensible
sdk file generated.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If a user changes sdk_extra_conf, it should cause populate_sdk_ext to
run.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the user sets the SDK_EXT_TYPE variable to "minimal" then the sdk won't
contain any sstate. The sstate can come from an sstate mirror and be
installed on demand as usual.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
sdk_update_targets isn't used by any code, so there is no reason to set
it.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
SSTATE_MIRRORS used by the builder may not be the same as what the
installer of the sdk will use. Therefore blacklist SSTATE_MIRRORS from
the builder configuration. Note: the actual SSTATE_MIRRORS for the sdk
can be added using sdk_extra_conf.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the script files we use to construct the SDK installer change then
that really ought to trigger re-execution of the do_populate_sdk(_ext)
task, so add file-checksums varflags to ensure that happens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At the end of the extensible SDK installation, if we've successfully
prepared the build system then we don't need ext-sdk-prepare.py. I had
thought earlier that this would be used when updating, but a different
mechanism was needed there so this script isn't used for that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Make the following improvements to the SDK update process:
* Use a manifest file with sha256sums to track files other than sstate
and metadata that we need to update - e.g. conf files. This allows us
to handle where files such as auto.conf may or may not be present,
as well as the configuration changing without affecting task signatures
- we still want the config files copied in that case rather than it
saying nothing needs to be done.
* Write the SSTATE_MIRRORS_append to site.conf rather than local.conf
so that local.conf remains static (since we don't want to trigger an
update every time). Also, If there is an SSTATE_MIRRORS value already
set in the configuration we can skip this and assume it contains the
needed packages.
* Allow the update process to be run in any directory, don't assume
we're already at the base of the SDK
* Where practical, fetch remote files into a temporary location and
then move them to the desired location at the end, to avoid a
failed update leaving the SDK in a broken state.
* Update all installed do_populate_sysroot / do_packagedata tasks
instead of using the SDK targets. This ensures any item installed
through dependencies after installation (e.g. when running
"devtool build") won't go stale.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a variable SDK_INCLUDE_PKGDATA which you can set to "1" to include
pkgdata for all recipes in the world target. There are a couple of uses
for this:
1) If you use "devtool add" to add a recipe that builds something which
depends on anything in world, the dependency can then be correctly
mapped to the recipe providing it and that recipe can be added to
DEPENDS, since we have the pkg-config and shared library dependency
data within pkgdata.
2) You'll be able to search for these recipes and any files they
package for the target with "devtool search" since that also uses
pkgdata
This of course assumes you've tailored world through EXCLUDE_FROM_WORLD
to only include recipes you'd want built in your distro, but I think
that's a reasonable assumption; failing that there is a
WORLD_PKGDATA_EXCLUDE variable that you can set to exclude any recipes
you don't want.
Note that this patch relies on functionality implemented in a recent
BitBake patch and will not work without it.
Implements [YOCTO #8600].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix a regression caused by OE-Core revision
eabeb26335b1a4eb1e68218160dbdbe8fdf36272 where we lost the task
dependency on packages in TOOLCHAIN_HOST_TASK (such as
meta-environment-extsdk-${MACHINE}) which led to errors about missing
packages when building the extensible SDK. This happened because I only
moved half of the dependencies coming from the standard SDK task to
do_sdk_depends.
While I'm at it, tidy up the do_populate_sdk_ext[depends] line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Create a separate task where we can just have the recursive dependencies
for the tasks corresponding to constructing the SDK content (i.e. from
the image contents). This avoids us recursing into dependencies from
buildtools and getting a bunch of nativesdk stuff, for example.
(This isn't an ideal way to have to implement it, but without
overcomplicating things on the BitBake side just for this use-case I
can't see a better way.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After the change to use --setscene-only when running bitbake to prepare
the SDK at the end of installation, add a check that the SDK got
prepared correctly by doing a dry-run and looking at the output for any
real tasks that we don't expect. In order to make this easier, the
preparation shell script was rewritten in python.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If auto.conf exists in the user's configuration we need to also run it
through the same filter and write the result into the ext SDK, or we
risk missing configuration applied on an autobuilder.
Fixes [YOCTO #8904].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
newline
If there is no newline at the end of local.conf, appending
INHERIT = "uninative" won't work, it will corrupt the line and the
installed eSDK will build things, making the "Preparing build system..."
step take an age.
Fixes [YOCTO #8897].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
prune_lockedsigs expects excluded_targets to be a list, whereas
previously it was passed in as a string.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Right now, the locked task hashes list for the extensible SDK locks
down only the sstate tasks.
Whilst asthetically pleasing, this gives two problems:
* Half the task are left floating meaning checksum mismatches
are a pain to debug
* The later code which copies relavent data files out the sstate
cache can't use any of this data.
This patch modifies things so all the checksums are listed in the locked
file. An exclusion of tasks probably makes more sense for the library
function rather than an allowed list.
The only sstate task being deliberaly excluded here was do_package
so add in a function to explictly exclude those sstate object files.
The net result of this that siginfo files for all tasks are included in
the SDK, which means commands like "bitbake -S printdiff" now function.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The user of the extensible SDK doesn't need to see these.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
During extensible SDK installtion, if the build system preparation step
fails we try to put something at the end of the environment setup script
to show an error when it is sourced, in case the user doesn't realise
that the partially-installed SDK is broken. However, an apostrophe in
the message (actually a single quote) appears to terminate the string
and therefore breaks the command. Drop it to avoid that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Ensure extensible SDK and standard SDK go into their own directories
* Record extra extensible SDK variables
* Write sstate-package-sizes.txt and sstate-task-sizes.txt files so you
can analyse the size of the contents
* Add BUILDHISTORY_SDK_FILES (similar to BUILDHISTORY_IMAGE_FILES) and
default it to pick up config files installed in the extensible SDK
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If the installation of buildtools fails then we should fail the entire
installation instead of blindly continuing on.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The configuration of the build system within the extensible SDK is
fixed, so there's not a lot of point in showing it; plus it just gets in
the way of the output that's interesting to the user in this context. So
let's hide it within the extensible SDK.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We inherit uninative in the extensible SDK configuration, and uninative
sets NATIVELSBSTRING to a fixed value, so we don't need to force the
value ourselves.
Fixes [YOCTO #8662].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should always pass a parameter to getVar, add missing default value.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Paul Eggleton
Richard Purdie
Randy Witt
Leonardo Sandoval
Aníbal Limón
Aníbal Limón