diff options
author | Martin Jansa <Martin.Jansa@gmail.com> | 2018-07-10 19:36:10 +0000 |
---|---|---|
committer | Martin Jansa <Martin.Jansa@gmail.com> | 2022-05-10 10:35:14 +0200 |
commit | 48f2e12861a328fe404f2835b55eeaf0eb80e868 (patch) | |
tree | 6856b998f0c58097dd5d892bc6e255c9f9498359 | |
parent | f24f8ec7b57779e411ac81b90efb44c1889546e0 (diff) | |
download | openembedded-core-contrib-jansa/artifacts.tar.gz |
image*.bbclass, kernel*.bbclass: create versioned hard links instead of version-less symlinksjansa/artifacts
* We used to create the actual artifact files with some version
in the filename and then created symlink without any version
which was updated to point to the latest one created.
In some scenarios it's useful to create all artifacts - typically
rootfs and kernel images with the same version - like release
build even when the kernel itself wasn't modified since the
previous release.
If we include the release version in the regular _NAME variables
then we'll need to re-run do_deploy and do_image which will cause
kernel to be rebuilt and image to be re-created even when the
only change since last build was the version number.
With this change we can re-use kernel and image from sstate when
nothing was changed and run only very fast do_deploy_links task
which just adds another hard link to existing artifact from
sstate.
* This is already used by various LGE builds as do_webos_deploy_fixup()
https://github.com/webosose/meta-webosose/blob/master/meta-webos/classes/webos_deploy.bbclass
but injecting this task in all the right places id difficult
and sometimes requires whole bbclass to be duplicated. Having
simpler way of versioning artifacts directly in oe-core might
be useful for others.
* move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME
that way e.g. kernel.do_deploy can be reused from sstate to
provide "version-less" artifacts and then very fast
do_deploy_links task just adds links with consistent suffixes
(by default the version from the recipe but could be easily set
to e.g. some release name when building some products).
* create hard links instead of symlinks, so that whatever version
the filename says is really there
* some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file
to be removed first or they might either append or update the
content of the image instead of creating new image file from
scratch - I have seen this only with one proprietary format we
generate with our own tool, so hopefully this isn't very common
* this is basically the mechanism are using in webOS with
WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from
jenkins job and then all artifacts (images as well as corresponding
kernel files) have the same version string)
* without this, you can still easily set the variables to contain
the version from jenkins job (excluded from sstate signature like
DATETIME currently is to prevent rebuilding it everytime even when
the content didn't change) but then when kernel is reused from sstate
you can have version 1.0 used on kernel artifacts and 2.0 on image
artifacts.
* if you don't exclude the version string with vardepsexclude, then
you get the right version in the filenames but for cost of
re-executing do_deploy every single time, which with rm_work will
cause all kernel tasks to be re-executed (together with everything
which depends on it like external modules etc).
* the implementation "from outside" is a bit tricky as shown in webOS
OSE, because first you need to reverse the meaning of IMAGE_NAME
and IMAGE_LINK_NAME like here, but also replace all symlinks with
hardlinks and then adjust all recipes/bbclasses to depend on our
do_deploy_fixup task instead of the original do_deploy
see the variable modifications:
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65
and then various bbclasses to hook do_webos_deploy_fixup task creating
the hardlinks for possible artifacts:
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/kernel.bbclass
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/image.bbclass
so hopefully with all these changes in oe-core other project can
achieve the same just by setting one variable IMAGE_VERSION_SUFFIX
* drop ${PKGE}-${PKGV}-${PR} from kernel artifacts names (this is the
latest build) and add it only in hardlinks created in do_deploy_links
so that we can use PKGR there again (because these links are generally
used only by human operators and they don't have their own TASKHASH or
the IMAGE_VERSION_SUFFIX might be set to some release name which they
do understand
* this allows to drop package_get_auto_pr from kernel do_deploy as well,
leaving only 2 EXTENDPRAUTO bumps for each kernel build (do_package
and do_deploy_links, unfortunatelly these will still have different
value, so if you're looking for the exact kernel image in deploy
directory based on kernel image package version seen on the device the
EXTENDPRAUTO part of PR will be different).
[YOCTO #12937]
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
-rw-r--r-- | meta/classes/cve-check.bbclass | 4 | ||||
-rw-r--r-- | meta/classes/image-artifact-names.bbclass | 6 | ||||
-rw-r--r-- | meta/classes/image.bbclass | 10 | ||||
-rw-r--r-- | meta/classes/kernel-artifact-names.bbclass | 6 | ||||
-rw-r--r-- | meta/classes/kernel-devicetree.bbclass | 33 | ||||
-rw-r--r-- | meta/classes/kernel.bbclass | 44 | ||||
-rw-r--r-- | meta/classes/qemuboot.bbclass | 2 | ||||
-rw-r--r-- | meta/classes/rootfs-postcommands.bbclass | 4 |
8 files changed, 66 insertions, 43 deletions
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 24ddb865ea..da42cf1efa 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -201,10 +201,10 @@ python cve_check_write_rootfs_manifest () { manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name) # if they are the same don't create the link if manifest_link != manifest_name: - # If we already have another manifest, update symlinks + # If we already have another manifest, update hardlinks if os.path.exists(os.path.realpath(manifest_link)): os.remove(manifest_link) - os.symlink(os.path.basename(manifest_name), manifest_link) + os.link(manifest_name, manifest_link) bb.plain("Image CVE report stored in: %s" % manifest_name) link_path = os.path.join(deploy_dir, "%s.json" % link_name) diff --git a/meta/classes/image-artifact-names.bbclass b/meta/classes/image-artifact-names.bbclass index f5769e520f..1fe4295acb 100644 --- a/meta/classes/image-artifact-names.bbclass +++ b/meta/classes/image-artifact-names.bbclass @@ -3,10 +3,10 @@ ################################################################## IMAGE_BASENAME ?= "${PN}" -IMAGE_VERSION_SUFFIX ?= "-${DATETIME}" +IMAGE_VERSION_SUFFIX ?= "${PKGE}-${PKGV}-${PKGR}-${DATETIME}" IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH" -IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}" -IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}" +IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}" +IMAGE_LINK_NAME ?= "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}" # IMAGE_NAME is the base name for everything produced when building images. # The actual image that contains the rootfs has an additional suffix (.rootfs diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 47776db2b0..23da1de3e6 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -595,14 +595,14 @@ python create_symlinks() { return for type in subimages: dst = os.path.join(deploy_dir, link_name + "." + type) - src = img_name + imgsuffix + type - if os.path.exists(os.path.join(deploy_dir, src)): - bb.note("Creating symlink: %s -> %s" % (dst, src)) + src = os.path.join(deploy_dir, img_name + imgsuffix + type) + if os.path.exists(src): + bb.note("Creating hardlink: %s -> %s" % (dst, src)) if os.path.islink(dst): os.remove(dst) - os.symlink(src, dst) + os.link(src, dst) else: - bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src)) + bb.note("Skipping hardlink, source does not exist: %s -> %s" % (dst, src)) } MULTILIBRE_ALLOW_REP =. "${base_bindir}|${base_sbindir}|${bindir}|${sbindir}|${libexecdir}|${sysconfdir}|${nonarch_base_libdir}/udev|/lib/modules/[^/]*/modules.*|" diff --git a/meta/classes/kernel-artifact-names.bbclass b/meta/classes/kernel-artifact-names.bbclass index e77107c893..3a04b63ddd 100644 --- a/meta/classes/kernel-artifact-names.bbclass +++ b/meta/classes/kernel-artifact-names.bbclass @@ -6,8 +6,10 @@ inherit image-artifact-names -KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}" -KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}" +KERNEL_VERSION_SUFFIX ?= "${IMAGE_VERSION_SUFFIX}" + +KERNEL_ARTIFACT_NAME ?= "${MACHINE}" +KERNEL_ARTIFACT_LINK_NAME ?= "${KERNEL_ARTIFACT_NAME}${KERNEL_VERSION_SUFFIX}" KERNEL_ARTIFACT_BIN_EXT ?= ".bin" KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}" diff --git a/meta/classes/kernel-devicetree.bbclass b/meta/classes/kernel-devicetree.bbclass index b4338da1b1..e5324658a4 100644 --- a/meta/classes/kernel-devicetree.bbclass +++ b/meta/classes/kernel-devicetree.bbclass @@ -83,29 +83,34 @@ do_deploy:append() { dtb_base_name=`basename $dtb .$dtb_ext` install -d $deployDir install -m 0644 ${D}/${KERNEL_IMAGEDEST}/$dtb_base_name.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext - if [ "${KERNEL_IMAGETYPE_SYMLINK}" = "1" ] ; then - ln -sf $dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name.$dtb_ext - fi - if [ -n "${KERNEL_DTB_LINK_NAME}" ] ; then - ln -sf $dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext - fi + ln -sf $dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name.$dtb_ext for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then cat ${D}/${KERNEL_IMAGEDEST}/$type \ $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext \ > $deployDir/$type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} - if [ -n "${KERNEL_DTB_LINK_NAME}" ]; then - ln -sf $type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} \ - $deployDir/$type-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} - fi if [ -e "${KERNEL_OUTPUT_DIR}/${type}.initramfs" ]; then cat ${KERNEL_OUTPUT_DIR}/${type}.initramfs \ $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext \ > $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} - if [ -n "${KERNEL_DTB_LINK_NAME}" ]; then - ln -sf ${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} \ - $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext${KERNEL_DTB_BIN_EXT} - fi + fi + fi + done + done +} +do_deploy_links:append() { + for dtbf in ${KERNEL_DEVICETREE}; do + dtb=`normalize_dtb "$dtbf"` + dtb_ext=${dtb##*.} + dtb_base_name=`basename $dtb .$dtb_ext` + ln -vf $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext + for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do + if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ] && [ -n "${KERNEL_DTB_LINK_NAME}" ]; then + ln -vf $deployDir/$type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \ + $deployDir/$type-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext.bin + if [ -e "${KERNEL_OUTPUT_DIR}/${type}.initramfs" ]; then + ln -vf $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \ + $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext.bin fi fi done diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass index 8299b394a7..e530392dd8 100644 --- a/meta/classes/kernel.bbclass +++ b/meta/classes/kernel.bbclass @@ -782,10 +782,6 @@ kernel_do_deploy() { fi TAR_ARGS="$TAR_ARGS --owner=0 --group=0" tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip -9n > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz - - if [ -n "${MODULE_TARBALL_LINK_NAME}" ] ; then - ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz - fi fi if [ ! -z "${INITRAMFS_IMAGE}" -a x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then @@ -793,22 +789,42 @@ kernel_do_deploy() { if [ "$imageType" = "fitImage" ] ; then continue fi - initramfsBaseName=$imageType-${INITRAMFS_NAME} - install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType.initramfs $deployDir/$initramfsBaseName${KERNEL_IMAGE_BIN_EXT} - if [ -n "${INITRAMFS_LINK_NAME}" ] ; then - ln -sf $initramfsBaseName${KERNEL_IMAGE_BIN_EXT} $deployDir/$imageType-${INITRAMFS_LINK_NAME}${KERNEL_IMAGE_BIN_EXT} - fi + install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType.initramfs $deployDir/$imageType-${INITRAMFS_NAME}${KERNEL_IMAGE_BIN_EXT} done fi } -# We deploy to filenames that include PKGV and PKGR, read the saved data to -# ensure we get the right values for both -do_deploy[prefuncs] += "read_subpackage_metadata" - addtask deploy after do_populate_sysroot do_packagedata -EXPORT_FUNCTIONS do_deploy +kernel_do_deploy_links() { + deployDir="${DEPLOY_DIR_IMAGE}" + if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then + deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR}" + mkdir "$deployDir" + fi + + for imageType in ${KERNEL_IMAGETYPES} ; do + ln -vf $deployDir/$imageType-${KERNEL_IMAGE_NAME}.bin $deployDir/$imageType-${KERNEL_IMAGE_LINK_NAME}${KERNEL_IMAGE_BIN_EXT} + done + + if [ ${MODULE_TARBALL_DEPLOY} = "1" -a -f $deployDir/modules-${MODULE_TARBALL_NAME}.tgz ] ; then + ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz + fi + + if [ ! -z "${INITRAMFS_IMAGE}" -a "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then + for imageType in ${KERNEL_IMAGETYPES} ; do + if [ "$imageType" = "fitImage" ] ; then + continue + fi + ln -vf $deployDir/$imageType-${INITRAMFS_NAME}.bin $deployDir/$imageType-${INITRAMFS_LINK_NAME}${KERNEL_IMAGE_BIN_EXT} + done + fi +} +do_deploy_links[prefuncs] += "read_subpackage_metadata" + +addtask deploy_links after do_deploy before do_build + +EXPORT_FUNCTIONS do_deploy do_deploy_links # Add using Device Tree support inherit kernel-devicetree diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass index 772a11a0aa..aa21529657 100644 --- a/meta/classes/qemuboot.bbclass +++ b/meta/classes/qemuboot.bbclass @@ -161,5 +161,5 @@ python do_write_qemuboot_conf() { if qemuboot_link and qemuboot_link != qemuboot: if os.path.lexists(qemuboot_link): os.remove(qemuboot_link) - os.symlink(os.path.basename(qemuboot), qemuboot_link) + os.link(qemuboot, qemuboot_link) } diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass index 45f9ac96bc..9b56bcc536 100644 --- a/meta/classes/rootfs-postcommands.bbclass +++ b/meta/classes/rootfs-postcommands.bbclass @@ -270,7 +270,7 @@ python write_image_manifest () { if manifest_link != manifest_name: if os.path.lexists(manifest_link): os.remove(manifest_link) - os.symlink(os.path.basename(manifest_name), manifest_link) + os.link(manifest_name, manifest_link) } # Can be used to create /etc/timestamp during image construction to give a reasonably @@ -343,7 +343,7 @@ python write_image_test_data() { if testdata_link != testdata_name: if os.path.lexists(testdata_link): os.remove(testdata_link) - os.symlink(os.path.basename(testdata_name), testdata_link) + os.link(testdata_name, testdata_link) } write_image_test_data[vardepsexclude] += "TOPDIR" |