summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMartin Jansa <Martin.Jansa@gmail.com>2018-07-10 19:36:10 +0000
committerMartin Jansa <Martin.Jansa@gmail.com>2020-09-17 18:36:28 +0200
commit2e75bbc1aff478846fbab7ed59d33192c5a9537b (patch)
tree8d3afbdd459321a011b031e65ce3125a6853dc7a
parent176e1ead31a38150026a953ba3ceac46db3bced1 (diff)
downloadopenembedded-core-contrib-jansa/artifacts.tar.gz
openembedded-core-contrib-jansa/artifacts.tar.bz2
openembedded-core-contrib-jansa/artifacts.zip
image*.bbclass, kernel*.bbclass: create versioned hard links instead of version-less symlinksjansa/artifacts
* We used to create the actual artifact files with some version in the filename and then created symlink without any version which was updated to point to the latest one created. In some scenarios it's useful to create all artifacts - typically rootfs and kernel images with the same version - like release build even when the kernel itself wasn't modified since the previous release. If we include the release version in the regular _NAME variables then we'll need to re-run do_deploy and do_image which will cause kernel to be rebuilt and image to be re-created even when the only change since last build was the version number. With this change we can re-use kernel and image from sstate when nothing was changed and run only very fast do_deploy_links task which just adds another hard link to existing artifact from sstate. * This is already used by various LGE builds as do_webos_deploy_fixup() https://github.com/webosose/meta-webosose/blob/master/meta-webos/classes/webos_deploy.bbclass but injecting this task in all the right places id difficult and sometimes requires whole bbclass to be duplicated. Having simpler way of versioning artifacts directly in oe-core might be useful for others. * move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME that way e.g. kernel.do_deploy can be reused from sstate to provide "version-less" artifacts and then very fast do_deploy_links task just adds links with consistent suffixes (by default the version from the recipe but could be easily set to e.g. some release name when building some products). * create hard links instead of symlinks, so that whatever version the filename says is really there * some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file to be removed first or they might either append or update the content of the image instead of creating new image file from scratch - I have seen this only with one proprietary format we generate with our own tool, so hopefully this isn't very common * this is basically the mechanism are using in webOS with WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from jenkins job and then all artifacts (images as well as corresponding kernel files) have the same version string) * without this, you can still easily set the variables to contain the version from jenkins job (excluded from sstate signature like DATETIME currently is to prevent rebuilding it everytime even when the content didn't change) but then when kernel is reused from sstate you can have version 1.0 used on kernel artifacts and 2.0 on image artifacts. * if you don't exclude the version string with vardepsexclude, then you get the right version in the filenames but for cost of re-executing do_deploy every single time, which with rm_work will cause all kernel tasks to be re-executed (together with everything which depends on it like external modules etc). * the implementation "from outside" is a bit tricky as shown in webOS OSE, because first you need to reverse the meaning of IMAGE_NAME and IMAGE_LINK_NAME like here, but also replace all symlinks with hardlinks and then adjust all recipes/bbclasses to depend on our do_deploy_fixup task instead of the original do_deploy see the variable modifications: https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65 and then various bbclasses to hook do_webos_deploy_fixup task creating the hardlinks for possible artifacts: https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/kernel.bbclass https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/image.bbclass so hopefully with all these changes in oe-core other project can achieve the same just by setting one variable IMAGE_VERSION_SUFFIX * drop ${PKGE}-${PKGV}-${PR} from kernel artifacts names (this is the latest build) and add it only in hardlinks created in do_deploy_links so that we can use PKGR there again (because these links are generally used only by human operators and they don't have their own TASKHASH or the IMAGE_VERSION_SUFFIX might be set to some release name which they do understand * this allows to drop package_get_auto_pr from kernel do_deploy as well, leaving only 2 EXTENDPRAUTO bumps for each kernel build (do_package and do_deploy_links, unfortunatelly these will still have different value, so if you're looking for the exact kernel image in deploy directory based on kernel image package version seen on the device the EXTENDPRAUTO part of PR will be different). [YOCTO #12937] Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
-rw-r--r--meta/classes/cve-check.bbclass4
-rw-r--r--meta/classes/image-artifact-names.bbclass6
-rw-r--r--meta/classes/image.bbclass10
-rw-r--r--meta/classes/kernel-artifact-names.bbclass6
-rw-r--r--meta/classes/kernel-devicetree.bbclass21
-rw-r--r--meta/classes/kernel.bbclass46
-rw-r--r--meta/classes/qemuboot.bbclass2
-rw-r--r--meta/classes/rootfs-postcommands.bbclass4
8 files changed, 66 insertions, 33 deletions
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 02fef7c205..3093fc81d8 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -133,10 +133,10 @@ python cve_check_write_rootfs_manifest () {
if manifest_name and os.path.exists(manifest_name):
manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)
- # If we already have another manifest, update symlinks
+ # If we already have another manifest, update hardlinks
if os.path.exists(os.path.realpath(manifest_link)):
os.remove(manifest_link)
- os.symlink(os.path.basename(manifest_name), manifest_link)
+ os.link(manifest_name, manifest_link)
bb.plain("Image CVE report stored in: %s" % manifest_name)
}
diff --git a/meta/classes/image-artifact-names.bbclass b/meta/classes/image-artifact-names.bbclass
index 3ac8dd731a..fbc4ae17e2 100644
--- a/meta/classes/image-artifact-names.bbclass
+++ b/meta/classes/image-artifact-names.bbclass
@@ -3,10 +3,10 @@
##################################################################
IMAGE_BASENAME ?= "${PN}"
-IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
+IMAGE_VERSION_SUFFIX ?= "${PKGE}-${PKGV}-${PKGR}-${DATETIME}"
IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
-IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
-IMAGE_LINK_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
+IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}"
+IMAGE_LINK_NAME ?= "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}"
# IMAGE_NAME is the base name for everything produced when building images.
# The actual image that contains the rootfs has an additional suffix (.rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 730c843c18..a27c2b1dbb 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -586,14 +586,14 @@ python create_symlinks() {
return
for type in subimages:
dst = os.path.join(deploy_dir, link_name + "." + type)
- src = img_name + imgsuffix + type
- if os.path.exists(os.path.join(deploy_dir, src)):
- bb.note("Creating symlink: %s -> %s" % (dst, src))
+ src = os.path.join(deploy_dir, img_name + imgsuffix + type)
+ if os.path.exists(src):
+ bb.note("Creating hardlink: %s -> %s" % (dst, src))
if os.path.islink(dst):
os.remove(dst)
- os.symlink(src, dst)
+ os.link(src, dst)
else:
- bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src))
+ bb.note("Skipping hardlink, source does not exist: %s -> %s" % (dst, src))
}
MULTILIBRE_ALLOW_REP =. "${base_bindir}|${base_sbindir}|${bindir}|${sbindir}|${libexecdir}|${sysconfdir}|${nonarch_base_libdir}/udev|/lib/modules/[^/]*/modules.*|"
diff --git a/meta/classes/kernel-artifact-names.bbclass b/meta/classes/kernel-artifact-names.bbclass
index a65cdddb3e..03656e9c59 100644
--- a/meta/classes/kernel-artifact-names.bbclass
+++ b/meta/classes/kernel-artifact-names.bbclass
@@ -6,8 +6,10 @@
inherit image-artifact-names
-KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
-KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}"
+KERNEL_VERSION_SUFFIX ?= "${IMAGE_VERSION_SUFFIX}"
+
+KERNEL_ARTIFACT_NAME ?= "${MACHINE}"
+KERNEL_ARTIFACT_LINK_NAME ?= "${KERNEL_ARTIFACT_NAME}${KERNEL_VERSION_SUFFIX}"
KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}"
KERNEL_IMAGE_LINK_NAME ?= "${KERNEL_ARTIFACT_LINK_NAME}"
diff --git a/meta/classes/kernel-devicetree.bbclass b/meta/classes/kernel-devicetree.bbclass
index 81dda8003f..b58fe356d5 100644
--- a/meta/classes/kernel-devicetree.bbclass
+++ b/meta/classes/kernel-devicetree.bbclass
@@ -74,19 +74,32 @@ do_deploy_append() {
install -d $deployDir
install -m 0644 ${D}/${KERNEL_IMAGEDEST}/$dtb_base_name.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext
ln -sf $dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name.$dtb_ext
- ln -sf $dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext
for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then
cat ${D}/${KERNEL_IMAGEDEST}/$type \
$deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext \
> $deployDir/$type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin
- ln -sf $type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \
- $deployDir/$type-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext.bin
if [ -e "${KERNEL_OUTPUT_DIR}/${type}.initramfs" ]; then
cat ${KERNEL_OUTPUT_DIR}/${type}.initramfs \
$deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext \
> $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin
- ln -sf ${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \
+ fi
+ fi
+ done
+ done
+}
+do_deploy_links_append() {
+ for dtbf in ${KERNEL_DEVICETREE}; do
+ dtb=`normalize_dtb "$dtbf"`
+ dtb_ext=${dtb##*.}
+ dtb_base_name=`basename $dtb .$dtb_ext`
+ ln -vf $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext $deployDir/$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext
+ for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
+ if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then
+ ln -vf $deployDir/$type-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \
+ $deployDir/$type-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext.bin
+ if [ -e "${KERNEL_OUTPUT_DIR}/${type}.initramfs" ]; then
+ ln -vf $deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext.bin \
$deployDir/${type}-${INITRAMFS_NAME}-$dtb_base_name-${KERNEL_DTB_LINK_NAME}.$dtb_ext.bin
fi
fi
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 48135b3d41..1b8478af02 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -716,10 +716,8 @@ kernel_do_deploy() {
fi
for imageType in ${KERNEL_IMAGETYPES} ; do
- baseName=$imageType-${KERNEL_IMAGE_NAME}
- install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType $deployDir/$baseName.bin
- ln -sf $baseName.bin $deployDir/$imageType-${KERNEL_IMAGE_LINK_NAME}.bin
- ln -sf $baseName.bin $deployDir/$imageType
+ install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType $deployDir/$imageType-${KERNEL_IMAGE_NAME}.bin
+ ln -sf $imageType-${KERNEL_IMAGE_NAME}.bin $deployDir/$imageType
done
if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
@@ -731,8 +729,6 @@ kernel_do_deploy() {
fi
TAR_ARGS="$TAR_ARGS --owner=0 --group=0"
tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip -9n > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
-
- ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
fi
if [ ! -z "${INITRAMFS_IMAGE}" -a x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
@@ -740,20 +736,42 @@ kernel_do_deploy() {
if [ "$imageType" = "fitImage" ] ; then
continue
fi
- initramfsBaseName=$imageType-${INITRAMFS_NAME}
- install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType.initramfs $deployDir/$initramfsBaseName.bin
- ln -sf $initramfsBaseName.bin $deployDir/$imageType-${INITRAMFS_LINK_NAME}.bin
+ install -m 0644 ${KERNEL_OUTPUT_DIR}/$imageType.initramfs $deployDir/$imageType-${INITRAMFS_NAME}.bin
done
fi
}
-# We deploy to filenames that include PKGV and PKGR, read the saved data to
-# ensure we get the right values for both
-do_deploy[prefuncs] += "read_subpackage_metadata"
-
addtask deploy after do_populate_sysroot do_packagedata
-EXPORT_FUNCTIONS do_deploy
+kernel_do_deploy_links() {
+ deployDir="${DEPLOY_DIR_IMAGE}"
+ if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then
+ deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR}"
+ mkdir "$deployDir"
+ fi
+
+ for imageType in ${KERNEL_IMAGETYPES} ; do
+ ln -vf $deployDir/$imageType-${KERNEL_IMAGE_NAME}.bin $deployDir/$imageType-${KERNEL_IMAGE_LINK_NAME}.bin
+ done
+
+ if [ ${MODULE_TARBALL_DEPLOY} = "1" -a -f $deployDir/modules-${MODULE_TARBALL_NAME}.tgz ] ; then
+ ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
+ fi
+
+ if [ ! -z "${INITRAMFS_IMAGE}" -a "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
+ for imageType in ${KERNEL_IMAGETYPES} ; do
+ if [ "$imageType" = "fitImage" ] ; then
+ continue
+ fi
+ ln -vf $deployDir/$imageType-${INITRAMFS_NAME}.bin $deployDir/$imageType-${INITRAMFS_LINK_NAME}.bin
+ done
+ fi
+}
+do_deploy_links[prefuncs] += "read_subpackage_metadata"
+
+addtask deploy_links after do_deploy before do_build
+
+EXPORT_FUNCTIONS do_deploy do_deploy_links
# Add using Device Tree support
inherit kernel-devicetree
diff --git a/meta/classes/qemuboot.bbclass b/meta/classes/qemuboot.bbclass
index 830363dcef..263c823ecb 100644
--- a/meta/classes/qemuboot.bbclass
+++ b/meta/classes/qemuboot.bbclass
@@ -140,5 +140,5 @@ python do_write_qemuboot_conf() {
if qemuboot_link != qemuboot:
if os.path.lexists(qemuboot_link):
os.remove(qemuboot_link)
- os.symlink(os.path.basename(qemuboot), qemuboot_link)
+ os.link(qemuboot, qemuboot_link)
}
diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index 6fa979e345..1b5cc60683 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -271,7 +271,7 @@ python write_image_manifest () {
manifest_link = deploy_dir + "/" + link_name + ".manifest"
if os.path.lexists(manifest_link):
os.remove(manifest_link)
- os.symlink(os.path.basename(manifest_name), manifest_link)
+ os.link(manifest_name, manifest_link)
}
# Can be used to create /etc/timestamp during image construction to give a reasonably
@@ -343,7 +343,7 @@ python write_image_test_data() {
testdata_link = os.path.join(deploy_dir, "%s.testdata.json" % link_name)
if os.path.lexists(testdata_link):
os.remove(testdata_link)
- os.symlink(os.path.basename(testdata_name), testdata_link)
+ os.link(testdata_name, testdata_link)
}
write_image_test_data[vardepsexclude] += "TOPDIR"