path: root/docs/packaged-staging.xml

<?xml version='1.0'   encoding='utf-8'?>
<!--
  ex:ts=4:sw=4:sts=4:et
  -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-->
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
                      "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<book>
    <bookinfo>
        <title>OpenEmbedded Packaged-Staging Project</title>
        <authorgroup>
            <corpauthor>Koen Kooi</corpauthor>
        </authorgroup>
        <copyright>
            <year>2006</year>
            <holder>Koen Kooi</holder>
        </copyright>
    </bookinfo>	

<chapter>
<title>Introduction</title>

<para>This branch will hold the code for the Google Summer of Code 2006 project "Putting staging under packagemanagement"</para>

<para>The goal is to populate (and de-populate) the staging area using packages OE generates instead of manually poking .h and .so files into it.</para>

<para>First we need to identify three classes of packages:
<itemizedlist>
	<listitem><para>'native'  - These packages run on the buildsystem (e.g. quilt-native)</para></listitem>
	<listitem><para>'cross'   - These packages run on the buildsystem, but creates things for the target system (e.g gcc-cross)</para></listitem>
	<listitem><para>'regular' - These packages run on the targetsystem (e.g. glibc)</para></listitem>
</itemizedlist>
</para>
<para>
And some terminology:
<itemizedlist>
	<listitem><para>'current metadata' 	- the metadata found in the org.openembedded.dev branch</para></listitem>
	<listitem><para>'the branch' 		- the org.openembedded.packaged-staging branch</para></listitem>
</itemizedlist>
</para>
</chapter>

<chapter>
<title>Installing packages</title> 
<para>
If we have packages that can be installed into the staging area we have two modes of operation:
            <itemizedlist>
                <listitem><para>Start over from scratch</para></listitem>
                <listitem><para>Incrementally install and remove packages</para></listitem>
             </itemizedlist>
</para>

<section>
	<title>Start over from scratch</title>
<para>
For each to be built package we will repopulate the staging directory from scratch. The benefits are a clean environment and avoids packages picking stuff up at random. Conflicts when installing packages can only happen when two different providers are trying to install the same file. This can only be a bug of the OpenEmbedded meta data. This approach will not require any conflict handling or resolution
</para>
</section>

<section>
	<title>Incrementally installing packages</title>
<para>In contrast to repopluating the staging area from scratch we install additional dependencies and remove conflicting packages. The installing and removing of packages is assumed to be faster than repopulating from scratch. Once a package completed we could consider removing the non-native depends to avoid a growing staging directory. One issue is with the clean task. We could assume that cleaning a package will remove it from the staging area as well. There is one possible problem with it. Let us assume we want to clean quilt-native, virtually every package DEPENDS on it, we would have to clean the staging area completely. If this is the wished behaviour needs to be discussed. 
</para>
</section>
</chapter>

<chapter>
	<title>Implementation</title>
<section>
	<title>Rearranging tasks</title>
<para>In the current metadata packaging happens after staging:
<screen>
============================================================
--- classes/base.bbclass	18c5b851966c94c02ada5f5c8563f8d59ecb0972
+++ classes/base.bbclass	97972c3da5a46e53731eca2d4935d1f7ed45b5aa
@@ -593,7 +593,7 @@ base_do_compile() {
 }

-addtask stage after do_compile
+addtask stage after do_install


============================================================
--- classes/package.bbclass	35e8e063abe6888f5dbc0a4e6862c7ce05e6e10d
+++ classes/package.bbclass	5eeb3a5518350fcf30706cbe34aadc9bae46c360
@@ -726,4 +726,4 @@ EXPORT_FUNCTIONS do_package do_shlibs do
 do_package[dirs] = "${D}"
 populate_packages[dirs] = "${D}"
 EXPORT_FUNCTIONS do_package do_shlibs do_split_locales mapping_rename_hook
+addtask package before do_stage after do_install
-addtask package before do_build after do_stage
</screen>

I haven't been able to spot any troubles with this change.
</para>
</section>

<section>
	<title>Keeping track of what regular packages OE generates</title>
<para>OE is able to do debian-style shared lib renaming (fontconfig -> libfontconfig0.ipk) and saves the mappings in staging/pkgmaps. The pkgmaps file is lacking some vital info like versions however. The resulting algo is pretty neat:
<screen>
PACKAGEFUNCS += "do_write_ipk_list"

python do_write_ipk_list () {
        import os, sys
        ipkdir = bb.data.getVar('DEPLOY_DIR_IPK', d, 1)
        stagingdir = bb.data.getVar('STAGING_DIR', d, 1)
        tmpdir = bb.data.getVar('TMPDIR', d, 1)
        p = bb.data.getVar('P', d, 1)
        pr = bb.data.getVar('PR', d, 1)

        packages = bb.data.getVar('PACKAGES', d, 1)
        if not packages:
                bb.debug(1, "PACKAGES not defined, nothing to package")
                return

        if packages == []:
                bb.debug(1, "No packages; nothing to do")
                return

        # Generate ipk.conf if it or the stamp doesnt exist
        listfile = os.path.join(stagingdir,"pkgmaps","%s-%s.spawn" %  ( p , pr ))
        os.system('mkdir -p ' + stagingdir + '/pkgmaps')
        if not os.access(listfile, os.R_OK):
                os.system('rm -f ' + listfile)
                f = open(listfile,"w")
                for spawn in packages.split():
                        #check if the packagename has changed due to debian shlib renaming
                        localdata = bb.data.createCopy(d)
                        pkgname = bb.data.getVar('PKG_%s' % spawn, localdata, 1)
                        if not pkgname:
                                pkgname = spawn
                        f.write("%s\n" % pkgname)
                f.close()
}
</screen>

So now OE will spit out a spawnfile for each revision of a recipe, instead of one per recipe as the pkgmap code does.
</para>
</section>

<section>
	<title>Installing regular packages</title>
<para>
Now we know what OE generates we can install those:
<screen>
PSTAGE_INSTALL_CMD      = "ipkg-cl install -force-depends -f ${DEPLOY_DIR_PSTAGE}/ipkg.conf -o "
PSTAGE_UPDATE_CMD       = "ipkg-cl update -f ${DEPLOY_DIR_PSTAGE}/ipkg.conf -o "
SPAWNFILE               = "${STAGING_DIR}/pkgmaps/${P}-${PR}.spawn"
SPAWNIPK                = ${spawn}
STAGING_BASEDIR         = "${STAGING_LIBDIR}/.."


#check for generated packages
 if [ -e ${SPAWNFILE} ]; then
         oenote "List of spawned packages found: ${P}.spawn"
         for spawn in `cat ${SPAWNFILE} | grep -v locale` ; do \
                 if [ -e ${DEPLOY_DIR_IPK}/${spawn}_* ]; then
                         ${PSTAGE_INSTALL_CMD} ${STAGING_BASEDIR} ${spawn}
                         # clean up .la files to avoid having references to the builddirs in the binaries
                         for lafile in ${STAGING_LIBDIR}/*.la ; do \
                                 sed -i s:installed=yes:installed=no:g ${lafile}
                         done
                 else
                         oenote "${spawn} not found, probably empty package"
                 fi
         done
         exit 0
 else
         oenote "Spawn file not found!"
 fi
</screen>
</para>
</section>


<section>
	<title>Handing native packages</title>
<para>

In its current state OE doesn't package the native packages it generates, so we'll have to write something that does:

<screen>
DEPLOY_DIR_PSTAGE       = "${DEPLOY_DIR}/pstage"

PSTAGE_BUILD_CMD        = "${IPKGBUILDCMD}"
PSTAGE_INSTALL_CMD      = "ipkg-cl install -force-depends -f ${DEPLOY_DIR_PSTAGE}/ipkg.conf -o "
PSTAGE_PKGNAME          = "staging-${PN}_${PV}-${PR}_${PACKAGE_ARCH}.ipk"

STAGING_BASEDIR         = "${STAGING_LIBDIR}/.."

do_stage_prepend() {

#assemble appropriate ipkg.conf
if [ -e ${DEPLOY_DIR_PSTAGE}/ipkg.conf ]; then
        rm ${DEPLOY_DIR_PSTAGE}/ipkg.conf
fi

ipkgarchs="${BUILD_ARCH} all any noarch ${TARGET_ARCH} ${IPKG_ARCHS} ${IPKG_EXTRA_ARCHS} ${MACHINE}"
    priority=1
    for arch in $ipkgarchs; do
      echo "arch $arch $priority" >> ${DEPLOY_DIR_PSTAGE}/ipkg.conf
      priority=$(expr $priority + 5)
    done
echo "src oe file:${DEPLOY_DIR_IPK}" >> ${DEPLOY_DIR_PSTAGE}/ipkg.conf
export OLD_PWD=`pwd`
cd ${DEPLOY_DIR_IPK} &amp;&amp; rm *${BUILD_ARCH}.ipk -f ; ipkg-make-index -p Packages . ; cd ${OLD_PWD}
${PSTAGE_UPDATE_CMD} ${STAGING_BASEDIR}

if [ ${PN} != "glibc-intermediate" ] ; then

        if [ -e ${DEPLOY_DIR_PSTAGE}/${PCROSS_PKGNAME} ]; then
                oenote "Cross stuff already packaged, using that instead"
                ${PSTAGE_INSTALL_CMD} ${CROSS_DIR}  ${DEPLOY_DIR_PSTAGE}/${PCROSS_PKGNAME}
        fi

        if [ -e ${DEPLOY_DIR_PSTAGE}/${PSTAGE_PKGNAME} ]; then
                oenote "Staging stuff already packaged, using that instead"
                ${PSTAGE_INSTALL_CMD} ${STAGING_DIR}  ${DEPLOY_DIR_PSTAGE}/${PSTAGE_PKGNAME}
                exit 0
        fi

        touch ${TMPDIR}/moved-staging
        mv ${STAGING_DIR} ${TMPDIR}/pstage

        if [ -e ${CROSS_DIR} ]; then
                mv ${CROSS_DIR} ${TMPDIR}/pcross
                touch ${TMPDIR}/moved-cross
        fi

        mkdir -p ${STAGING_BINDIR}
        mkdir -p ${STAGING_LIBDIR}
        mkdir -p ${STAGING_INCDIR}
        mkdir -p ${STAGING_DATADIR}/aclocal
else
        oenote "Glibc-intermediate detected!"
fi #if !glibc intermediate
}
</screen>

The above gets prepended to the do_stage() method specified in the recipe. I works by rotating away the staging dir so the next step can easily find the added files:

<screen>
do_stage_append() {

if [ ${PN} != "glibc-intermediate" ] ; then
        mkdir -p ${DEPLOY_DIR_PSTAGE}

        #make a package for staging
        mkdir -p ${STAGING_DIR}/CONTROL

        echo "Package: staging-${PN}"           >  ${STAGING_DIR}/CONTROL/control
        echo "Version: ${PV}-${PR}"             >> ${STAGING_DIR}/CONTROL/control
        echo "Description: ${DESCRIPTION}"      >> ${STAGING_DIR}/CONTROL/control
        echo "Section: ${SECTION}"              >> ${STAGING_DIR}/CONTROL/control
        echo "Priority: Optional"               >> ${STAGING_DIR}/CONTROL/control
        echo "Maintainer: ${MAINTAINER}"        >> ${STAGING_DIR}/CONTROL/control
        echo "Architecture: ${PACKAGE_ARCH}"    >> ${STAGING_DIR}/CONTROL/control
        echo "Source: ${SRC_URI}"               >> ${STAGING_DIR}/CONTROL/control

        ${PSTAGE_BUILD_CMD} ${STAGING_DIR} ${DEPLOY_DIR_PSTAGE}
        rm -rf ${STAGING_DIR}
       #move back stagingdir so we can install packages
        mv ${TMPDIR}/pstage ${STAGING_DIR}
        rm ${TMPDIR}/moved-staging

        ${PSTAGE_INSTALL_CMD} ${STAGING_DIR}  ${DEPLOY_DIR_PSTAGE}/${PSTAGE_PKGNAME}
else
        oenote "Glibc-intermediate detected (again)"
fi #if !glibc-intermediate
}
</screen>

A package will fail to stage if it needs files from staging during its own staging. We can solve that, and the ugly check for glibc-intermediate:

<screen>
... stage stuff...
ipkg list_installed | awk '{print $1}' > installed-list
cat installed-list | xargs ipkg -recursive -force-depends remove
... make a package of the remaing bits ...
cat installed-list | xargs ipkg install
ipkg install newly-packaged-stuff
</screen>

</para>
</section>
 

</chapter>

<chapter>
	<title>Conclusions</title>
<section>
	<title>Current state</title>
<para>
The metadata in the packaged-staging branch well, once it's done building the toolchains. The infamous 'bitbake nano' testcase works as well as other packages present in the packaged-staging branch.
</para>
</section>

<section>
	<title>How to proceed from here</title>
<para>
First the gcc-cross problem has the be fixed, with the solution mentioned in the previous chapter. After that a org.openembedded.dev.packaged-staging branch should be created where patched will be applied on top of org.openembedded.dev to test 'bitbake angstrom-bootstrap-image'. Once that passes all tests succesfully, it can be propagated back to org.openembedded.dev.
</para>
</section>


</chapter>

</book>