summaryrefslogtreecommitdiffstats
path: root/meta/lib/oe
AgeCommit message (Collapse)Author
2018-12-26base.bbclass, classextend.py: Drop catering to gcc-initialKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15oe: Fix opkg status list parse - Missing postinstRaul Martins
While parsing opkg package status, last package status was not properly handled, resulting in final image without postinst and pkg depends Signed-off-by: Raul Martins <raul.martins@alta-rt.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15lib/oe/elf.py: Add powerpc64 architecture definition for muslSerhey Popovych
Add the ELF definition for the powerpc64 architecture when building with musl as libc. Signed-off-by: Serhey Popovych <serhe.popovych@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15lib/oe/recipeutils: Add a new function to mimic do_checkpkgRichard Purdie
The code in distrodata.bbclass related to the do_checkpkg task is rather dated, has holes in it (ignoring some recipes) and has horrible locking and csv related issues. We should use modern APIs such as tinfoil to make the calls we need directly against bitbake, cutting out the middleman and clarifing the code. This change imports the bits of distrodata.bbclass that are needed by the automated upgrade helper (AUH) into a standalone function which uses the tinfoil API. This can then be used by AUH and by the tests in oeqa/selftest/distrodata as well as by any other standalone script that needs this functionality. Its likely it can be further improved from here but this is a good start and appears to function as before, with slightly wider recipe coverage as some things skipped by distrodata are not skipped here (images, pieces of gcc, nativesdk only recipes). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-14lib/oe/utils: Set stderr for host_gcc_version()Robert Yang
Fixed: $ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc $ rm -fr tmp/hosttools/ && bitbake -p [snip] ERROR: Error running gcc --version: It didn't print the error message, now it is: ERROR: Error running gcc --version: ccache: error: Could not find compiler "gcc" in PATH For the error itself, it is because ccache is not in my HOSTTOOLS, so this is an expected error. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05lib/oe/utils: Improve multiprocess_lauch exception handlingRichard Purdie
We've seen a cryptic: "ERROR: Fatal errors occurred in subprocesses, tracebacks printed above" message from oe-selftest with no other traceback information. Improve the traceback logging to try and give a better indication of any errors that is ocurring. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-03lib/oe/recipeutils: drop obsolete functionsPaul Eggleton
These date from the time before Tinfoil's API covered this functionality (back when you could actually access cooker from a tinfoil-based script). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-03lib/oe/recipeutils: patch_recipe(): fix replacing varflag valuesPaul Eggleton
The code here wasn't correctly getting the variable history for varflags, so for example if you did a devtool upgrade on a recipe where the SRC_URI checksums were in the .inc file (typical for python recipes in order to support both python 2 and 3) then after the upgrade the new values would be set in the recipe and the old values were left in the .inc, which is not right. Teach the code here how to get the history for varflags so it works properly. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-03lib/oe/recipeutils: patch_recipe(): fix handling of values across ↵Paul Eggleton
includes/classes If we were setting a variable and part of the variable's value was being set in a class or a .inc file, we were still just setting the value outright on the assumption that it was too hard to do otherwise. With some careful use of the variable history we can do better for certain situations i.e. when the recipe does not currently set the value outright. Additionally, correctly remove _appends for variables we are changing if we're trying to remove the value added in the _append. Fixes [YOCTO #12623] and partially fixes [YOCTO #9360]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22terminal: Cope with unreleased versions of tmuxMike Crowe
When tmux is built from a non-release Git version, its version number is "next-X" where X appears to be the expected version number for the next release. For example, when built from the current state of master, running "tmux -V" yields: tmux next-2.9 Currently check_tmux_pane_size only checks for the version being less than 1.9, so it seems unfair to fail with an obscure Python error in this case. Let's just use the version number after the "next-" prefix if it is present. Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-21package_manager.py: fix the message used to catch failing postinsts from dnfAlexander Kanavin
Latest dnf versions have tweaked it. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16lib/oe/package_manager: Avoid race problems when calling list_pkgs()Richard Purdie
list_pkgs() for rpm calls RpmPM() which would try and create a copy of the package feed. This can be called for example from buildhistory whilst some other task may be working on an SDK or image construction, causing the package feed to disappear part way through an operation. Avoid the need to copy the package index just to list the installed packages, avoiding the race. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-13locale: Allow usage of cross-localedef for ARCAlexey Brodkin
With this it's possible to build locale data for ARC and not do it instead on the first boot. Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-28package_manager.py: correct the deploydir when packagefeed-stability inheritedMingli Yu
After create_packages_dir added in below commit: 85e72e1 package_manager: Filter to only rpms we depend upon When add below line into conf/local.conf INHERIT += "packagefeed-stability" There comes below error when do_rootfs Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies): [snip] bb.utils.remove(subrepo_dir, recurse=True) [snip] In create_packages_dir function, there is a logic as bb.utils.remove(subrepo_dir, recurse=True) to clean subrepo_dir which is actually as example is /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm. But currently when inherit packagefeed-stability class, the deploydir should be /$Prj/tmp/deploy/rpm-prediff, not the default /$Prj/tmp/deploy/rpm. If use /$Prj/tmp/deploy/rpm, then result in the logic as below: os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm") Update to the actual deploydir to guarantee the logic as below: os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm") Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18package_manager: handle remove() with no packagesRoss Burton
If remove() is called with an empty package list, ensure we do nothing instead of calling the underlying package manager with an invalid command line. [ YOCTO #12900 ] Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-10-18package_manager: implement PACKAGE_EXCLUDE for opkgRoss Burton
opkg has supported the --add-exclude option to install since 0.3.0, so use it to implement support for PACKAGE_EXCLUDE. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18rootfs: possibility to add packages to debug rootfsAndrej Valek
Static libraries are not included in rootfs, it means, that sources are not going into debug rootfs. This option enables to install additional packages even if the standard package is not installed. Signed-off-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11lib/oe/utils: add eol to format_pkg_list()grygorii tertychnyi via Openembedded-core
Append '\n' to the non-empty formatted string before return. If you write it to the (manifest) file, it will ensure file ends with a newline. Many GNU utilities have problems processing the last line of a file if it is not '\n' terminated. E.g. if the last line is not terminated by a newline character, then "read" will read it but return false, leaving the broken partial line in the read variable(s). It can also break or adversely affect some text processing tools, that operate on the file. Signed-off-by: grygorii tertychnyi <gtertych@cisco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25rootfs.py: Install -src packages when PACKAGE_DEBUG_SPLIT_STYLE is ↵Joshua Watt
"debug-with-srcpkg" When the source packages are created, they need to be included in the debug image created when IMAGE_GEN_DEBUGFS = "1" for it to be usable for debugging. [YOCTO #12930] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-25rootfs: always update the opkg indexIoan-Adrian Ratiu
The previous logic assumed that if $BUILD_IMAGES_FROM_FEEDS=1 then a complete set of ipk feeds from which to build the image is already present under $IPK_FEED_URIS at do_rootfs runtime. $IPK_FEED_URIS usually contains "file://${DEPLOY_DIR_IPK}" which renders the above assumption bad because some recipes in the current build can contain code like do_install[nostamp] = "1" which will cause rebuilds bumping $PR and invalidating the index. Even when the index is manually re-created before an image build ("bitbake package-index"), the nostamp will cause failures because the dependency gets rebuilt before do_rootfs in the "bitbake <image>" call. So make the opkg rootfs index logic the same as for rpm/deb, to always update the index in $DEPLOY_DIR_IPK to fix the above nostamp failure. Feeds outside $DEPLOY_DIR_IPK added to $IPK_FEED_URIS continue to work as usual, for eg. by using a http:// URI. Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-21lib/oe/elf.py: Add LatticeMico32 architecture definitionNathan Rossi
Add the ELF definition for the LaticeMico32 architecture. This architecture is 'elf' OS only as it does not target Linux. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-09-21package_manager: use normalized path when doing the filtered copyMax Krummenacher
The linking/copying of the package files failes if the deploy dir is set in a non normalized way e.g. like this DEPLOY_DIR = "${TOPDIR}/../deploy" Then the simple string replacement which is used to calculated the link destination from the link source fails, as the link source is normalized but the deploydir prefix is not. Normalizing deploydir fixes this. Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-09-13insane: Recognise BPF as a valid EM_MACHINE typeKhem Raj
BPF Linux ELF objects are generated with kernel-selftests with >= 4.18 kernel and when clang is enabled which packages BPF objects into packages, therefore recongnise this as a valid ELF target Add a selftest for BPF Do not flag BPF objects in target, since they pretty much will be ok for most of kernels architectures we care do support BPF Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-12oe/utils.py: Add vartrue()Robert Yang
It can be used to simplify code like: "${@['iffalse', 'iftrue'][var]}" Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05populate_base_sdk: Stop running gcc --version all the timeRichard Purdie
Running 'gcc --version' for every image recipe is slow and increases parsing time/resource usage for no good reason. Only compute the value in when we're really running the task/function. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05package_manager.py: add noarch to buildarch_compatKai Kang
It fails to run rpmbuild to build a noarch package on target when it contains 'BuildArch: noarch' in the spec file: | error: No compatible architectures found for build Add 'noarch' to buildarch_compat in configure file rpmrc to fix it. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30lib/oe/package_manager: turn postinst failure warnings into bitbake failuresAlexander Kanavin
Sumo release provides a transition period so that deferrals to first boot via 'exit 1' can be converted to pkg_postinst_ontarget(). For the next release however, postinst script failures should be treated as such. [YOCTO #12607] Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23lib/oe/patch.py: Clean up getstatusoutput usageRobert Yang
We can't use subprocess.check_output() or subprocess.call() here since the one who invokes runcmd() needs handle CmdError() exception (error out or ignore it). Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23lib/oe/gpg_sign.py: Clean up getstatusoutput usageRobert Yang
Replace usage of oe.utils.getstatusoutput() with direct subprocess calls. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23lib/oe/utils: Fix get_multilib_datastore to work for original tuneRichard Purdie
Currently the original datastore returned by this function doesn't always work as the tune isn't set back to the original. Fix it to work like all_multilib_tune_list() in utils.bbclass and correct the data returned. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23lib/oe/package.py: use bb.utils.break_hardlinks helperRasmus Villemoes
This does the same thing, but is more efficient in case st_nlinks is (already) 1. Depends on bitbake commit 7ae93cf40ab91965147055100432961436bce46c . Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-16oe.package_manager: support loading intercepts from multiple pathsChristopher Larson
- if POSTINST_INTERCEPTS is set, use the listed intercept files, or - if POSTINST_INTERCEPTS_PATH is set, load from the listed paths, or - if POSTINST_INTERCEPTS_DIR is set, load from it (for compatibility), or - load from ${COREBASE}/meta/postinst-intercepts Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15package_manager: Add definitions for riscv machinesRichard Purdie
Add definitions to the locale table for riscv architecture. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15sstate/lib.oe.path: Ensure file sparseness is preservedRichard Purdie
Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15package_manager/sdk: Use filtered copies of the deploy ipk/deb directoriesRichard Purdie
Similar to rpm, use copies of the ipk/deb directories for rootfs construction. This means the image creation code can no longer "see" recipes wich aren't in its dependency chain which is good for a variety of reasons including determinism, incompatible recipe (e.g. systemd/sysvinit) package conflicts and locking performance. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15package_manager: Remove rpm specific pieces of create_packages_dir()Richard Purdie
This function is generic, tweak the variable names and move out the rpm specific directory name to make it truly generic and reusable for deb/ipk. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15image: Add locale archive optimisationRichard Purdie
Refactor the locale archive function from the SDK to also make it work during general image creation. This reduces the size of the locales from 900MB to 220MB in core-image-lsb-sdk. The exception handling around subprocess was dropped as the standard subprocess exception printing is better handled than the catchall exception. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-10lib/oe: Fix collections ABCs DeprecationWarning in Python 3.7+Khem Raj
- Prefer collections.abc (new in Python 3.3) over collections for abstract base classes - In Python 3.8, the abstract base classes in collections.abc will no longer be exposed in the regular collections module. This will help create a clearer distinction between the concrete classes and the abstract base classes." - https://docs.python.org/3.7/whatsnew/3.7.html#deprecated - see https://github.com/python/cpython/commit/c66f9f8d3909f588c251957d499599a1680e2320 Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-09package: skip strip on signed kernel modulesfoocampo
Executing strip action on kernel modules removes the signature. Is not possible to strip and keep the signature, therefore avoid strip signed kernel modules. Signed-off-by: Omar Ocampo <omar.ocampo.coronado@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01multiconfig: Enable multiconfig dependencies on oe-coreAlejandro Enedino Hernandez Samaniego
This patch enables multiconfig dependencies (mcdepends) to be used on recipes using the following format: task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on" For the sake of simplicity consider the following example: Assuming we have set up multiconfig builds, one for qemux86 and one for qemuarm, named x86 and arm respectively. Adding the following line to an image recipe (core-image-sato): do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" Would state that core-image-sato:do_image from x86 will depend on core-image-minimal:do_rootfs from arm so it can be executed. This patch makes modifications to bitbake.conf to enable mcdepends, and to sstatesig and staging.bbclass to avoid conflicts between packages from different multiconfigs. [YOCTO #10681] Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26lib/oe: split out machine to ELF data dictionaryRoss Burton
2018-07-26logging: use warning instead warnChen Qi
The warn method is deprecated. We should use the documented warning instead. Quoting from the python's official doc: """ Note: There is an obsolete method warn which is functionally identical to warning. As warn is deprecated, please do not use it - use warning instead. """ Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-07-24lib/oe/utils: Drop now unused multiprocess_execRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24package/package_manager: multiprocess_exec -> multiprocess_launchRichard Purdie
After this replacement, the parent exception handling works so we don't need subprocess wrapping with bb.error in the underlying functions. The underlying contexts also have better module handling so the imports can be cleaned up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24package: Call file to determine elf status in parallelRichard Purdie
This allows the calls to is_elf (which calls file) to happen in parallel allowing a speedup of do_package and do_populate_sysroot for native recipes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24package: Refactor to remove isElf/is_elf function duplicationRichard Purdie
There are probably further cleanups needed here but this at least removes the major code duplication between these two similar funcitons, keeping the kernel module ".ko" extension check for efficiency to avoid opening and reading file contents in the general case. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24package: Don't use subshell to execute fileRichard Purdie
We don't need any functionality from the shell here, its just extra fork overhead. Therefore remove it and use subprocess directly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24utils: Add multiprocess_launch API and testcaseRichard Purdie
The current methods of spawning processes for parallel execution have issues around collection of results or exceptions. Take the code from package_ipk/deb, make it generic, add a results collection mechanism, fix the exception handling and for it into a standard library function. Also add a test case which tests both the success and failure modes of operation to stop this functionality regressiing again. In particular, compared to multiprocess_exec, this fork off the parent approach means we can pass in the datastore and functions work in the same scope as the parent. This removes some of the complexities found trying to scale multiprocess_exec to wider use. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-18classes/buildhistory: handle packaged files with names containing spacesPaul Eggleton
The FILELIST field of the package info file in the buildhistory repository is a space-separated list of all of the files in the package. If a name of a file packaged by a recipe contains a space character then of course the result was that we didn't handle its name properly. To fix that, use quotes around any filename containing spaces and at the other end use these quotes to extract the proper entries. Fixes [YOCTO #12742]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-07-18lib/oe/buildhistory_analysis: drop related field featurePaul Eggleton
The original idea here was that changes to certain fields might be able to be explained if there was a change to another field, for example if RDEPENDS changed it might be because DEPENDS changed. Thus we were printing this kind of thing out with each change. Unfortunately in practice this turned out to be noisy and not particularly useful, so we might as well remove it. Fixes [YOCTO #7336]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>