summaryrefslogtreecommitdiffstats
path: root/meta/lib
AgeCommit message (Collapse)Author
2017-09-01package_manager: Clean up horrible 'None' return valuesRichard Purdie
If this fails the exception will now be raised. Lets use that and drop all this 'None' return value ugliness. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-01lib/oe/utils: Handle exceptions in multiprocess_execRichard Purdie
Currently exceptions that happen in pool commands are ignored. Any errors would be printed on the console but everything else is silent. Switch to use pool.map_async which allows for an error_callback which we can use to detect exceptions and make sure these errors are handled. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-01lib/oe/package: Improve filedeprunner subprocess handlingRichard Purdie
Currently the exit code of the spawned program isn't checked so it can fail and the do_package task will continue merrily upon its way. Use subprocess.check_output() to ensure we check the exit code and redirect stderr to stdout so if it fails, we see the error output. We can then drop the existing exception handling as the subprocess exception gives a much better error. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31selftest/prservice: Adapt to BB_SERVER_TIMEOUTRichard Purdie
In the memory resident mode, the user may not see a message about the server starting, it would be in the cookerdeamon logfile. We don't need this to test the server is functioning correctly so just drop the test. Add in an extra check that the file we expected to be created was created when exporting PR values. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31oe-selftest: tinfoil: add a test for variable historyPaul Eggleton
I recently found that variable history wasn't working properly for recipes when we enable history tracking, resulting in minor functionality loss in devtool upgrade, so add a test to ensure this doesn't regress now that it's fixed. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31oe-selftest: devtool: test find-recipe and edit-recipePaul Eggleton
We weren't testing the devtool find-recipe and edit-recipe subcommands, with the result that when they regressed recently we didn't notice. Add some code into the test_devtool_add to test this (since we need a recipe in the workspace, and adding a new test with all that preamble would seem a bit excessive for these simple checks). Also take the opportunity to refactor the test a little bit so that the recipe name and version are variables rather than hardcoding them everywhere. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31devtool: upgrade: workaround for recipes which apply patches conditional ↵Paul Eggleton
upon class If we're upgrading a recipe that appends additional patches for, say, class-native, and we're just upgrading the target variant, then when we copied the recipe into the workspace we skipped copying the additional patches for the native variant. This caused warnings because the workspace recipe is preferred. Look at SRC_URI for all variants when copying files to work around this. More work is needed to make it easier to work with recipes that use BBCLASSEXTEND where you need to build more than one variant at once, but this at least fixes the immediate ugliness. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31devtool: upgrade: fix handling of non-absolute pathsPaul Eggleton
If your BBLAYERS has non-absolute paths in it (e.g. "${COREBASE}/../something") then none of the paths matched in copy_recipe_files() with the result that no files got copied and you ended up with an error later on because the recipe file couldn't be found at the destination. Fix this as well as adding an explicit check to see if no files got copied - error out earlier if so. Fixes [YOCTO #10981]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-30package_manager.py: improve reproducibilityJuro Bystricky
When creating Packages.gz, do not save the time stamp in the compressed file metadata. Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-30signing-keys: do not use DISTRO_VERSION in key namesAlexander Kanavin
DISTRO_VERSION may contain the current date, and so is prone to mismatches particularly when keys are created on one date, and dnf is configured to use the keys on another date. [YOCTO #11983] Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-29qemurunner: Tweak qemu pid starting timeout codeRichard Purdie
We're seeing timeouts in the autobuilder testing code. Increase these timeouts to 120, print the length of time we're having to wait, change the error messages to really be errors and don't print empty logs, its not helpful, print a message about the empty log instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27wic: setlftest: test expanding MBR imageEd Bartosh
Added test_expand_mbr_image test case to the wic oe-selftest suite. The test expands directdisk wic image to 1Gb target, checks if it's expanded correctly and boots it in qemu to make sure the image is bootable, i.e. bootloader, kernel, boot and root partitions are still functional. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27terminal.py: add support for mate-terminal in check_terminal_version()Andre McCurdy
Not currently used, but include for completeness. Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27terminal.py: fix devshell with mate-terminalAndre McCurdy
Without the --disable-factory option, mate-terminal fails to start with the error: | There was an error creating the child process for this terminal | Failed to execute child process "oe-gnome-terminal-phonehome" (No such file or directory) The --disable-factory option was removed by: http://git.openembedded.org/openembedded-core/commit/?id=e8dca725ed8211a874472300a3ed50e494039ab9 apparently based on an assuption that mate-terminal continues to track gnome-terminal since forking from it. However, based on the mate-terminal man page in the upstream master branch, the option is still supported: https://github.com/mate-desktop/mate-terminal Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27terminal.py: avoid 100% cpu while waiting for phonehome pid fileAndre McCurdy
Some of the less common terminal types haven't been tested with the recent phonehome pid file changes and there may be error cases where the pid file is never created. Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27package.py: strip_execs: Support for .ko modulesTobias Hagelborn
* Support stripping of .ko modules verifying file extension and check of content "vermagic=" Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27package.py: Fix some lint errorsTobias Hagelborn
- rename type to exec_type not to shadow type - rename isELF is_elf Signed-off-by: Tobias Hagelborn <tobiasha@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24core/target/ssh.py: use reader to handle partial dataRobert Yang
This can avoid UnicodeDecodeError error. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24gpg_sign: depress the owership warnings on homedirJia Zhang
A gpg command, e.g, called by get_gpg_version(), may trigger such a warning: gpg: WARNING: unsafe ownership on homedir '/home/user/.gnupg' This sort of warning is harmless so it can be depressed through specifying --no-permission-warning. Signed-off-by: Jia Zhang <lans.zhang2008@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23core/target/ssh.py: replace decode errorsRobert Yang
There might be wild strings when read from target (especially when reading ptest results), replace the errors to avoid breaking the test. Fixed: (Not always happen) $ bitbake core-image-sato -ctestimage [snip] status, output = self.target.run('ptest-runner', 0) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run status, output = self._run(sshCmd, processTimeout, True) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run status, output = SSHCall(command, self.logger, timeout) File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall run() File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run output = process.communicate()[0].decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23utils/logparser.py: fix section checkRobert Yang
The section might be None or '', so use "if section" for it. Fixed: File "/buildarea/lyang1/poky/meta/lib/oeqa/utils/logparser.py", line 113, in log_as_files prefix += section TypeError: Can't convert 'NoneType' object to str implicitly [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23runtime/cases/_ptest.py: rename it to ptest.pyRobert Yang
It works now. [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23runtime/cases/_ptest.py: add skip statusRobert Yang
The packages' test cases maybe skipped, check and save them. [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23oeqa/utils/logparser.py: add skip statusRobert Yang
Some test cases maybe skipped, let's parse it. [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23runtime/cases/_ptest.py: revive itRobert Yang
* Make it work with current oeqa * Skip the test if ptest is not in DISTRO_FEATURES * Skip the test if ptest-pkgs is not in IMAGE_FEATURES * The logs are saved to: testimage/ptest_log -> testimage/ptest_log.<datetime> * This provides data that could be used to detect regressions in ptest results [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23runqemu: Use virtio to mount cdrom drivesRichard Purdie
The IDE driver in the kernel is fragile and in 4.12 is causing backtraces. To unblock 4.12 kernel merging use the virtio CD driver instead to mount iso images which should be faster and more stable. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23oeqa/parselogs: Ignore device node warnings on mipsRichard Purdie
THe newner kernels have ope firmware support enabled which leads to warnings during boot on mips. These aren't interesting and we should ignore them. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23oe/path.py: copyhardlinktree: don't overwrite existing symlinksIoan-Adrian Ratiu
Before GNU tar 1.24, only the archive creation command had the '-h' argument to preserve and follow symlinks. After >= 1.24 via commit 14efeb9f956e38d7be (tar: --dereference consistency) the capability to preserve symlinks was also added to the archive extraction command. -h is default at archive creation but is not default at extraction, meaning that it will replace symlinks with directories even if the original filesystem directory tree and archive contains them. Add -h to the copyhardlinktree extraction step so the build can support symlinks in variables like ${DEPLOY_DIR_IPK/RPM/DEB}. Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-19devtool/copy_buildsystem: adds meta-skeleton layer in the eSDK installation.Juan M Cruz Alcaraz
The eSDK installation requires the meta-skeleton layer. The build system might use the meta-skeleton recipes as layout to create custom recipes. An example is the recipetool script that uses the meta-skeleton kernel recipe when creating a custom kernel recipe. [YOCTO #11102] Signed-off-by: Juan M Cruz Alcaraz <juan.m.cruz.alcaraz@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18copy_buildsystem: include layer tree during build structure creationAndrej Valek
When buildsystem with layer structure is going to be copied, only the last meta-XXX layer is taken. For example, during ext_sdk bblayers creating: layers/oe/meta \ layers/oe/meta-oe \ layers/oe/meta-networking \ layers/oe/meta-webserver \ ... It restructured meta-oe, meta-networking,... contents into meta-oe. Recipes from meta-oe will be on the same level like meta-networking, meta-webserver, ... . It should take the whole meta path instead of the last one. layers/oe/meta \ layers/oe/meta-oe/meta-oe \ layers/oe/meta-oe/meta-networking \ layers/oe/meta-oe/meta-webserver \ ... Now the directory structure is the same like during build creation. Signed-off-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Pascal Bach <pascal.bach@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18sign_rpm: Allow pkg signing by chunks through RPM_GPG_SIGN_CHUNKLeonardo Sandoval
Commit d58b1d196 moved from chunk to serial signing, but neither of both approaches allowed the user to select the chunks size. This patch allows the user to select a chunk size through RPM_GPG_SIGN_CHUNK defaulting to BB_NUMBER_THREADS, considered a good default. Indirectly, this change reduces the number of processes spawn to number-of-packages/RPM_GPG_SIGN_CHUNK. Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16package.bbclass: Restore functionality to detect RPM dependenciesPeter Kjellerstedt
During the transition to dnf and rpm4, the functionality to automatically make RPM determine dependencies was lost. Before the transition, an OE specific tool called rpmdeps-oecore had been added to the rpm suit. It was based on the rpmdeps tool that is part of rpm. For each file specified on its command line, it would output the provides and requires that RPM could determine. During the transition to rpm4, rpmdeps-oecore was replaced with the standard rpmdeps. However, what no one noticed was that unless rpmdeps is given options, e.g., -P or -R, to tell it what it should output, it will not output anything. Thus, it would do all the work to determine the requirements, but would keep silent about it. And since no output from rpmdeps is expected unless there are requirements, there were no warnings indicating that everything was not working as expected. Porting the old rpmdeps-oecore to work with rpm4 is not really possible since it relied on being able to access internals of RPM that are no longer available. However, it turned out that rpmdeps had a debug option, --rpmfcdebug, that would output exactly the information that we need, albeit in a different format and to stderr. To make this usable, rpmdeps has now received a new option, --alldeps, which sends the information we need to stdout. (From OE-Core rev: 958501b3d9201aaabb81ec644c6049e0c9b737e7) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16gpg_sign: perform rpm signing seriallyLeonardo Sandoval
gpg signing in file batches (which was default to 100) is a memory expensive computation, causing trouble in some host machines (even on production AB as seen on the bugzilla ID). Also, in terms of performance, there is no real gain when rpm signing is done in batches. Considering the latter issues, perform the rpm signing serially. Log showing errors observed recently at AB workers: | gpg: signing failed: Cannot allocate memory | gpg: signing failed: Cannot allocate memory | error: gpg exec failed (2) | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/core2-64-poky-linux/base-passwd/3.5.29-r0/deploy-rpms/core2_64/base-passwd-dev-3.5.29-r0.core2_64.rpm: [YOCTO #11914] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16dnf: expand dnf selftest to test signed package feedsMarkus Lehtonen
[YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16oeqa: fix dnf testsMarkus Lehtonen
Rename one dnf runtime test that it will recognized as a python module and thus also found by the oe test loader. Also, fix value of TEST_SUITES in dnf selftest so that all test dependencies are satisfied and the runtime test may be successfully run from there. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16package_manager.py: enable dnf's repo_gpgcheck if feed signing is enabledMarkus Lehtonen
If package feed signing is enabled enable repo gpg signature check for rpm repositories added via PACKAGE_FEED_URIS. This has the implication that all repositories added via this mechanism must be signed with the same key. [YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16lib/oe/package_manager: re-implement rpm feed signingMarkus Lehtonen
[YOCTO #11209] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16selftest/tinfoil: Filter LogRecords in test_event_waitRichard Purdie
As the code stands today, an event mask does not mask LogRecord events since the log levels are controlled separately. We therefore need to accept (and ignore) LogRecord events in this test to avoid errors. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16selftest: fix distrodata.py to use per-recipe UPSTREAM_VERSION_UNKNOWN settingAlexander Kanavin
... instead of a global exception list which was problematic. [YOCTO #11896] Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-11sstatetests: limit the number of signature comparisons when differLeonardo Sandoval
For perfomance reasons, limit the number of signature comparisons when stamps differ. The limit set is hardcoded to 20. [YOCTO #11651] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-11oeqa/buildoptions: remove unsafe-references-in-script testRoss Burton
This QA test is about to be deleted, so remove it from selftest. Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-11context: Include a command line argument to run all except certain testsLeonardo Sandoval
A new command line argument (-R, which is the oposite of current -r) that allows to run all test cases except the ones indicated through the command line. Some command line examples: * Run all except the distro test case: $ oe-selftest -R distrodata * Run all except the archiver test case and a single bblayers unit test $ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove [YOCTO #11847] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-11oeqa/runtime_test: use subtests in test_postinst_rootfs_and_bootRoss Burton
As this test has two nested loops and actually runs six times, use UnitTest.subTest() so we can tell which instance is failing, and to run all variations instead of failing on the first one. Also set PACKAGE_CLASSES to just the type we need to reduce the verboseness of the output, and consolidate the feature generation to be neater. Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-09action: new bitbake-layer plugin to create a simple layerLeonardo Sandoval
Though the script bitbake-layers (from the bitbake project), this plugin creates a simple layer with a example recipe, the latter with a single task (do_build). Layer's license and priority is MIT and 6, respectively. Example recipe and layer's priority can be specified through the command line. [YOCTO #11567] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-09package_manager: Fix support for NO_RECOMMENDATONSRichard Röjfors
When support for dnf was introduced the check of the no NO_RECOMMENDATIONS variable got broken. This fixes the issue by compairing to the string "1" rather than the number 1. Signed-off-by: Richard Röjfors <richard@puffinpack.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-08terminal: wait for terminal task to finish before proceddingLeonardo Sandoval
This commit generalizes the work done in [1] and [2], both fixing issues on several areas (the former with -c patch and gnome-terminal and the latter with -c menuconfig and several terminals, including gnome-terminal and tmux). The main idea is to get the PID from the new spawned terminal and loop there until finished. [1] 76e8ab47c9: terminal: Fix gnome-terminal to work with recent versions [2] 7d02ea283b: cml1.bbclass: wait until menuconfig terminal finishes Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-08sstatetests: Use python function instead of bitbake-diffsigs scriptLeonardo Sandoval
Using a python function instead of launching a subprocess fasten the diffsigs computation. [YOCTO #11651] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30imagefeatures: Add a test for various hypervisor formatsTom Rini
We add a new test that will create core-image-minimal for wic.vmdk, wic.vdi and wic.qcow2. We then confirm via qemu-img that the resulting file is of the expected type. Cc: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30imagefeatures: Add a test for many CONVERSION_CMDs being chainedTom Rini
Add a new test to create a long (and not otherwise useful) image, ext4.bmap.gz.bz2.lzo.xz.u-boot and also the sha256sum of it. Check that the resulting sha256sum is valid. Cc: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30imagefeatures: Extend test_bmap to validate gzipTom Rini
We extend the existing test_bmap test to also produce an ext4.bmap.gz file and then have gzip confirm that it contains valid gzip data. This tests that we are able to chain at least 2 CONVERSION_CMDs together. Cc: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>