aboutsummaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/selftest/cases
AgeCommit message (Collapse)Author
2018-11-30oe-selftest: add some tests for recipeutils modulepaule/devtool-recipeutils-fixesPaul Eggleton
Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-11-27oeqa/selftest/buildoptions: Improve ccache test failure outputRichard Purdie
The current failure mode doesn't show us what the logs actually looked like and later cleans can lose them. Show the whole log in case of failure to aid debugging intermittent problems on the autobuilder. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-26oeqa/selftest/signing: Use do_populate_lic target instead of do_packageRichard Purdie
This should speed the test up signficiantly without any loss of functionality for the purposes of the test. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-26oeqa/selftest/signing: Allow tests not to need gpg on the hostRichard Purdie
We ideally don't want to use gpg from the host. This is straightforward for package management but not for sstate. For sstate, create a second build directory to run the test in using gnupg-native from the original build directory. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-26oeqa/selftest/signing: Skip tests if gpg isn't foundRichard Purdie
Raising an assertionError in the class setup isn't a particuarly good way to indicate gpg isn't installed. Instead skip the tests if the required binary isn't present. For the signing tests we do require it to be present and can't use a prebuilt one. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09oeqa/selftest/recipetool: Fix problems from changing upstream sourceMohamad Noor Alim Hussin
The upstream source tarball checksums changed. Use the copy from our source mirror to avoid failures. [YOCTO #12979] Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09image-buildinfo,oeqa/selftest/containerimage: Ensure image-buildinfo doesn't ↵Richard Purdie
break tests Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files to fail due to the presence of an unexpected file: ['./', './etc/', - './etc/build', './etc/default/', './etc/default/postinst', Tweak the class to allow it to be disabled and disable it from the test just in case it was enabled. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-06oeqa/selftest: Add test for Yocto source mirror functionality/completenessRichard Purdie
We've had a number of occasions where the Yocto Project source mirrors have not been complete or functioning correctly. This adds a test so that if this happens we find out out it sooner. It also only works over http meaning we should be able to test that anyone behind an http only proxy (no git protocol) also has functional fetches for OE-Core and layers built by the core of the project. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-21oe-selftest: devtool: Support meta being a symbolic linkPeter Kjellerstedt
oe-selftest's devtool tests have been broken since commit 2457cd57 (oe-selftest: devtool: avoid parallel races by using temporary copy of core) if meta is a symbolic link. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-16sstatetests.py: reset TCLIBCAPPENDKai Kang
It appends TCLIBCAPPEND to TMPDIR in meta/conf/distro/defaultsetup.conf: TMPDIR .= "${TCLIBCAPPEND}" It affects some oe selftest cases in sstatetests.py. Reset TCLIBCAPPEND for these cases. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-09selftest/runqemu: Handle SystemExitRichard Purdie
The sigchld handler in runqemu can raise a SystemExit when qemu shuts down. Rather than backtracing, accept this as a successful test result. ERROR: runqemu.QemuTest.test_qemu_can_shutdown (subunit.RemotedTestCase) testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 183, in test_qemu_can_shutdown qemu_shutdown_succeeded = self._start_qemu_shutdown_check_if_shutdown_succeeded(qemu, shutdown_timeout) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 175, in _start_qemu_shutdown_check_if_shutdown_succeeded time.sleep(1) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/qemurunner.py", line 100, in handleSIGCHLD raise SystemExit SystemExit Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13oeqa/selftest/recipetool: fix non-determinism in cmake testRoss Burton
The Navit recipe can depend on fribidi which is now part of oe-core. Since the pkgconfig lookup is based on pkgdata the results can change depending on what has been built, which leads to occasional failures. Build gtk+ before the test to ensure that the pkgdata is populated with more of the dependencies, and add fribidi to the generated DEPENDS checklist. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13oeqa/selftest/runtime_test: Ensure we build/use gnupg-nativeRichard Purdie
Without this, we see errors if gpg is missing from the host system for "oe-selftest -r runtime_test.TestImage.test_testimage_dnf". Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13insane: Recognise BPF as a valid EM_MACHINE typeKhem Raj
BPF Linux ELF objects are generated with kernel-selftests with >= 4.18 kernel and when clang is enabled which packages BPF objects into packages, therefore recongnise this as a valid ELF target Add a selftest for BPF Do not flag BPF objects in target, since they pretty much will be ok for most of kernels architectures we care do support BPF Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-12statetests.py: drop test_sstate_allarch_samesigs_multilibKai Kang
allarch is disabled when multilib is used, so sstate oeqa case test_sstate_allarch_samesigs_multilib is useless. Remove check for allarch part and rename to test_sstate_nativesdk_samesigs_multilib. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-04oeqa/selftest/runtime_test: Fix ↵Richard Purdie
runtime_test.TestImage.test_testimage_install after ssl changes The test installs socat and expects it to run but assumes all its dependencies are already installed (which includes libssl). Recent changes mean this isn't the case so force libssl into the image in advance to work around this issue for now. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30lib/oe/package_manager: turn postinst failure warnings into bitbake failuresAlexander Kanavin
Sumo release provides a transition period so that deferrals to first boot via 'exit 1' can be converted to pkg_postinst_ontarget(). For the next release however, postinst script failures should be treated as such. [YOCTO #12607] Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28selftest/package: Add package separated debug symbols hardlink testHongxu Jia
Tweak recipe selftest-hardlink - addition of libexecdir to simulate multiple directories - add gdb.sh to run gdb from script which is invoked at test time. - rename `hello' -> `hello1' to workaround name confliction with the one in lmbench Add test_gdb_hardlink_debug to selftest/package - run a qemu and invoke gdb.sh to gdb binaries of selftest-hardlink - check gdb to read symbols from separated debug hardlink file - check debug symbols works correctly [Test without commit `package.bbclass: only one hardlink of separated debug info file in each directory'] 2018-08-26 01:27:30,195 - oe-selftest - INFO - test_gdb_hardlink_debug (package.PackageTests) 2018-08-26 01:30:29,005 - oe-selftest - INFO - gdbtest /usr/bin/hello1 2018-08-26 01:30:36,539 - oe-selftest - INFO - gdbtest /usr/bin/hello2 2018-08-26 01:30:43,568 - oe-selftest - INFO - gdbtest /usr/libexec/hello3 2018-08-26 01:30:50,157 - oe-selftest - ERROR - No debugging symbols found. GDB result: Reading symbols from /usr/libexec/hello3...(no debugging symbols found)...done.^M (gdb) Function "main" not defined.^M Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]^M (gdb) Starting program: /usr/libexec/hello3 ^M Hello World!^M [Inferior 1 (process 320) exited normally]^M (gdb) The program is not being run.^M (gdb) 2018-08-26 01:30:51,180 - oe-selftest - INFO - ... FAIL 2018-08-26 01:30:51,181 - oe-selftest - INFO - Traceback (most recent call last): File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 148, in test_gdb_hardlink_debug self.fail('GDB %s failed' % binary) AssertionError: GDB /usr/libexec/hello3 failed [Test without commit `package.bbclass: only one hardlink of separated debug info file in each directory'] [Test with commit `package.bbclass: only one hardlink of separated debug info file in each directory'] 2018-08-26 12:40:30,976 - oe-selftest - INFO - test_gdb_hardlink_debug (package.PackageTests) 2018-08-26 12:42:15,149 - oe-selftest - INFO - gdbtest /usr/bin/hello1 2018-08-26 12:42:24,064 - oe-selftest - INFO - gdbtest /usr/bin/hello2 2018-08-26 12:42:31,078 - oe-selftest - INFO - gdbtest /usr/libexec/hello3 2018-08-26 12:42:38,646 - oe-selftest - INFO - gdbtest /usr/libexec/hello4 2018-08-26 12:42:46,824 - oe-selftest - INFO - ... ok [Test with commit `package.bbclass: only one hardlink of separated debug info file in each directory'] Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15selftest/package: Improve test to cover sparseness and hardlinking from sstateRichard Purdie
The sparseness test was sometimes working and sometimes failing depending on whether sstate was valid. This adds an explict test of sstate to the test for both hardlinking and sparseness. Tweak the test name to cover the fact its tests sparseness too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15selftest/package: Add test to ensure sparse files are preservedRichard Purdie
Add a new element to the hardlink test to check we also preseve file sparseness during the packing process. This should ensure we don't regress this issue again. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-07selftest/package: Fix hardlink test when using sstateRichard Purdie
The build target won't cause the package task to run if things were already built and in sstate. Ensure we run the package task explicitly to ensure the test works as intended. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-31oe-selftest: devtool: avoid parallel races by using temporary copy of corePaul Eggleton
Some of the devtool tests make changes to files under meta/ - legitimately since we want these tests to be working with real recipes and associated files. Unfortunately with the new oe-selftest parallelisation this can break other tests if files go missing at the wrong time (among other scenarios). To avoid this issue, simply take a copy of the core repository and use that for these tests. (We copy the entire repository since changing the path of meta/ influences COREBASE and thus we need to have things like scripts/ alongside as well). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-07-31oeqa/selftest: Add package hardlink testRichard Purdie
We keep breaking the preservation of hardlinks during the packaging process. Add a selftest which tests this to try and prevent this breaking again. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26selftest/wic: Fix/cleanup class setup codeRichard Purdie
The code was using Wic instead of WicTestCase for image_is_ready and native_sysroot which was confusing some tests. Fix image_is_ready. For native_sysroot, convert the only user to just run the call itself as there is no point in executing this for all the other tests which don't use the value. Whilst here, move _get_image_env_path to the base class alongside its parent variabe and fix it to use WicTestCase. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25selftest/bbtests: Add test for bitbake execution outside the build directoryRichard Purdie
Also fix a related test's error message to match what it does. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25selftest/bblayers: Place the test layer directory in builddirRichard Purdie
Placing the layer in meta means the directory is in an unclean state which may influence other tests. Use our build directory instead since we 'own' that. This helps keep oe-selftest parallelisation clean. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25selftest/wic: clean up runCmd error code handlingRichard Purdie
runCmd handles checking exit 0 internally unless specified otherwise and its error messages are much more useful including the failed command output. Clean up the unneeded 0 exit code checks in the wic tests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-25selftest/wic: Improve error message for test_fixed_sizeRichard Purdie
Currently this can fail with a message like 127 != 0 which is unhelpful. If we remove the ignore_status=False, the debugging from runCmd is much more helpful printing status.output. Also remove the now unneeded exit code check. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24oe-selftest: fix for changes to buildhistory-diffPaul Eggleton
Now we're not reporting the related fields (as of openembedded-core 8658b3677b9f7cb70806061c41570c709086ef05) we shouldn't expect to see PR reported here since it's not monitored by buildhistory-diff. However, with a bit of messing about we can check for the exact output that we should now see as a result of the test changing PR to go backwards. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24utils: Add multiprocess_launch API and testcaseRichard Purdie
The current methods of spawning processes for parallel execution have issues around collection of results or exceptions. Take the code from package_ipk/deb, make it generic, add a results collection mechanism, fix the exception handling and for it into a standard library function. Also add a test case which tests both the success and failure modes of operation to stop this functionality regressiing again. In particular, compared to multiprocess_exec, this fork off the parent approach means we can pass in the datastore and functions work in the same scope as the parent. This removes some of the complexities found trying to scale multiprocess_exec to wider use. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa/selftest/wic: Split tests into two class groupsRichard Purdie
This improves test parallelism. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15devtool: Split tests into multiple classesRichard Purdie
This allows better parallelism between the different tests as currently this block takes the longest time to execute. devtool tests are still all grouped into the "devtool" module for ease of exection. This also makes it easier to execute some subset of devtool tests for testing devtool changes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa: Default to buffer mode for testsRichard Purdie
Currently some tests run in buffer mode and some don't. Those that don't can corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent. If there is useful output on stdout/stderr, it will be displayed if the test fails. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa/selftest/wic: Ensure initramfs exists for test_iso_imageRichard Purdie
AssertionError: Command 'wic create mkhybridiso --image-name core-image-minimal -o /var/tmp/wic.oe-selftest/' returned non-zero exit status 1: ERROR: _exec_cmd: gzip -f -9 -c /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio > /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio.gz returned '1' instead of 0 output: gzip: /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio: No such file or directory This is because in a clean build directory, the initramfs may not be rebuilt. Add a call to ensure it is built to avoid the error. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa/selftest/wic: Use a subdir of builddir, not /var/Richard Purdie
Using /var/ leave wic open to races with other processes on the system, use a subdir of builddir instead to avoid this. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa/selftest/devtool: Ensure dbus is built befoe running testRichard Purdie
If dbus isn't build first the dbus dependency of dbus-wait can't be detected through pkgconfig and the test fails: AssertionError: {'DEPENDS': {'dbus'}} != {} - {'DEPENDS': {'dbus'}} + {} : Some expected variables not found in recipe: {'DEPENDS': {'dbus'}} Ensure dbus is built and present in the sysroot. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-15oeqa/selftest/package: Use setUpLocal, not setUpRichard Purdie
The main setUp function needs to be called and tests are meant to define setUpLocal. For some reason this one didn't leading to errors with recent code changes. Fix it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-13oqqa/selftest/lic_checksum: Avoid cache reparseRichard Purdie
By having a consistent config for the test, a second reparse is avoided and the test runs slightly faster. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-13oeqa/selftest/image_tyypedep: Avoid undefined variable failureRichard Purdie
If something goes wrong with the test, dep can be undefined. Avoid a traceback for this and handle it more gracefully. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-07rootfs-postcommands: split ssh_allow_empty_passwordJackie Huang
"allow root login" should not be bundled in ssh_allow_empty_password, because some distro may want only one of "allow root login" and "allow empty password", so split it out into ssh_allow_root_login and add new imagefeature allow-root-login so they can be controlled separately, debug-tweaks will still include both of them. Signed-off-by: Jackie Huang <jackie.huang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02oe-selftest: Add bitbake-layer create-layer testJoshua Watt
Adds a test that creates a layer, adds it to bblayers.conf, then ensure that it appears properly in bitbake-layers show-layers. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-27oeqa: add Git mirror testRoss Burton
Add an automated test to exercise that fetching a git: recipe... 1) works over git without mirrors 2) fails without git connectivity or mirrors 3) works without git connectivity but with a mirror (2) is done by setting GIT_PROXY_COMMAND to 'false' which should break any git network operations. [ YOCTO #12805 ] Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-06-07oeqa: replace Alex Kanavin's @intel email address with a personal oneAlexander Kanavin
As I will be leaving Intel, this address will no longer be valid, so swap it for my personal one for now. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-04btrfs-tools: update to 4.15.1Alexander Kanavin
Drop upstreamed 0001-Fix-build-with-musl-missing-header-include-for-dev_t.patch Add ftw-subdir-walk.patch as it resolves the RECIPE_NO_UPDATE_REASON. Add --disable-zstd as libzstd isn't provided in oe-core. Fix wic testcase, as the minimal fs size is now bigger. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-20oeqa/selftest/runqemu: qemu nfs testing not ready for deploymentRichard Purdie
This test shouldn't have merged yet since we don't run portmap/rpcbind on the autobuilder infrastructure and the test therefore cannot succeed. We need to document this, set it up, then enable the test. The test itself is fine and good to have so its left in the code but disabled for now. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-19oe-selftest: runqemu: add tests for qemu boot and shutdownYeoh Ee Peng
QA team were testing qemu boot image and shutdown on each qemu architecture manually. Add automated test to test qemu boot on ext4 and nfs, finally check that it can shutdown properly. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-04-07devtool: Ensure added layer sets LAYERSERIES_COMPATRichard Purdie
Now that we see warnings if LAYERSERIES_COMPAT is unset, the auto generated code from devtool needs to set this to avoid warnings which break various tests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-12oe-selftest: add a test for failing package post-installation scriptletsAlexander Kanavin
The test runs a scriptlet that has an intentionally failing command in the middle and checks for two things: 1) that bitbake does warn the user about the failure 2) that scriptlet execution stops at that point. The test is run for all three package types: rpm, deb, ipk. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-06selftests: Add test case for booting a generic EFI boot partition imageCalifornia Sullivan
Simple test case that adds 'efi' to MACHINE_FEATURES, sets WKS_FILE to "efi-bootdisk.wks.in", installed required boot items, and attempts to boot the wic image. Quick check to make sure that the feature actually works. Signed-off-by: California Sullivan <california.l.sullivan@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24sstatetests: pass the right files when comparing sigfilesRoss Burton
Signed-off-by: Ross Burton <ross.burton@intel.com>