summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa
AgeCommit message (Collapse)Author
2020-08-25oeqa: runtime_tests: Extra GPG debuggingJoshua Watt
This patch enables extra debugging for gpg to try and get more debugging information when [YOCTO #14003] occurs Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-22wic: Add 512 Byte alignment to --offsetJoshua Watt
Allows the --offset argument to use the "s" or "S" suffix to specify that it is reporting the number of 512 byte sectors. This is required for some SoCs where the mask ROM looks for an item at a sector that isn't aligned to a 1KB boundary. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-20oeqa/manual/bsp-hw.json : remove X_server_can_start_up_with_runlevel_5_boot testTeohJayShen
remove the test as graphical mode is already being validated Signed-off-by: TeohJayShen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-20oeqa/manual/bsp-hw.json : remove shutdown_system testTeohJayShen
remove the test as shutdown function is already being validated Signed-off-by: TeohJayShen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-18buildgalculator: Patch to fix build with -fno-commonKhem Raj
We need to apply a patch to galculator for it to build with gcc-10+ Remove double definition of 'prefs' variable Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-18buildcpio.py: Apply patch to fix build with -fno-commonKhem Raj
This sed expression implements the needed patch to fix compilation with -fno-common, hopefully this patch will get included in 2.14 release and we can remove this operation Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-18selftest/tinfoil: Increase wait event timeoutRichard Purdie
We're seeing this on the autobuilder when IO load is high. Increase the timeout to give this a better chance of working out ok since there is no particular reason we only need to wait 5s and searching for files is IO sensitive. [YOCTO #14001] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-08oeqa: write @OETestTag content into json test reports for each caseAlexander Kanavin
This allows using these tags for classification and filtering of test results according to various organization-specific criteria, such as teams responsible for the test, internal test ids, feature domains and so on. Test name itself meanwhile can stay short and human-readable. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-08oeqa/selftest/imagefeatures: Add testcase for fitImageUsama Arif
This testcase generates the Image Tree Source and the corresponding fitImage containing a kernel and a ramdisk. It then checks if the these files exist and if the right fields are present in the right order in the Image Tree Source. Tested with: oe-selftest -r imagefeatures.ImageFeatures.test_fit_image Signed-off-by: Usama Arif <usama.arif@arm.com> Cc: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-06ltp: make copyFrom scp command non-fatalMatthew
[YOCTO #13802] Make the scp failure non-fatal so the ltp tests continue to run and the rest of the logs will be available to see afterwards. Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27deb: Move package manager to its own dirFredrik Gustafsson
This is a part of a refactor that will split the package manager code so that it's possible to use other package managers in other layers. RP: Fixes to parse/build Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27ipk: Move package manager to its own dirFredrik Gustafsson
This is part of a refactor that will split the package manager code so that it's possible to use other package managers in other layers. RP: Fixes to parse/build Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27rpm: Move package manager to its own dirFredrik Gustafsson
This is part of a refactor that will split the package manager code so that it's possible to use other package managers in other layers. RP: Fixes to parse/build Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-22oeqa/qemurunner: Add priority/nice information for running processesRichard Purdie
We're seeing failures due to system load. In theory we've set process nice levels which should compensate for this. Add debugging so we can find out if they're being correctly applied. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-14oeqa/utils/qemurunner: Fix missing pid file tracebacksRichard Purdie
One element of the error message guarded against None as a value but I missed the other, fix this. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-13qemurunner: Add extra debug info when qemu fails to startRichard Purdie
When qemu fails to start we're struggling to work out why. Add more debug info which can at least confirm/rule out various things. This code is only on the error handling path and more info shoudl help us debug issues. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-13qemurunner: Ensure pid location is deterministicRichard Purdie
The pid location could vary due to changes in cwd as only a filename is specified, not a full path. This in theory could be resulting in some of our autobuilder failures. Whilst its difficult to know if this is causing a problem, Using a full path removes any question of such an issue. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12oeqa/selftest/sstatetests: Avoid polluting DL_DIRRichard Purdie
Every selftest run is currently polluting the autobuilder DL_DIR. Avoid this by using a temporary directory for the test which is cleaned up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-08oeqa/core/loader: fix regex to include numbersVacek, Patrick
The previous version only included the numbers 1 and 2 in the allowed characters for the module name. In the past, this was (\w+) so all numbers were allowed. Now it explicitly includes all numbers again. Signed-off-by: Patrick Vacek <patrickvacek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03logrotate.py: fix testimage occasionally failureChangqing Li
testcase test_systemd_failed occasionally failed with below error: Failed to start Rotate log files. logrotate.service: Failed with result 'exit-code'. logrotate.service: Main process exited, code=exited, status=1/FAILURE error: stat of /var/log/logrotate_test failed: No such file or directory error: logrotate_test:1 lines must begin with a keyword or a filename (possibly in double quotes) above failure caused since testcase test_logrotate_wtmp add /etc/logrotate.d/logrotate_test, which need /var/log/logrotate_test, but there is no such file. so when logrotate.service is triggerd by logrotate.timer after testcase test_logrotate_wtmp is runned, the testcase test_systemd_failed will fail. these 3 lines are useless, so remove them to fix above problem. Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03oeqa/targetcontrol: Attempt to fix log closure warning messageRichard Purdie
We continue to see the warning message: WARNING: lib/bb/daemonize.py:76: ResourceWarning: unclosed file <_io.TextIOWrapper name='build/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200703011821' mode='a' encoding='UTF-8'> I've been unable to reprodue this but believe its caused by garbage collection of the FileHandler used in QemuTarget being delayed until after a new tinfoil instance is created by a subseqent test. Force the log file to be closed when we stop using it to avoid this. [YOCTO #13961] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03oeqa/selftest: recipetool/devtool: Avoid load_plugin test raceRichard Purdie
This bug has plagued the autobuilder for a couple of years and we've struggled to reproduce/debug it. The problem is the "lib" directory in meta-poky used during the load_plugin tests for recipetool and devtool can race and one can delete the files from the other leading to test failures. Deleting the lib directory only if empty will avoid this. [YOCTO #13070] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03oeqa/core/loader: refine regex to find moduleLee Chee Yang
test case in format <module name>.<class name>.<test case name> this is clear when test cases is only 3 item deep. but confused when it is 4 item deep, eg, oelib.types.TestList.test_list_nosep in this case, oelib and oelib.types can both be treated as module since module name contains only lower cases and class name should contain atleast one upper case. so, always treat leading item without upper case as module also allow module name to contain dot. [YOCTO #13941] Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02oescripts: ignore whitespaces when comparing linesChen Qi
list-packageconfig-flags.py determines the whitespace numbers according to the longest package name. This is reasonable for the tool as it's trying to generate pretty output. However, in this selftest case, the output is hardcoded. This results in "pinentry gtk2 libcap ncurses qt secret" not recognized as correct as the expected line is: "pinentry gtk2 libcap ncurses qt secret". The difference is only about whitespaces. So we should ignore the whitespaces when comparing lines. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02oescripts.py: fix typoChen Qi
Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28oeqa/selftest: Clean up separate builddir in success case when non-threadedRichard Purdie
If oe-selftest is run without -j, the separate build directory "build-st" isn't cleaned up afterwards. Mirror the behaviour of the -j option to handle this the same way, only preserve upon failure. To do this, the remove function needs to be moved to the selftest context module so that it can be accessed without requiring the testtools and subunit modules the -j option requires. A dummy wrapper class is used to wrap the tests and clean up afterwards. [YOCTO #13953] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-23oeqa/runtime/cases/ptest: Make output content path absoluteAndrej Valek
The output content is created in current directory, because json content has no defined absolute path to WORKDIR as in bitbake. Signed-off-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-23oeqa/utils/command: Improve stdin handling in runCmdRichard Purdie
Occasionally we've been seeing leftover threads from runCmd. The stdin test assumes we clean up all threads but the code assumes that the daemonic thread can be left behind. The issue can be reproduced by adding a time.sleep(10) to the end of writeThread() which will mean it stays resident past the end of the command. We may as well add it to the threads list and clean it up properly, hopefully removing the race in the tests from the autobuilder. [YOCTO #13055] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-23oeqa/selftest/runcmd: Add better debug for thread count mismatch failuresRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-17oeqa/runtime: Add OERequirePackage decoratorKonrad Weihmann
Add new decorator which behaves like OEHasPackage, but fails the testcase if a dependency isn't met. This helps to identify missing packages in the image under test when using static test suite lists, otherwise a missing package won't fail the overall test suite and errors might slip through unnoticed Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-12buildhistory: Add simplistic file move detectionRichard Purdie
We'd like to use buildhistory more during patch review however its proving hard, particularly where whole subtrees of files move, such as a kernel version upgrade, or where a software module moves include directory. This adds file rename matching which covers our common case of library moves, kernel upgrades and more. A new test case is also added so that someone in the future can change the code and test the logic is still doing the expected things. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10wic: Fix --extra-space argument handlingJoshua Watt
467f84e12b ("wic: Add --offset argument for partitions") broke the --extra-space argument handling in wic. Fix the option and add a unit test for the argument. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-06archiver: Speed up testsPaul Barker
When running tests locally without the advantage of a populated sstate cache, the archiver tests can take an unnecessarily long time. Using different target recipes can speed things up a lot by reducing the number of dependencies that are built. For the test case test_archiver_srpm_mode we can still test the conditions described in bug #11121 by using a target recipe that is known to inherit the nopackages class as well as a more normal recipe, instead of having to compute all the depepndencies of core-image-sato. Before: archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name: PASSED (20.50s) archiver.Archiver.test_archiver_filters_by_type: PASSED (14.16s) archiver.Archiver.test_archiver_filters_by_type_and_name: PASSED (79.08s) archiver.Archiver.test_archiver_mode_configured: PASSED (558.19s) archiver.Archiver.test_archiver_mode_diff: PASSED (14.59s) archiver.Archiver.test_archiver_mode_dumpdata: PASSED (14.41s) archiver.Archiver.test_archiver_mode_mirror: PASSED (14.02s) archiver.Archiver.test_archiver_mode_mirror_combined: PASSED (19.99s) archiver.Archiver.test_archiver_mode_mirror_excludes: PASSED (14.24s) archiver.Archiver.test_archiver_mode_mirror_gitsm: PASSED (13.47s) archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow: PASSED (22.15s) archiver.Archiver.test_archiver_mode_original: PASSED (14.15s) archiver.Archiver.test_archiver_mode_patched: PASSED (14.57s) archiver.Archiver.test_archiver_mode_recipe: PASSED (14.56s) archiver.Archiver.test_archiver_srpm_mode: PASSED (83.03s) oe-selftest () - Ran 15 tests in 911.905s After: archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name: PASSED (23.66s) archiver.Archiver.test_archiver_filters_by_type: PASSED (15.16s) archiver.Archiver.test_archiver_filters_by_type_and_name: PASSED (15.33s) archiver.Archiver.test_archiver_mode_configured: PASSED (15.94s) archiver.Archiver.test_archiver_mode_diff: PASSED (15.79s) archiver.Archiver.test_archiver_mode_dumpdata: PASSED (15.39s) archiver.Archiver.test_archiver_mode_mirror: PASSED (15.28s) archiver.Archiver.test_archiver_mode_mirror_combined: PASSED (21.22s) archiver.Archiver.test_archiver_mode_mirror_excludes: PASSED (15.11s) archiver.Archiver.test_archiver_mode_mirror_gitsm: PASSED (13.61s) archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow: PASSED (23.21s) archiver.Archiver.test_archiver_mode_original: PASSED (14.87s) archiver.Archiver.test_archiver_mode_patched: PASSED (15.50s) archiver.Archiver.test_archiver_mode_recipe: PASSED (15.62s) archiver.Archiver.test_archiver_srpm_mode: PASSED (20.66s) oe-selftest () - Ran 15 tests in 257.170s Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-05archiver: Capture git submodules in mirror archiverPaul Barker
Using the new Fetch.expanded_urldata() function we can get URL data for all git submodules. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04oe-selftest: Recursively patch test case pathsPaul Barker
This ensures that builddir is updated correctly to point to the new selftest build directory when we're given a list of test suites instead of a list of test cases. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04oe-selftest: Support verbose log outputPaul Barker
Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04oe-selftest: Allow overriding the build directory used for testsPaul Barker
This may be useful if the parent directory of the original builddir is not writable, on a lower performance drive, etc. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04archiver: Fix test case for srpm archiver modePaul Barker
We can't assume that rpm packaging will always be enabled when running oe-selftest. In particular when using nodistro instead of poky this is not enabled by default. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04wic: Add --offset argument for partitionsJoshua Watt
Add support for an --offset argument when defining a partition. Many SoCs require that boot partitions be located at specific offsets. Prior to this argument, most WKS files were using the --align attribute to specify the location of these fixed partitions but this is not ideal because in the event that the partition couldn't be placed in the specified location, wic would move it to the next sector with that alignment, often preventing the device from booting. Unlike the --align argument, wic will fail if a partition cannot be placed at the exact offset specified with --offset. Changes in V2: * Fixed a small typo that prevented test_fixed_size_error from passing Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04oeqa/targetcontrol: Rework exception handling to avoid warningsRichard Purdie
We're seeing: WARNING: bitbake/lib/bb/cookerdata.py:136: ResourceWarning: unclosed file <_io.FileIO name='tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemurunner_log.20200601181912' mode='ab' closefd=True which can only be caused by the qemu.stop() method not being called. Tweak the error handling to fix the blanket exception handler which is likely meaning this function isn't getting called. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-03apt: update to 1.8.2.1Alexander Kanavin
I took the opportunity to rewrite the recipe from scratch; there was just too much baggage in it. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-03meta-selftest: add test for .patch file with long filename and without subjectMartin Jansa
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-03meta-selftest: add test of .gitignore in tarballMartin Jansa
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-30qemurunner: fix ip fallback detectionKonrad Weihmann
When falling back from detecting ip from /proc/./cmdline the output of runqemu is acutally 'Network configuration: ip=192.168.7.2::192.168.7.1::255.255.255.0' which doesn't match the given regex and leading to run failure, although IP is detectable. Fix regex by inserting an optional 'ip=' prefix to first IP Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27qemu: enable virglrenderer and glx options subject to 'opengl' DISTRO_FEATUREAlexander Kanavin
Note that to actually use accelerated GL passthrough, there are two options 1) a suitable frontend need to be also enabled - gtk+ and SDL both seem to work well. Previously I struggled to make SDL work, but now it seems fine. 2) it is also possible to render off-screen with -display egl-headless option, and see the output with a VNC viewer (for which, qemu needs to be started with a VNC server): $ runqemu kvm egl-headless publicvnc Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-24oeqa/concurrencytest: don't delete build directory for failed testsSteve Sakoman
Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-21testresults.json: add duration of the tests as wellAlexander Kanavin
This is printed by testimage, but isn't actually saved. It's a useful metric for tracking execution times. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-18image_types: declare support for wic.zstDiego
Declare images in wic.zst format as supported, as bmaptool now supports zstd: https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6 Given the very fast decompression speed of zstd over, for example, gzip, writing images with bmaptool can provide big write speed improvements when decompression speed is the limiting factor (especially with very sparse images). Signed-off-by: Diego Rondini <diego.rondini@kynetics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-12selftest/imagefeatures: Enable sanity test for IMAGE_GEN_DEBUGFSYeoh Ee Peng
Add new testcase to check IMAGE_GEN_DEBUGFS. Test makes sure that debug filesystem is created accordingly. Test also check for debug symbols for some packages as suggested by Ross Burton. [YOCTO #10906] Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-07virgl: skip test on Debian 9Alexander Kanavin
Debian 9 has an older version of mesa, where dri drivers do not link against glapi explicitly, which causes problems when they are loaded by newer mesa-native: pokybuild@debian9-ty-2:~$ ldd -d /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so |grep undefined undefined symbol: _glapi_tls_Dispatch (/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so) undefined symbol: _glapi_tls_Context (/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so) compared to Debian 10: pokybuild@debian10-ty-2:~$ ldd -d /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so|grep undefined pokybuild@debian10-ty-2:~$ Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>