aboutsummaryrefslogtreecommitdiffstats
path: root/meta/classes/testimage.bbclass
AgeCommit message (Collapse)Author
2019-05-17oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto
Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28oeqa: make it work for multiple usersRobert Yang
There are failures when multiple users run oe-selftest on the same host: PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved- tests/201812250324_qemu' This is because /tmp/oe-saved-tests was created by user A, while user B tries to write data in it, then the error will happen. This patch can fix the problem. Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than /tmp/oe-saved-tests/ to fix the problem. (From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28meta/classes/testimage.bbclass: Only validate IMAGE_FSTYPES when is QEMUAníbal Limón
When use simpleremote target the flash/boot process is executed manually, the IMAGE_FSTYPES validation is only needed when execute testimage against qemu. The supported_fstypes comes from oeqa.core.target.qemu module. (From OE-Core rev: e7dc5963adbacc091fe8943119262166977623ad) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage.bbclass: remove boot parameter systemd.log_targetKai Kang
Boot parameter systemd.log_target=console affects command journalctl's output and causes oe selftest case test_systemd_boot_time fail to pass. | Error at obtaining the boot time from journalctl | RESULTS: | RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time - Testcase -1: SKIPPED (0.74s) systemd.log_target=console was introduced by oe-core commit a0bb649 and work with parameter systemd.log_level to enable systemd debug. systemd.log_level has been removed already, so remove systemd.log_target too to make case test_systemd_boot_time pass. (From OE-Core rev: caa776bdcf8ea34c857f45970370bf771075f4bc) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Add possibility to pass parmeters to qemuErik Botö
Add a variable called TEST_QEMUPARAMS in testimage.bbclass to make it possible to pass parameters to qemu. This can be useful for e.g. increasing the amount of RAM available during testimage runs. (From OE-Core rev: 1a9163f5779d233c884c8fd50e0812eabab4fdf3) Signed-off-by: Erik Botö <erik.boto@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Add support for slirpYeoh Ee Peng
Enable testimage to support qemu slirp. Configure "QEMU_USE_SLIRP" & "TEST_SERVER_IP" variables to enable slirp. [YOCTO#10713] (From OE-Core rev: 3df9ee85ce7fe52f0893fd33aea3bf1fcc6ead0a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Enable autorunning of the package manager testsuitesRichard Purdie
Now that the hangs in httpservice are fixed we can let these tests auto skip as appropriate. (From OE-Core rev: 42a0d70291d551578e21f590fcb85ca72a78ccb5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Further cleanup DEFAULT_TEST_SUITESRichard Purdie
Now the test markup of the development tools tests is complete, this can be further tweaked to auto run the correct tests. (From OE-Core rev: bd4f8d12fe1f9f2643ee9e68fa2bb981134294fb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Simplfy DEFAULT_TEST_SUITES logicRichard Purdie
Now that the tests have correct markup to automatically determine which images they can run against, clean up the default test suites logic to be simpler and not image specific. Some cleanup of the compiler tests still needs to be completed but this is a good first step. The only downside to this is more noise during testing as we now see many skipped messages for simple images like core-image-minimal. The auto type is being removed since it currently breaks badly due to the socat mandatory inclusion from the meta-selftest layer which is a problem which needs to be addressed seperately. (From OE-Core rev: 4966bc33845752eb0aeae54b72e8ba0146a7ed52) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28testimage: Remove duplicate dependenciesRichard Purdie
(From OE-Core rev: 16d4368e7b85d5179374adb0a2d1ac215793e104) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-01-28oeqa: Fix for QEMU_USE_KVMRobert Yang
Fixed: MACHINE = "qemux86" QEMU_USE_KVM = "qemux86" IMAGE_CLASSES += "testimage" $ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs [snip] File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean raise ValueError("Invalid boolean value '%s'" % value) ValueError: Invalid boolean value 'qemux86' Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm will be enabled if target_arch == build_arch or both of them are x86 archs. (From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2018-12-06testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation ↵Richard Purdie
warnings Use our own lsb function instead as used elsewhere by the codebase. (From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29testimage: Improvements to the json loggingRichard Purdie
Tweak the preceeding commit to: * Add STARTTIME to the identifier to make it unique * Log DISTRO * Use LOG_DIR * Store the layer config in a more natural json format * Drop '_' function prefixes Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29testimage.bbclass: write testresult to json filesYeoh Ee Peng
As part of the solution to replace Testopia to store testresult, OEQA testimage need to output testresult into json files, where these json testresult files will be stored into git repository by the future test-case-management tools. By default, json testresult file will be written to "oeqa" directory under the "WORKDIR" directory. To configure multiple instances of bitbake to write json testresult to a single testresult file at custom directory, user will define the variable "OEQA_JSON_RESULT_DIR" with the custom directory for json testresult. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-25testimage.bbclass: define ptest suite variableArmin Kuster
Ptest has dependencies on other tests so contain them in a variable. This allows layers to not worry about maintaining the depends themselves. Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure opkg/apt test suites are runRichard Purdie
Currently only the dnf package manager tests are automatically added to the list of tests to run. Improve the code to handle automatic addition of the apt and opkg tests too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure package-index is triggered for opkg/apt testsRichard Purdie
If the opkg/apt tests are run without a package-index they will fail. Trigger this here for now as a dependency until the code can be improved. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure TESTIMAGEDEPENDS is appended to for qemuall overrideRichard Purdie
The current qemuall override overwrites other dependencies unintentionally. Tweak the code to avoid this by appending to the variable. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-11testimage.bbclass: fix qemu_use_kvm handlingEmmanuel Roullit
QEMU_USE_KVM can either be a boolean or a whitespace separated list of kvm supported machines. For the 'intel-corei7-64' machine, defined in meta-intel, kvm could not be used as the 'x86' substring is not part of its machine name. By changing the order of this 'or' statement and setting the 'QEMU_USE_KVM' variable to 'intel-corei7-64', it is possible to run the 'testimage' task with kvm support successfully. Signed-off-by: Emmanuel Roullit <emmanuel.roullit@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21bitbake.conf/testimage: Add in compatiblity hacksRichard Purdie
The testimage changes break the autobuilder. The plan is to revert these hacks but due to various changes happening with the autobuilder, we need these for now to keep things working until we can move to the new autobuilder codebase. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21image/testimage: Rework auto image test executionRichard Purdie
The TEST_IMAGE interface has never particularly worked and image testing currently gets configured manually. This reworks the interface to better serve the needs of its users, replacing it with TESTIMAGE_AUTO. This does away with the need for the separate class due to better bitbake APIs for changing tasks. TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It also adds in dependencies so that any SDK for which testing is requested will automatically be built first. The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: move codes into testimage_mainChen Qi
testimage-auto is expected to run testimage task's codes automatically. But in fact, it's currently missing some codes, including testimage_sanity and create_rpm_index. This leads to the problem of unexpected runtime failure of test_dnf_makecache. The error message is as below. RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR This error is caused by the fact that create_rpm_index is not executed before running the tests. There's no reason why such codes should not be in testimage_main, so move them into it. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: also check 'auto' to create rpm indexChen Qi
Having 'auto' in TEST_SUITES will also run the 'dnf' test cases, so also check it to determine whether to create rpm index or not. This is to fix the following error when TEST_SUITES = "auto". RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: fix behavior of empty TEST_SUITESChen Qi
The current behaviour of TEST_SUITES is very confusing. setting: TEST_SUITES = "" result: Execute all test cases. setting: TEST_SUITES = "some_case_not_exist" result: Error out with 'Empty test suite' message. The expected behaviour of TEST_SUITES should be: 1. when 'auto' is in it, execute as many test cases as possible 2. when no valid test case is specified, error out and give user message The first one is implemented by a previous patch. The second one is fixed in this patch. Also add debug message to show test cases to be executed. This is for easier debugging. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-14runtime/dnf: Add new dnf test casesJose Perez Carranza
Add test cases to test “exclude” and “installroot“ options, also modify the logic of filtering packages on the feed to have all the packages needed by the tests. [YOCTO #10744] Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-18testimage: run systemtap test on SDK imagesRoss Burton
Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-01-29testimage: enable gi test suiteRoss Burton
(From OE-Core rev: d8243c4588d4f1bb057fd917bfea130c4907e24c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-05testimage.bbclass: add ptest to the list of runtime tests whenever possibleAlexander Kanavin
If no ptest packages are installed in the image, the test does nothing; if ptest packages are installed in the image, then they should be run without user having to enable that manually. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18targetcontrol.py: use oe.types.boolean for QEMU_USE_KVMRobert Yang
So that both QEMU_USE_KVM = "True" and "1" will work. [YOCTO #12343] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-11-10testimage: Ensure full logs are shown for failuresRichard Purdie
Currently, the fact an error message is shown means the rest of the task logs are suppressed. In this case we don't want that as it hides the real errors and useful information. Therefore override this behaviour. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09testimage: Pass the logger into OERuntimeTestContextExecutor.getTarget()Richard Purdie
I have no idea why we didn't do this but it means the code has nowhere to log to unless we do this. This means we can then use the logger to log data to the task logs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23testimage.bbclass: update commentsRobert Yang
It's very important to add IMAGE_CLASSES += "testimage" in local.conf firstly, otherwise the var like TEST_LOG_DIR (defined in testimage.bbclass) will not be in testdata.json. [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08testimage: Use the renamed buildlzipKhem Raj
buildiptables has been replaced with buildlzip Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-12testimage.bbclass: Correct the comment to state right dir for test casesKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-09oeqa: Change the order to logDetails and logSummaryAníbal Limón
Is better to log the summary at end to see in an easy way the actual result of the test run. [YOCTO #11622] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-05-30classes/test{export,images}: Change modules to listAníbal Limón
With the new filters by <module>.[class].[testname] the modules arg is expected to be a list so use split into TEST_SUITES variable. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-30oeqa/core: Move OETestContext.log{Summary, Details} into OETestResultAníbal Limón
Those methods are used to write in the log the results so it makes sense to have defined at OETestResult because is a format of the result itself. [YOCTO #11450] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-21testimage.bbclass: add additional dependencybrian avery
qemu-native-helper has an additional task that needs to be run in order for testimage to work. This task is usually run by default in a full build but there are use cases where it might be skipped. This commit adds the dependency explicitly. Also, this commit adds a try/catch error message to make it clearer what you need to do if you try to run testimage before you have built or downloaded the image artifacts. [YOCTO #11375] Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-03-28oeqa: QEMU_USE_KVM can list machinesPatrick Ohly
Previously, QEMU_USE_KVM=True enabled the use of kvm only when "x86" was in the MACHINE name. That is too limiting, because for example intel-corei7-64 can also use kvm but it wasn't possible to enable that without changing OE-core. That traditional usage is still supported. In addition, QEMU_USE_KVM can be set to a list of space-separated MACHINE names for which kvm is to be enabled. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-21meta/lib/oeqa: change sdk test from cvs to cpiobrian avery
We currently fetch,configure,build, and install cvs as our test for the sdk. cvs unfortunately, requires a default editor in order to run. The change in 94790a8254d6 that checks to see if you have something like vi installed is fragile since you may have a different default editor. This patch switches from using cvs as a test to using cpio. cpio also uses autotools so the functionality tested is equivalent. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-03-14testimage: Don't use systemd.log_level=debug loggingRichard Purdie
This causes MBs of log messages which overload the standard 200kb syslog limit used by busybox syslog by default. This means some tests which rely on messages being detected in syslog, e.g. rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size fail (AssertionError: 54 not greater than or equal to 80 : Cound not find sufficient amount of rpm entries in /var/log/messages, found 54 entries). We enabled this to aid debugging of some systemd race issues, those are now resolved so we can disable this. Leave the log level parameter as a comment to save others having to look it up if they need debugging. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13testimage.bbclass: fix runtime test for rpm, port smart tests to dnfAlexander Kanavin
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
2017-03-01recipes: Make use of the new bb.utils.filter() functionPeter Kjellerstedt
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-23oeqa/runtime/context.py: Fix use of getTarget() with testexportMariano Lopez
The idea on getTarget is to use kwargs to send custom variables to different targets, instead of this, a new variable was added (just used for custom targets) and this broke testexport. So in order to fix it, just add the custom variable to kwargs. This fixes the use of getTarget() in testexport class that was introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-15testimage.bbclass: Add check for empty test suiteMariano Lopez
If for some reason (most likely incorrect TEST_SUITES var) you end with an empty test suite, testimage will start the target (qemu by default) and will run an empty suite, so better have a check before starting the target. [YOCTO #10979] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-15testimage: Implement custom target loadingJuro Bystricky
This patch implements custom target loading for testimage, currently missing due to major changes to the test framework. Custom targets can be defined in various meta-layers, so we need an extra path information in order to find them. Any other additional info is retrieved as usual via the variables TEST_TARGET and TEST_SUITES Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31oeqa/runtime/cases: Rename syslog module to oe_syslogAníbal Limón
Debian based distros has a builtin syslog module so when try to load tests using unittest it references the builtin module instead of runtime/cases. [YOCTO #10964] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31testimage.bbclass: Fixes about suites and dependenciesMariano Lopez
There are some missing suites on sato-sdk and lsb-sdk images so add it. The createrepo-native needs to be built before run testimage in order to create a repository from install packages. The DL_DIR data also needs to be updated from the current bitbake environment instead of use the value from testdata file. [YOCTO #10964] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31testimage.bbclass: Allow to run tests on autobuilder's imagesMariano Lopez
With the change to the new framework data store dependecy was removed, instead a new file is generated and used in testimage. When testing builds from the autobuilders the test data values are from the autobuilder, including the paths. Some tests require paths to current environment in order to run, this commit will update such paths and fix the error of running images donwloaded from autobuilders. [YOCTO #10964] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-26testimage.bbclass: Add systemd test automaticallyMariano Lopez
This adds systemd test automatically when an image is build with systemd DISTRO_FEATURE [YOCTO #10935] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-04 09:31:27 +0000