aboutsummaryrefslogtreecommitdiffstats
path: root/meta/classes/testimage.bbclass
AgeCommit message (Collapse)Author
2018-10-29testimage: Improvements to the json loggingRichard Purdie
Tweak the preceeding commit to: * Add STARTTIME to the identifier to make it unique * Log DISTRO * Use LOG_DIR * Store the layer config in a more natural json format * Drop '_' function prefixes Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29testimage.bbclass: write testresult to json filesYeoh Ee Peng
As part of the solution to replace Testopia to store testresult, OEQA testimage need to output testresult into json files, where these json testresult files will be stored into git repository by the future test-case-management tools. By default, json testresult file will be written to "oeqa" directory under the "WORKDIR" directory. To configure multiple instances of bitbake to write json testresult to a single testresult file at custom directory, user will define the variable "OEQA_JSON_RESULT_DIR" with the custom directory for json testresult. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-25testimage.bbclass: define ptest suite variableArmin Kuster
Ptest has dependencies on other tests so contain them in a variable. This allows layers to not worry about maintaining the depends themselves. Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure opkg/apt test suites are runRichard Purdie
Currently only the dnf package manager tests are automatically added to the list of tests to run. Improve the code to handle automatic addition of the apt and opkg tests too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure package-index is triggered for opkg/apt testsRichard Purdie
If the opkg/apt tests are run without a package-index they will fail. Trigger this here for now as a dependency until the code can be improved. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11testimage: Ensure TESTIMAGEDEPENDS is appended to for qemuall overrideRichard Purdie
The current qemuall override overwrites other dependencies unintentionally. Tweak the code to avoid this by appending to the variable. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-11testimage.bbclass: fix qemu_use_kvm handlingEmmanuel Roullit
QEMU_USE_KVM can either be a boolean or a whitespace separated list of kvm supported machines. For the 'intel-corei7-64' machine, defined in meta-intel, kvm could not be used as the 'x86' substring is not part of its machine name. By changing the order of this 'or' statement and setting the 'QEMU_USE_KVM' variable to 'intel-corei7-64', it is possible to run the 'testimage' task with kvm support successfully. Signed-off-by: Emmanuel Roullit <emmanuel.roullit@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21bitbake.conf/testimage: Add in compatiblity hacksRichard Purdie
The testimage changes break the autobuilder. The plan is to revert these hacks but due to various changes happening with the autobuilder, we need these for now to keep things working until we can move to the new autobuilder codebase. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21image/testimage: Rework auto image test executionRichard Purdie
The TEST_IMAGE interface has never particularly worked and image testing currently gets configured manually. This reworks the interface to better serve the needs of its users, replacing it with TESTIMAGE_AUTO. This does away with the need for the separate class due to better bitbake APIs for changing tasks. TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It also adds in dependencies so that any SDK for which testing is requested will automatically be built first. The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: move codes into testimage_mainChen Qi
testimage-auto is expected to run testimage task's codes automatically. But in fact, it's currently missing some codes, including testimage_sanity and create_rpm_index. This leads to the problem of unexpected runtime failure of test_dnf_makecache. The error message is as below. RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR This error is caused by the fact that create_rpm_index is not executed before running the tests. There's no reason why such codes should not be in testimage_main, so move them into it. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: also check 'auto' to create rpm indexChen Qi
Having 'auto' in TEST_SUITES will also run the 'dnf' test cases, so also check it to determine whether to create rpm index or not. This is to fix the following error when TEST_SUITES = "auto". RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04testimage.bbclass: fix behavior of empty TEST_SUITESChen Qi
The current behaviour of TEST_SUITES is very confusing. setting: TEST_SUITES = "" result: Execute all test cases. setting: TEST_SUITES = "some_case_not_exist" result: Error out with 'Empty test suite' message. The expected behaviour of TEST_SUITES should be: 1. when 'auto' is in it, execute as many test cases as possible 2. when no valid test case is specified, error out and give user message The first one is implemented by a previous patch. The second one is fixed in this patch. Also add debug message to show test cases to be executed. This is for easier debugging. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-14runtime/dnf: Add new dnf test casesJose Perez Carranza
Add test cases to test “exclude” and “installroot“ options, also modify the logic of filtering packages on the feed to have all the packages needed by the tests. [YOCTO #10744] Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-18testimage: run systemtap test on SDK imagesRoss Burton
Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-01-29testimage: enable gi test suiteRoss Burton
(From OE-Core rev: d8243c4588d4f1bb057fd917bfea130c4907e24c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-05testimage.bbclass: add ptest to the list of runtime tests whenever possibleAlexander Kanavin
If no ptest packages are installed in the image, the test does nothing; if ptest packages are installed in the image, then they should be run without user having to enable that manually. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18targetcontrol.py: use oe.types.boolean for QEMU_USE_KVMRobert Yang
So that both QEMU_USE_KVM = "True" and "1" will work. [YOCTO #12343] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-11-10testimage: Ensure full logs are shown for failuresRichard Purdie
Currently, the fact an error message is shown means the rest of the task logs are suppressed. In this case we don't want that as it hides the real errors and useful information. Therefore override this behaviour. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09testimage: Pass the logger into OERuntimeTestContextExecutor.getTarget()Richard Purdie
I have no idea why we didn't do this but it means the code has nowhere to log to unless we do this. This means we can then use the logger to log data to the task logs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23testimage.bbclass: update commentsRobert Yang
It's very important to add IMAGE_CLASSES += "testimage" in local.conf firstly, otherwise the var like TEST_LOG_DIR (defined in testimage.bbclass) will not be in testdata.json. [YOCTO #11547] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08testimage: Use the renamed buildlzipKhem Raj
buildiptables has been replaced with buildlzip Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-12testimage.bbclass: Correct the comment to state right dir for test casesKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-09oeqa: Change the order to logDetails and logSummaryAníbal Limón
Is better to log the summary at end to see in an easy way the actual result of the test run. [YOCTO #11622] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-05-30classes/test{export,images}: Change modules to listAníbal Limón
With the new filters by <module>.[class].[testname] the modules arg is expected to be a list so use split into TEST_SUITES variable. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-30oeqa/core: Move OETestContext.log{Summary, Details} into OETestResultAníbal Limón
Those methods are used to write in the log the results so it makes sense to have defined at OETestResult because is a format of the result itself. [YOCTO #11450] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-21testimage.bbclass: add additional dependencybrian avery
qemu-native-helper has an additional task that needs to be run in order for testimage to work. This task is usually run by default in a full build but there are use cases where it might be skipped. This commit adds the dependency explicitly. Also, this commit adds a try/catch error message to make it clearer what you need to do if you try to run testimage before you have built or downloaded the image artifacts. [YOCTO #11375] Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-03-28oeqa: QEMU_USE_KVM can list machinesPatrick Ohly
Previously, QEMU_USE_KVM=True enabled the use of kvm only when "x86" was in the MACHINE name. That is too limiting, because for example intel-corei7-64 can also use kvm but it wasn't possible to enable that without changing OE-core. That traditional usage is still supported. In addition, QEMU_USE_KVM can be set to a list of space-separated MACHINE names for which kvm is to be enabled. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-21meta/lib/oeqa: change sdk test from cvs to cpiobrian avery
We currently fetch,configure,build, and install cvs as our test for the sdk. cvs unfortunately, requires a default editor in order to run. The change in 94790a8254d6 that checks to see if you have something like vi installed is fragile since you may have a different default editor. This patch switches from using cvs as a test to using cpio. cpio also uses autotools so the functionality tested is equivalent. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-03-14testimage: Don't use systemd.log_level=debug loggingRichard Purdie
This causes MBs of log messages which overload the standard 200kb syslog limit used by busybox syslog by default. This means some tests which rely on messages being detected in syslog, e.g. rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size fail (AssertionError: 54 not greater than or equal to 80 : Cound not find sufficient amount of rpm entries in /var/log/messages, found 54 entries). We enabled this to aid debugging of some systemd race issues, those are now resolved so we can disable this. Leave the log level parameter as a comment to save others having to look it up if they need debugging. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13testimage.bbclass: fix runtime test for rpm, port smart tests to dnfAlexander Kanavin
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
2017-03-01recipes: Make use of the new bb.utils.filter() functionPeter Kjellerstedt
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-23oeqa/runtime/context.py: Fix use of getTarget() with testexportMariano Lopez
The idea on getTarget is to use kwargs to send custom variables to different targets, instead of this, a new variable was added (just used for custom targets) and this broke testexport. So in order to fix it, just add the custom variable to kwargs. This fixes the use of getTarget() in testexport class that was introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-15testimage.bbclass: Add check for empty test suiteMariano Lopez
If for some reason (most likely incorrect TEST_SUITES var) you end with an empty test suite, testimage will start the target (qemu by default) and will run an empty suite, so better have a check before starting the target. [YOCTO #10979] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-15testimage: Implement custom target loadingJuro Bystricky
This patch implements custom target loading for testimage, currently missing due to major changes to the test framework. Custom targets can be defined in various meta-layers, so we need an extra path information in order to find them. Any other additional info is retrieved as usual via the variables TEST_TARGET and TEST_SUITES Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31oeqa/runtime/cases: Rename syslog module to oe_syslogAníbal Limón
Debian based distros has a builtin syslog module so when try to load tests using unittest it references the builtin module instead of runtime/cases. [YOCTO #10964] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31testimage.bbclass: Fixes about suites and dependenciesMariano Lopez
There are some missing suites on sato-sdk and lsb-sdk images so add it. The createrepo-native needs to be built before run testimage in order to create a repository from install packages. The DL_DIR data also needs to be updated from the current bitbake environment instead of use the value from testdata file. [YOCTO #10964] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31testimage.bbclass: Allow to run tests on autobuilder's imagesMariano Lopez
With the change to the new framework data store dependecy was removed, instead a new file is generated and used in testimage. When testing builds from the autobuilders the test data values are from the autobuilder, including the paths. Some tests require paths to current environment in order to run, this commit will update such paths and fix the error of running images donwloaded from autobuilders. [YOCTO #10964] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-26testimage.bbclass: Add systemd test automaticallyMariano Lopez
This adds systemd test automatically when an image is build with systemd DISTRO_FEATURE [YOCTO #10935] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23testimage.bbclass: Add package install featureMariano Lopez
This allows to use the package install feature with the new OEQA framework. [YOCTO #10234] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2017-01-23testimage.bbclass: Add support for package extractionMariano Lopez
testimage support the installation of packages without a package manager in the target. This adds support for package extraction required to support the installation feature. [YOCTO #10234] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2017-01-23runtime/cases/smart.py: Migrate smart testsMariano Lopez
This migrates the smart test from the old framework to the new one. This has its own commit because smart test was using bb and oe libraries that are available when exporting the test cases to run in a different host. Because of the removal of bb and oe libraries index and packages feeds creation will be managed in testimage bbclass. [YOCTO #10234] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2017-01-23testimage.bbclass: Migrate class to use new runtime frameworkMariano Lopez
This migrates testimage class to use the new framework. Most of the code added here is to get rid off the data store dependency. [YOCTO #10234] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2017-01-23oeqa/utils/dump: Move get_host_dumper to OERuntimeTestContextExecutor classAníbal Limón
To avoid getVar calls inside a utils module, also moves get_host_dumper import inside testexport isn't needed. [YOCTO #10231] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2017-01-23oeqa/runtime: Fix TargetBuildProject instancesAníbal Limón
TargetBuildProject was refactored to avoid bitbake dependency so the instance don't allow to pass data store anymore. classes/testimage: Export proxies before run tests The TargetBuildProject based tests download archives from network. Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2017-01-19oeqa/utils/qemurunner.py: Be sure to stop qemu-systemMariano Lopez
When runqemu fails, qemu-system process would keep running and won't be killed, setpgrp() was used when runqemu was a shell script but it seems it doesn't work always with python. This would kill qemu-system explicity and to avoid leaving it behind. Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-12-16meta: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-10-09testimage: disable build tests for qemumips and qemumips64Joshua Lock
It's not uncommon for qemumips[64] builds on the Yocto Project autobuilder to fail during Sanity Tests after a very long timeout period. This is due to the MIPS emulation in QEMU being slow and some of the build tests taking a very long time on MIPS machines. This patch works around this slowness by disabling the more complex build tests for QEMU MIPS machines. [YOCTO #10340] Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-03testimage.bbclass: Use bb.fatal() instead of raising FuncFailedUlf Magnusson
This sets a good example and avoids unnecessarily contributing to perceived complexity and cargo culting. Motivating quote below: < kergoth> the *original* intent was for the function/task to error via whatever appropriate means, bb.fatal, whatever, and funcfailed was what you'd catch if you were calling exec_func/exec_task. that is, it's what those functions raise, not what metadata functions should be raising < kergoth> it didn't end up being used that way < kergoth> but there's really never a reason to raise it yourself FuncFailed.__init__ takes a 'name' argument rather than a 'msg' argument, which also shows that the original purpose got lost. Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-23testimage.bbclass: Add package manager dependencyMariano Lopez
The feature to install packages in the target requires to build the package manager. It would fail, with very obtuse errors, if a test requires to install something and the package manager hasn't been build. This will add the package manager as dependency for testimage. [YOCTO #10260] Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-09-13testimage: add createrepo-native to testimage dependsbavery
The smart tests need createrepo to be in the native sysroot. It needs to be one of the depends for testimage. Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-10 14:44:04 +0000