aboutsummaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa
Commit message (Collapse)AuthorAgeFilesLines
* oeqa: avoid class setup method to run when skipping the whole classChen Qi2019-06-212-0/+10
| | | | | | | | | | | | | For now, even if we have specified to skip the whole module/class via command line, e.g., `oe-selftest -R gotoolchain', the class setup method is still run. This at least results in unnecessary builds, and at worst results in ERROR, if the setup method fails. So improve the skipping mechanism to avoid class setup method to run when specified to skip. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* context.py: avoid skipping tests by meaningless command argumentChen Qi2019-06-211-1/+1
| | | | | | | | | Currently `oe-selftest -R a' will skip 'archiver' tests. This is not expected. Fix it so that the '-R' should be followed by actual module/class/test names. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/logparser: ignore test failure commentaryRoss Burton2019-06-191-2/+2
| | | | | | | | | | | The output format for Python and GLib both can be of this form: FAIL: foobar (Segmentation fault) In this case the test is called foobar not foobar_segmentation_fault. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runtime/cases/logrotate: make test more reliableAnuj Mittal2019-06-191-1/+3
| | | | | | | | | | | | | | | | | | | By default logrotate uses 'dateext' in logrotate.conf which results in a date string being appended at the end of log name. In cases when a test that installs configuration in logrotate.d is executed first (dnf for example), it might result in errors when logrotate test is executed: error: destination /var/log/dnf.librepo.log-20190528 already exists, skipping rotation error: destination /var/log/dnf.rpm.log-20190528 already exists, skipping rotation error: destination /var/log/hawkey.log-20190528 already exists, skipping rotation Tweak the test to use it's own temporary configuration that doesn't enable dateext. Fixes [YOCTO #13363] Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* manifest.py: fix test_SDK_manifest_entriesChen Qi2019-06-181-5/+2
| | | | | | | | TOOLCHAIN_OUTPUTNAME could be overridden. So use this variable directly instead of its default value ${SDK_NAME}-toolchain-${SDK_VERSION}. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* context.py: fix skipping functionChen Qi2019-06-181-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current codes to skip test cases are logically correct, but they do not work correctly in reality. It does skip the tests as the command line argument specifies, but the related information is wrong. e.g. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test" 2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test" The archiver.Archiver.xxx tests should be skipped by 'archiver' command line argument, not 'runtime_test'. Change to use a function generator to achieve the desired effect. After the change, the effect is as follows. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver) 2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers" 2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers" [snip] Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/devtool: fix URI to MarkupSafe packageOleksandr Kravchuk2019-06-141-1/+1
| | | | | | | | | PyPi packages are now hosted at files.pythonhosted.org. [YOCTO #13243] Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oescripts.py: avoid error when cairo module is not availableChen Qi2019-06-141-1/+2
| | | | | | | | | | | | | When running 'oe-selftest -r oescripts', the following error appeared. cls.skipTest('Python module cairo is not present') TypeError: skipTest() missing 1 required positional argument: 'reason' This is because the host does not have the cairo python module installed. Fix this problem by using unittest's SkipTest exception. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftests: add tests for INCOMPATIBLE_LICENSEQuentin Schulz2019-06-121-0/+41
| | | | | | | | | | | | | | One bug went unnoticed without these selftests: an INCOMPATIBLE_LICENSE with a non-SPDX license for a package with that non-SPDX license wasn't enforcing the denial of build for said package. See 4b6ce4604cc15e289a48f8586d58a101b7a70b52 ("meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE") While adding a test for that particular case, let's add a few more so that we cover a handful more use cases of INCOMPATIBLE_LICENSE. Signed-off-by: Quentin Schulz <quentin.schulz@streamunlimited.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* recipetool: add python3 supportMaciej Pijanowski2019-06-101-2/+40
| | | | | | | | | | | | | | Add support for generating python3 recipes using the recipetool / devtool. Drop python2 support at the same time. Tested with: oe-selftest -r recipetool.RecipetoolTest [YOCTO #13264] Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Add reproducible build selftestJoshua Watt2019-06-101-0/+160
| | | | | | | | | | | | | Adds an initial test for reproducible builds to the OE selftest. This initial test builds core-image-minimal using sstate, then does a clean build without sstate in another build directory, and finally does a binary comparison of the resulting package files between the two builds. The test is currently always skipped since it doesn't pass yet, but it can easily be enabled locally Signed-off-by: Joshua Watt <JPEWHacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* qemurunner: fix undefined variableAdrian Freihofer2019-06-081-0/+1
| | | | | | | | While hacking on this I got an Exception. It's better to define variables also in python. Signe:-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime: add simple test for sconsTim Orling2019-06-083-0/+43
| | | | | | | This test simply compiles a hello world program using scons. Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gpg_sign/selftest: Fix secmem parameter handlingRichard Purdie2019-06-071-1/+2
| | | | | | | | | | | | | | | | | | | | | | | We keep seeing "cannot allocate memory" errors from rpm when signing packages on the autobuilder. The following were tried: * checking locked memory use (isn't hitting limits) * Restricting RPM_GPG_SIGN_CHUNK to 1 * Limiting to 10 parallel do_package_write_rpm tasks * Allowing unlimied memory overcommit * Disabling rpm parallel compression and the test still failed. Further invetigation showed that the --auto-expand-secmem wasn't being passed to gpg-agent which meant the secmem couldn't be expanded hence the errors when there was pressure on the agent. The reason this happens is that some of the early gpg commands can start the agent without the option and it sticks around in memory so a version with the correct option may or may not get started. We therefore add the option to all the key gpg calls. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Set both the threadport&serverport with tcpserial ↵Kevin Hao2019-06-071-2/+2
| | | | | | | | | | | | | | | | | | | | | parameter After the commit ad522ea6a64e ("runqemu: Let qemuparams override default settings"), the order of the two "-serial" parameters when running the qemu have been switched. The effect of this is that the logging thread will use ttyS1 (of course can't capture the kernel boot message anymore), and the test command will run on the ttyS0. So the output of the test command may be mangled by the kernel message (such as call trace), and let the test command produce a fake timeout error message. We can't fix it by just adjusting the order of the threadport and serverport, since it will break some machines such as qemuarm64 which use the virtio serial. So using the tcpserial to setup both the threadport and serverport. [YOCTO Bug 13309] Signed-off-by: Kevin Hao <kexin.hao@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: modify to be multi-machineJon Mason2019-06-041-1/+1
| | | | | | | | | | Currently, the code will sum all of the different machine results into a single report of the tests results. This can lead to confusion as to which machine may be experiencing issues. Modify the code to store the results in a per machine basis and report them accordingly. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gdb: Upgrade from 8.2.1 to 8.3Alistair Francis2019-05-211-1/+2
| | | | | | | | Bump GDB and change the oeqa selftest expression to allow ptests to pass. Signed-off-by: Alistair Francis <alistair.francis@wdc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/targetcontrol.py: fix qemuparams not work in runqemu with launch_cmdHongxu Jia2019-05-151-1/+1
| | | | | | | | | | | | | | | As runqemu with launch_cmd means directly run the command, don't need set rootfs or env vars. Since commit [a847dd7202 runqemu: Let qemuparams override default settings] applied in oe-core, if launch_cmd contains "qemuparams='***'", it does not work, which is overridden by latter qemuparams="-serial tcp:127.0.0.1" in QemuRunner.launch(); So we set qemuparams as a parameter in runqemu, the fix makes it work Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/target/ssh: Replace suggogatepass with ignoring errorsRichard Purdie2019-05-131-2/+2
| | | | | | | | | We continued to see encoding problems with ssh commands run in oeqa. After much research the conclusion was we should use ignore the errors since some occasional bad locale encoding is better than the unicode decoding issues we were seeing which crashed large parts of tests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Automate manual pybootchart testsRichard Purdie2019-05-122-26/+34
| | | | | | | | | Automate the current manual pybootchart tests. This includes a check for the cairo dependency, skipping the test if appropriate. Based on original patch from Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrenttest: Patch subunit module to handle classSetup failuresRichard Purdie2019-05-091-4/+23
| | | | | | | | | | | | | | | | | | | | Currently setupClass errors were not being mapped back to the failing tests and they were hence being marked as UNKNOWN and the test statistics were inaccurate. This is because whilst the errors were being encoded into the test results stream, the decoder doesn't cope with an error outside a testStart event. We patch in an addError handler to the outsideTest parser so that this does get handled in a way similar to the non-concurrent case. It would be nice if we didn't have to do this but there doesn't seem to be any other way to fix this other than forking subunit. We also make a minor change so another of our changes can cope with tests without a start time. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Fix subunit setupClass/setupModule failure handlingRichard Purdie2019-05-091-2/+2
| | | | | | | The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk: use bash to execute SDK test commandsMikko Rapeli2019-05-092-2/+3
| | | | | | | | | | | | | The commands only work with with bash. If /bin/sh is dash like in Debian, the command execution fails with errors like: Standard Output: /bin/sh: 5: export: --sysroot: bad variable name and all SDK tests fail. Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Handle unexpectedSucessesRichard Purdie2019-05-091-2/+8
| | | | | | | | | | Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-09157-126/+577
| | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Drop OETestIDRichard Purdie2019-05-0966-532/+8
| | | | | | | | These IDs refer to testopia which we're no longer using. We would now use the test names to definitively reference tests and the IDs can be dropped, along with their supporting code. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/ssh: Avoid unicode decode exceptionsRichard Purdie2019-05-011-2/+2
| | | | | | | | | | | | | This code really needs to be rewritten to not split potential multibyte characters, for now work around it to avoid exceptions like: File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run data = reader.read(1024, 4096) File "/usr/lib64/python3.6/codecs.py", line 503, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* manual compliance: remove bits done at runtimeArmin Kuster2019-04-291-89/+1
| | | | | Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ltp_compliance: add new runtimeArmin Kuster2019-04-291-0/+104
| | | | | | | | | | | | | | | test runtimes in sec. AIO: 14 MEM: 94 MSG: 89 SEM: 30 SIG: 194 THR: 399 TMR: 867 TPS: 23 Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logparser: Add LTP compliance sectionArmin Kuster2019-04-291-0/+37
| | | | | Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ltp: add runtime testArmin Kuster2019-04-291-0/+124
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds the framework for running ltp tests. Here are some times: math: 61 syscalls: 3957 dio: 18472 io: 29 mm: 551 ipc: 48 sched: 165 nptl: 46 pty: 37 containers: 52 controllers: 9625 filecaps: 27 cap_bounds: 27 fcntl-locktests: 29 connectors: 27 timers: 37 commands: 165 net.ipv6_lib: 30 input: 29 fs_perms_simple: 31 fs: 3476 fsx: 30 fs_bind: 28 fs_ext4: 28 cve: 675 Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logparser: Add decoding ltp logsArmin Kuster2019-04-291-0/+25
| | | | | Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto2019-04-261-4/+16
| | | | | | | | | | | | | | | | Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Fix typo in previous commitRichard Purdie2019-04-091-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemurunner: Dont mix binary and non-binary stringsRichard Purdie2019-03-251-0/+1
| | | | | | | self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add in a decode since its being used as a string. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: updated test id namingYeoh Ee Peng2019-03-192-93/+93
| | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file, where test_module match the file name itself. This file was using test_module name that does not match the file name itself. Fixed test_module name as well as the test_suite name. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/toaster: transfer manual toaster test cases to oeqaYeoh Ee Peng2019-03-182-0/+3742
| | | | | | | | | | | | | | | | | | | | As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/kernel-dev.json: test id updatedsangeeta jain2019-03-181-18/+18
| | | | | | | | | | | | | | | All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/compliance-test.json: test id updated and missing test cases addedsangeeta jain2019-03-181-4/+56
| | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/compliance-test.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. Added 2 missing test cases for Beaglebone Stress Test. Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removedsangeeta jain2019-03-181-220/+46
| | | | | | | | | | | | | | | | | | | Two changes made in oeqa/manual/bsp-hw.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases. Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* metadata.py: return info after fallback methodsChen Qi2019-03-071-0/+1
| | | | | | | | | | | If python3 git module is not installed, we fall back to some git command to get info. A previous commit (b8d22ed6) accidently deleted the return statement, causing errors like below. Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* imagefeatures: Add test to blacklist building busyboxTom Rini2019-03-061-0/+26
| | | | | | | | | | Now that we have a packagegroup that can be used to replace the overall functionality of busybox as base-utils, add a test that we can continue to build a fairly complex image without busybox being available. This also doubles as documentation-by-test of how to do this. Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* masterimage: remove code duplicationAndré Draszik2019-03-041-40/+0
| | | | | | | | | | | | Two identical classes exist for SystemdbootTarget, introduced in commit f9a61d3400ad ("gummiboot: Remove/change gummiboot references with systemd-boot") poky commit 2dce2648e3d8 Remove the duplicated copy. Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: add tests for virgl GL accelerationAlexander Kanavin2019-03-041-0/+66
| | | | | | | | Note that the tests require that the host machine has a X display, has mesa development files installed and is able to create OpenGL contexts. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: Add multiconfig testRichard Purdie2019-03-031-0/+43
| | | | | | | | | Add a test for a multiconfig build which mixes tiny and musl builds along with using the mcextend class to combine and package multiple images into another image. This gives the multiconfig a decent test in a scenario users may use. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* parselogs.py: ignore regulatory.db load failure messages for x86*Naveen Saini2019-03-031-0/+2
| | | | | | | [YOCTO #13098] Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/metadata: Add commit_count to fallback logicRichard Purdie2019-02-281-2/+4
| | | | | | | Currently if python3-git isn't installed we can get odd behaviours when the commit_count is absent. Avoid this set of bugs by adding a fallback here. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: Drop unused importRichard Purdie2019-02-281-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/gitarchive: Handle case where parent is only on originRichard Purdie2019-02-251-2/+2
| | | | | | | | | | | | | The parent code currently assumed that any parent branch is locally checked out which may not be the case. Use the local branch by default but fall back to the origin. This also means removing the later saftey check as the branch may not exist locally. This fixes the autobuilder resulttool test pushing code. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improvements to allow integration to the autobuilderRichard Purdie2019-02-211-58/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2 (git 2.18.0) at 2022-08-16 19:13:29 +0000