summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/core
AgeCommit message (Collapse)Author
2019-12-16oeqa/target/ssh oeqa/target/qemu: expose server listening port to testsAndré Draszik
Allow tests to access the listening port as just introduced. Note that when using qemu this infrastructure shouldn't be needed, but we still need to set the port to 0 so that a listening port is determined automatically (e.g. by the python http server). Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-05oeqa/concurrencytest: Optimise for autobuilder/clobberdirRichard Purdie
We're seeing huge delays on the autobuilder during oe-selftest builddir deletion. For example there is a currently running selftest we could do with the results from and its been going 13 hours, at least 8 of which was in deletion of the builddirs. There are a variety of ways we could solve this problem however the autobuilder has a mechanism for deferred deletion, "clobberdir" which it already uses for this kind of work. Whilst in general hardcoding things like this is horrible, I believe in this case the benefits (and resulting improvements on my sanity if nothing else) mean this is a case where we should do it. If/as/when someone can come up with a better solution that is fine and this can be replaced. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14oeqa/core: Add qemu checksArmin Kuster
Some test should not be run in QEMU systems so add some checks to make that easier Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14oeqa/core: Add a check for MACHINEArmin Kuster
Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-19oeqa/core/decorator: add skipIfFeatureAndré Draszik
skipIfFeature will skip a test if a given DIST_FEATURE or IMAGE_FEATURE is enabled. Signed-off-by: André Draszik <git@andred.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27oeqa/core/case.py: Encode binary data of logNathan Rossi
Do not decode the log content into a string only to re-encode it as binary data again. Some logs might un-intentionally contain bytes that do not decode as utf-8, as such preserve the log file content as it was on disk. Handle the decoding on the resulttool side, but also handle the failure to decode the data. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27oeqa/core/utils/concurrencytest.py: Handle exceptions and detailsNathan Rossi
Handle the streaming of exception content with details data. The testtools package allows both 'err' and 'details' kwargs but can only pass one of them to the parent. To handle the passing of exception traceback and details data at the same time, encode the traceback into the details object and remove the 'err' arg from the add* result call. This encodes the traceback similar to how 'err' is handled without any details object. Decoding is already done by testtools when the traceback is encoded in the details object. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19oeqa/concurrenttest: Use ionice to delete build directoriesRichard Purdie
Autobuilder type infrastructure can benefit from deletion of certain files as background IO due to the way Linux filesystem priority works. We have problems where build directories as part of oe-selftest being delete starves the running tasks of IO to the point builds take much longer to compelte. Having this option of running the deletion at "idle" helps a lot with that. Use the new option added to bb.utils.prunedir(). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-11oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpersNathan Rossi
Add the OEPTestResultTestCase class as a mix-in class to provide helper functions for interacting with ptestresults within the extraresults object generated by the test case. This class also provides default compression of log text and log files. Also add support to resulttool for decoding/decompressing log files embedded in the test results. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07oeqa/core/decorator: Fix super class modifying subclass tagsNathan Rossi
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07oeqa/core/runner.py: Fix OETestTag listingNathan Rossi
Use the __oeqa_testtags attribute added by OETestTag and display no, one or more tags separated by a comma. Also change the formatting of the output so that the list of tests is formatted as "<test> (<tags>)" and remove the table header for "id" (it is no longer available). Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07oeqa/core: Rework OETestTag and remove unused OETestFilterNathan Rossi
Rework OETestTag so that it does not rely on the existing decorator code base and instead inserts the tags into an attribute on the decorated target (e.g. class/type or method). This allows the use of OETestTag on classes and method. In order to filter tagged tests rework the loaders filtering code, removing the generic-ness (with validation and attributes/etc.) and replace it with a "tags_filter" parameter which is a function that filters a test based on the tags it has. This allows the loader user to filter on tags in more specific ways (e.g. include all untagged tests and any tests tagged with foo). Plumb all this through the context code and testing code. Update the associated tests to pass correctly with the changes. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07oeqa/core: Implement proper extra result collection and serializationNathan Rossi
Implement handling of extra result (e.g. ptestresult) collection with the addition of a "extraresults" extraction function in OETestResult. In order to be able to serialize and deserialize the extraresults data, allow OETestResult add* calls to take a details kwarg. The subunit module can handle cross-process transfer of binary data for the details kwarg. With a TestResult proxy class to sit inbetween to encode and decode to and from json. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06oeqa/core/tests: Fix test_data module testsNathan Rossi
These two tests relied on the context containing the results information. This was moved into the OETestResults class. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06oeqa/core/tests: Skip test_fail_duplicated_moduleNathan Rossi
At some point the modules in tests/cases/loader/invalid/ were removed, this test relied on having tests/cases/loader/invalid/oetag.py to trigger a ImportError in the loader module. Skip this test. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-21oeqa: avoid class setup method to run when skipping the whole classChen Qi
For now, even if we have specified to skip the whole module/class via command line, e.g., `oe-selftest -R gotoolchain', the class setup method is still run. This at least results in unnecessary builds, and at worst results in ERROR, if the setup method fails. So improve the skipping mechanism to avoid class setup method to run when specified to skip. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-21context.py: avoid skipping tests by meaningless command argumentChen Qi
Currently `oe-selftest -R a' will skip 'archiver' tests. This is not expected. Fix it so that the '-R' should be followed by actual module/class/test names. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18context.py: fix skipping functionChen Qi
The current codes to skip test cases are logically correct, but they do not work correctly in reality. It does skip the tests as the command line argument specifies, but the related information is wrong. e.g. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test" 2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test" The archiver.Archiver.xxx tests should be skipped by 'archiver' command line argument, not 'runtime_test'. Change to use a function generator to achieve the desired effect. After the change, the effect is as follows. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver) 2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers" 2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers" [snip] Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-13oeqa/target/ssh: Replace suggogatepass with ignoring errorsRichard Purdie
We continued to see encoding problems with ssh commands run in oeqa. After much research the conclusion was we should use ignore the errors since some occasional bad locale encoding is better than the unicode decoding issues we were seeing which crashed large parts of tests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09oeqa/concurrenttest: Patch subunit module to handle classSetup failuresRichard Purdie
Currently setupClass errors were not being mapped back to the failing tests and they were hence being marked as UNKNOWN and the test statistics were inaccurate. This is because whilst the errors were being encoded into the test results stream, the decoder doesn't cope with an error outside a testStart event. We patch in an addError handler to the outsideTest parser so that this does get handled in a way similar to the non-concurrent case. It would be nice if we didn't have to do this but there doesn't seem to be any other way to fix this other than forking subunit. We also make a minor change so another of our changes can cope with tests without a start time. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09oeqa/runner: Fix subunit setupClass/setupModule failure handlingRichard Purdie
The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09oeqa/core/runner: Handle unexpectedSucessesRichard Purdie
Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09meta/lib+scripts: Convert to SPDX license headersRichard Purdie
This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09oeqa: Drop OETestIDRichard Purdie
These IDs refer to testopia which we're no longer using. We would now use the test names to definitively reference tests and the IDs can be dropped, along with their supporting code. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01oeqa/ssh: Avoid unicode decode exceptionsRichard Purdie
This code really needs to be rewritten to not split potential multibyte characters, for now work around it to avoid exceptions like: File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run data = reader.read(1024, 4096) File "/usr/lib64/python3.6/codecs.py", line 503, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto
Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24testimage.bbclass: add support for passing runqemu paramsAlexander Kanavin
This is particularly useful when setting up GL tests. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26oeqa/concurrencytest: fix for locating meta-selftestRobert Yang
The previous code assumed builddir and meta-selftest are in the same dir, but this isn't always true, builddir can be anywhere, use get_test_layer() to locate meta-selftest can fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22oeqa/qemu: Add support for slirpYeoh Ee Peng
Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip and port attribute to enable connection with qemu running with slirp. [YOCTO#10713] Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22oeqa/qemu & runtime: qemu do not need ip input from externalYeoh Ee Peng
Qemu do not use the ip input from external. It will retrieve ip from QemuRunner instance and assign ip value. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16oeqa/concurrencytest: Avoid unclosed file warningsRichard Purdie
Avoid an unclosed file per thread warning when running selftests concurrently by closing the result stream. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-13oeqa/core/runner: Correctly markup regexsRichard Purdie
Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking the regexs correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-13oeqa/loader: Fix deprecation warningRichard Purdie
Clean up the warning: meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec() _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-13oeqa/runner: Sort the test result output by result classRichard Purdie
We want to see failures/errors listed last since this is the most easily visible part of the log on consoles or autobuilder output and makes human processing easier rather than having to scroll up and scan for a single failure. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-13oeqa/runner: Always show a summary of success/fail/error/skip countsRichard Purdie
Its useful to have the counts of success/failure/error/skipped at the end of the results to allow for easier human reading of what happened. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-06oeqa/runtime/ptest: Inject results+logs into stored json results fileRichard Purdie
This allows the ptest results from ptest-runner, run in an image to be transferred over to the resulting json results output. Each test is given a pass/skip/fail so individual results can be monitored and the raw log output from the ptest-runner is also dumped into the results json file as this means after the fact debugging becomes much easier. Currently the log output is not split up per test but that would make a good future enhancement. I attempted to implement this as python subTests however it failed as the output was too confusing, subTests don't support any kind of log output handling, subTest successes aren't logged and it was making things far more complex than they needed to be. We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest will pass currently and we don't want that to fail the whole image test run. Its assumed there would be later analysis of the json output to determine regressions. We do have to change the test runner code so that 'unexpectedsuccess' is not a failure. Also, the test names are manipuated to remove spaces and brackets with "_" used as a replacement and any duplicate occurrences truncated. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29oeqa/core/runner: Don't add empty log entriesRichard Purdie
There is no point in adding empty log entries to the json result files, only add them if there is log data. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29oeqa/core/runner: write testresult to json filesYeoh Ee Peng
As part of the solution to replace Testopia to store testresult, OEQA need to output testresult into single json file, where json testresult file will be stored in git repository by the future test-case-management tools. The json testresult file will store more than one set of results, where each set of results was uniquely identified by the result_id. The result_id would be like "runtime-qemux86-core-image-sato", where it was a runtime test with target machine equal to qemux86 and running on core-image-sato image. The json testresult file will only store the latest test content for a given result_id. The json testresult file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE), result (eg. PASSED, FAILED, ERROR), test log, and result_id. Based on the destination json testresult file directory provided, it could have multiple instances of bitbake trying to write json testresult to a single testresult file, using locking a lockfile alongside the results file directory to prevent races. Also the library class inside this patch will be reused by the future test-case-management tools to write json testresult for manual test case executed. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18oeqa/core/runner: refactor for OEQA to write json testresultYeoh Ee Peng
Refactor the original _getDetailsNotPassed method to return testresult details (test status and log), which will be reused by future OEQA code to write json testresult. Take the opportunity to consolidate and simplify the logic used to gather test status and log within the TestResult instance. Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23oeqa/core/decorator: add skipIfInDataVarRichard Leitner
skipIfInDataVar will skip a test if a value is in a certain variable. Signed-off-by: Richard Leitner <richard.leitner@skidata.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23oeqa/runner: Print any errors/failures earlyRichard Purdie
Its a pain to have to wait until oe-selftest finishes to see the failures for example. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23oeqa/runner: Use the proper logger functions instead of print()Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23oeqa/context: Only set buffer mode for non-concurrent testsRichard Purdie
Periodically we'd see: NOTE: core-image-sato-1.0-r0 do_testsdk: ====================================================================== NOTE: core-image-sato-1.0-r0 do_testsdk: ERROR: broken-runner NOTE: core-image-sato-1.0-r0 do_testsdk: ---------------------------------------------------------------------- NOTE: core-image-sato-1.0-r0 do_testsdk: testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 122, in _run_test test.run(process_result) File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 1194, in run protocol = TestProtocolServer(result, self._passthrough, self._forward) File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 514, in __init__ stream = stream.buffer AttributeError: '_io.StringIO' object has no attribute 'buffer' which seems to occur if a result arrives before all the runner threads have started. The runner's result handling changes sys.stdout to a buffer temporarily which can be seen in other threads and it can sometimes fail. Since the tests are running in a separate process we don't need this buffer handling in the concurrent case so only set when not parallelising. The concurrent class handle setting buffer mode internally. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23oeqa/concurrencytest: Ensure subunit streams are flushed at exitRichard Purdie
Without this, error output such as that in the teardown can be lost and processes may recieve signals they're not expecting causing other strange errors. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26oeqa/loader: Ensure invalid test names don't trigger a tracebackRichard Purdie
oe-selftest -r <invalid test name> currently triggers a traceback. Ensure this doesn't happen and the user gets a sensible error message. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-18oeqa/decorator: Improve reliabilityRichard Purdie
Checking if the dependency had any failure is unreliable, for example if the underlying data doesn't get transferred and the list is empty, success of the dependency is assumed. Since we now have success data available, change the code to use it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-16oeqa: Remove xmlrunnerRichard Purdie
This isn't present on modern distros by default and doesn't work with testtools, needing multiple code paths in the code. Remove it in favour of finding a better replacement for results collection/analysis. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-16oeqa/runner: Simplify codeRichard Purdie
There doesn't appear to be any reason we need this _results indirection any more so remove it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-16oeqa/core/threaded: Remove in favour of using concurrenttestsRichard Purdie
We have several options for parallel processing in oeqa, parallel execution of modules, threading and mulitple processes for the runners. After much experimentation is appears the most scalable and least invasive approach is multiple processes using concurrenttestsuite from testtools. This means we can drop the current threading code which is only used by the sdk test execution. oeqa/decorator/depends: Remove threading code Revert "oeqa/sdk: Enable usage of OEQA thread mode" This reverts commit adc434c0636b7dea2ef70c8d2c8e61cdb5c703b1. Revert "oeqa/core/tests: Add tests of OEQA Threaded mode" This reverts commit a4eef558c9933eb32413b61ff80a11b999951b40. Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode" This reverts commit d3d4ba902dee8b19fa1054330cffdf73f9b81fe7. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-16oeqa: Add selftest parallelisation supportRichard Purdie
This allows oe-selftest to take a -j option which specifies how much test parallelisation to use. Currently this is "module" based with each module being split and run in a separate build directory. Further splitting could be done but this seems a good compromise between test setup and parallelism. You need python-testtools and python-subunit installed to use this but only when the -j option is specified. See notes posted to the openedmbedded-architecture list for more details about the design choices here. Some of this functionality may make more sense in the oeqa core ultimately. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>