Age | Commit message (Collapse) | Author |
|
This is particularly useful when setting up GL tests.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous code assumed builddir and meta-selftest are in the same dir, but
this isn't always true, builddir can be anywhere, use get_test_layer() to
locate meta-selftest can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip
and port attribute to enable connection with qemu running with slirp.
[YOCTO#10713]
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking
the regexs correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no point in adding empty log entries to the json result files, only
add them if there is log data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.
The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.
Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.
Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.
Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
skipIfInDataVar will skip a test if a value is in a certain variable.
Signed-off-by: Richard Leitner <richard.leitner@skidata.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its a pain to have to wait until oe-selftest finishes to see the
failures for example.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Periodically we'd see:
NOTE: core-image-sato-1.0-r0 do_testsdk: ======================================================================
NOTE: core-image-sato-1.0-r0 do_testsdk: ERROR: broken-runner
NOTE: core-image-sato-1.0-r0 do_testsdk: ----------------------------------------------------------------------
NOTE: core-image-sato-1.0-r0 do_testsdk: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 122, in _run_test
test.run(process_result)
File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 1194, in run
protocol = TestProtocolServer(result, self._passthrough, self._forward)
File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 514, in __init__
stream = stream.buffer
AttributeError: '_io.StringIO' object has no attribute 'buffer'
which seems to occur if a result arrives before all the runner threads
have started. The runner's result handling changes sys.stdout to a buffer
temporarily which can be seen in other threads and it can sometimes fail.
Since the tests are running in a separate process we don't need this buffer
handling in the concurrent case so only set when not parallelising. The
concurrent class handle setting buffer mode internally.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this, error output such as that in the teardown can be lost
and processes may recieve signals they're not expecting causing other
strange errors.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
oe-selftest -r <invalid test name>
currently triggers a traceback. Ensure this doesn't happen and the user gets a
sensible error message.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Checking if the dependency had any failure is unreliable, for example
if the underlying data doesn't get transferred and the list is empty,
success of the dependency is assumed.
Since we now have success data available, change the code to use it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This isn't present on modern distros by default and doesn't work with
testtools, needing multiple code paths in the code. Remove it in favour
of finding a better replacement for results collection/analysis.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There doesn't appear to be any reason we need this _results indirection
any more so remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have several options for parallel processing in oeqa, parallel
execution of modules, threading and mulitple processes for the runners.
After much experimentation is appears the most scalable and least
invasive approach is multiple processes using concurrenttestsuite
from testtools. This means we can drop the current threading code
which is only used by the sdk test execution.
oeqa/decorator/depends: Remove threading code
Revert "oeqa/sdk: Enable usage of OEQA thread mode"
This reverts commit adc434c0636b7dea2ef70c8d2c8e61cdb5c703b1.
Revert "oeqa/core/tests: Add tests of OEQA Threaded mode"
This reverts commit a4eef558c9933eb32413b61ff80a11b999951b40.
Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode"
This reverts commit d3d4ba902dee8b19fa1054330cffdf73f9b81fe7.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows oe-selftest to take a -j option which specifies how much test
parallelisation to use. Currently this is "module" based with each module
being split and run in a separate build directory. Further splitting could
be done but this seems a good compromise between test setup and parallelism.
You need python-testtools and python-subunit installed to use this but only
when the -j option is specified.
See notes posted to the openedmbedded-architecture list for more details
about the design choices here.
Some of this functionality may make more sense in the oeqa core ultimately.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently some tests run in buffer mode and some don't. Those that don't can
corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent.
If there is useful output on stdout/stderr, it will be displayed if the test
fails.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current code assumes if something isn't a failure of some
kind, it was a pass. When test case IDs weren't matching, this lead
to very confusing output where things would fail, then be listed as
passing.
This adds code to track successes, ensuring we don't end up in this
position again with unmatched entries being listed as UNKNOWN.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We can directly compare the test case IDs rather than representations,
then if we're using subunit to split the tests, the comparisions still
work as intended.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When running testimage task for core-image-sato-sdk, the following
error appeared.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte
Checking the codes, I found it's caused by setting a 1024 limit for the
read method of the StreamReader object.
Comments from the manual:
"""
The chars argument indicates the number of decoded code points or bytes to
return. The read() method will never return more data than requested, but
it might return less, if there is not enough available.
"""
When running `systemctl status --full' on target, this error occurs.
This patch increase the bytes limit to 4096 to fix the error.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The var might not be set, resulting in unexpected error.
RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR
The above error is due to MULTILIBS being not set, which is the default
for OE. This patch fixes this problem.
Also, the debugging message in skipIfNotInDataVar is currently confusing.
Instead of
DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test
it should be
DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test
This patch also fixes it.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use 'cls' instead of 'obj' to better reflect that registerDecorator
actually serves as a class decorator.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
OETestFilter is a subclass of OETestDecorator. It wants to make
use of @abstractmethod decorator. But such decorator requires
metaclass to be ABCMeta to have effect. So add it now to achieve
the designed behaviour.
Comments from python's manual:
"""
Using this decorator requires that the class's metaclass is ABCMeta
or is derived from it.
"""
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In previous OEQA, having 'auto' in TEST_SUITES results in executing
as many test cases as possible.
This behaviour is broken for now. From the codes in core/loader.py,
I can see that it tries to use another keyword 'all'. But in fact,
it does not work.
I've checked the current manual. The manual says using 'auto'.
Below is the current information in manual.
"""
Alternatively, you can provide the "auto" option to have all applicable
tests run against the image.
TEST_SUITES_append = " auto"
"""
So we should restore this behaviour. This patch does so.
Also, output warning message is some module is named as 'auto', as this
is a reserved keyword.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some test cases (eSDK.oeSDK*, runtime_test/*) does not match
with current regex, fix it accept all.
[YOCTO #12385]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since we are depercating the ELF image type, we should not test it
here either.
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The value could be False in which case we should pass that through.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This ensures the console is kept clear of confusing output but that
the main logs contain good debugging information.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The logger handling in oeqa was confused at best. This patch:
a) Passes in a logger through various qemu runner pieces
b) Uses that logger consistently in the code
c) Creates a logger for QemuRunner outside the bitbake namespace
meaning we don't conflict with the tinfoil logging changes
The result of this is more consistency. For runtime tests in testimage,
the logs always contain the debug info, nothing is shwon on the console.
For the oe-selftests, logs are intercepted and only shown if the test
fails.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In general we don't need to see the output of runqemu however if it fails
we do. Use the buffer option that already exists in TestResult but allow
us to trigger it on a per test basis.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For test modules in a package, e.g., oelib.license, running
`oe-selftest -r oelib.license` or `oe-selftest -r
oelib.license.TestSimpleCombinations` would fail with a message that
the specified test cases could not be found. This was due to the
parsing in _built_modules_dict(), which failed to distinguish between
<package>.<module>.<class> and <module>.<class>.<testcase> and treated
both cases as the latter.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This can avoid UnicodeDecodeError error.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There might be wild strings when read from target (especially when
reading ptest results), replace the errors to avoid breaking the test.
Fixed: (Not always happen)
$ bitbake core-image-sato -ctestimage
[snip]
status, output = self.target.run('ptest-runner', 0)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run
status, output = self._run(sshCmd, processTimeout, True)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run
status, output = SSHCall(command, self.logger, timeout)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall
run()
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run
output = process.communicate()[0].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte
[YOCTO #11547]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A new command line argument (-R, which is the oposite of current -r) that allows
to run all test cases except the ones indicated through the command line.
Some command line examples:
* Run all except the distro test case:
$ oe-selftest -R distrodata
* Run all except the archiver test case and a single bblayers unit test
$ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove
[YOCTO #11847]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Since OEQA framework uses Python logging functionality to report test
results there is a class that wraps PyUnit writes into logging commands
(OEStreamLogger), so don't buffer the actual test execution to have
insight of what is currently executing.
This fix will change a little the test output format adding an '\n'
previous the test result, for example:
From:
test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok
To:
test_nonmatching_checksum (lic_checksum.LicenseTests)
... ok
This is because the new line added by the PyUnit StreamLogger because
currently we don't have a manner to identify when a test execution
starts at report level (write msg).
[YOCTO #11827]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
found
If some test module/case is specified to run and isn't found the OEQA
framework didn't notice it, so complete the implementation using
modules_required and validate for the test case prescense.
Raise an exception when the test module/case required isn't found.
[YOCTO #11645]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python versions has different features from branches 3.4.x, 3.5.x and
3.6.x, i expected in wrong mode that was incremental for example changes
in 3.4.4 be in 3.5.x but that's not true.
The _make_failed_test internal method differs and is only available in
certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not
realeses have been made including it.
So to avoid futher problems inspect the _make_failed_test and generates
function definition according what parameters are needed, the unique
supossition is that exception argument is always passed.
Related to,
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
Python unittest change the signature of the _make_failed_test
after python 3.4.4 don't pass the method name.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.
[YOCTO #11632]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The details of a test failure is upper on the unittest output
so don't log twice the actual failure.
[YOCTO #11622]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.
[YOCTO #10828]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|