| Age | Commit message (Collapse) | Author |
|---|
|
This can avoid UnicodeDecodeError error.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
There might be wild strings when read from target (especially when
reading ptest results), replace the errors to avoid breaking the test.
Fixed: (Not always happen)
$ bitbake core-image-sato -ctestimage
[snip]
status, output = self.target.run('ptest-runner', 0)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run
status, output = self._run(sshCmd, processTimeout, True)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run
status, output = SSHCall(command, self.logger, timeout)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall
run()
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run
output = process.communicate()[0].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte
[YOCTO #11547]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A new command line argument (-R, which is the oposite of current -r) that allows
to run all test cases except the ones indicated through the command line.
Some command line examples:
* Run all except the distro test case:
$ oe-selftest -R distrodata
* Run all except the archiver test case and a single bblayers unit test
$ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove
[YOCTO #11847]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Since OEQA framework uses Python logging functionality to report test
results there is a class that wraps PyUnit writes into logging commands
(OEStreamLogger), so don't buffer the actual test execution to have
insight of what is currently executing.
This fix will change a little the test output format adding an '\n'
previous the test result, for example:
From:
test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok
To:
test_nonmatching_checksum (lic_checksum.LicenseTests)
... ok
This is because the new line added by the PyUnit StreamLogger because
currently we don't have a manner to identify when a test execution
starts at report level (write msg).
[YOCTO #11827]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
found
If some test module/case is specified to run and isn't found the OEQA
framework didn't notice it, so complete the implementation using
modules_required and validate for the test case prescense.
Raise an exception when the test module/case required isn't found.
[YOCTO #11645]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python versions has different features from branches 3.4.x, 3.5.x and
3.6.x, i expected in wrong mode that was incremental for example changes
in 3.4.4 be in 3.5.x but that's not true.
The _make_failed_test internal method differs and is only available in
certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not
realeses have been made including it.
So to avoid futher problems inspect the _make_failed_test and generates
function definition according what parameters are needed, the unique
supossition is that exception argument is always passed.
Related to,
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
Python unittest change the signature of the _make_failed_test
after python 3.4.4 don't pass the method name.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.
[YOCTO #11632]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The details of a test failure is upper on the unittest output
so don't log twice the actual failure.
[YOCTO #11622]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.
[YOCTO #10828]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Our filtering allows to specify which tests to run using,
<module_name>.[test_class].[test_name]
But the module name logic was restricted to only accept one level,
for example: runtime_test vs oelib.types, to support multiple
submodules use only the first part for filtering.
This allows to run the whole tests in a module with more than tree
levels.
Due to the ambiguity on the test filtering options with test cases
with more than tree levels the supported sytnax is,
<module>
or
<module>.[submoduleN].[test_class].[test_name]
[YOCTO #11632]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Is better to log the summary at end to see in an easy way
the actual result of the test run.
[YOCTO #11622]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
After Python 3.5 the init method contains initialization of
_loading_packages set to avoid infinite loops in recursive
loading.
Fix,
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 70, in <module>
ret = main()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 57, in main
results = args.func(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 214, in run
rc = self._internal_run(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 175, in _internal_run
self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/context.py", line 51, in loadTests
self.suites = self.loader.discover()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/loader.py", line 268, in discover
pattern='*.py', top_level_dir=path)
File "/usr/lib/python3.5/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/usr/lib/python3.5/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/usr/lib/python3.5/unittest/loader.py", line 473, in _find_test_path
self._loading_packages.add(name)
AttributeError: 'OETestLoader' object has no attribute '_loading_packages'
Signed-off-by: Anibal Limon <anibal.limon@linux.intel.com>
|
|
In order to maintain compatibility with oe-selftest, the
load_test_components needs to be re-used, so the script
executor needs to pass to only load components supported
by certain script (oe-test, oe-selftest).
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
|
|
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This pre runner will serve to allow Test components executes code
previously of the run a suite.
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By default, the xml runner class prints out the docstring for every
unit test but it order to keep the same format as the standard
runner, avoid docstring output setting descriptions to False.
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The OETestLoader.modules must be an iterable object, otherwise
checking presence of specific modules with the 'in' operator fails
with the following command/error:
% oe-test core
.. - core - INFO - Running bitbake -p
Traceback (most recent call last):
File "..poky2/scripts/oe-test", line 108, in <module>
ret = main()
File "..poky2/scripts/oe-test", line 93, in main
results = args.func(logger, args)
File "..poky2/meta/lib/oeqa/core/context.py", line 235, in run
self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
File "..poky2/meta/lib/oeqa/core/context.py", line 53, in loadTests
self.suites = self.loader.discover()
File "..poky2/meta/lib/oeqa/core/loader.py", line 204, in discover
pattern='*.py', top_level_dir=path)
File "/usr/lib64/python3.4/unittest/loader.py", line 275, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/usr/lib64/python3.4/unittest/loader.py", line 339, in _find_tests
yield self.loadTestsFromModule(module)
File "..poky2/meta/lib/oeqa/core/loader.py", line 266, in loadTestsFromModule
and module.__name__ in self.modules \
TypeError: argument of type 'NoneType' is not iterable
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Most of the text include on this text refactor came from [1].
[1] http://lists.openembedded.org/pipermail/openembedded-architecture/2016-December/000351.html
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This will enable only run certain module tests and filter by
class and test name.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A common operation is to list tests, currently only selftest
support it, this changes enables this functionality into the
core framework.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add needed tests to validate the OEQA Threaded mode, the remaining
parts are tested by the OEQA without Threaded mode.
- test_loader.py: Add a test to validate rules when creating the
list of test suites.
- test_decorators.py: Add oetimeout test because the threaded mode
uses Timer instead of signal.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In python signals are only allowed to delivery into the main thread,
to support the threading mode test if the runner is threaded and
use threading.Timer instead.
There are some considerations like SIGALRM interrupts the execution
after N seconds but the Timer only starts a Thread to notice the
timeout and the exception will be raised when the test run ends.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The _skipTestDependency needs to know if the thread mode is
enabled because the _results are by thread.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This class sets the {loader,runner}Class to the threaded versions and
overrides loadTests method to be able specify the process_num.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The OETestRunnerThreaded overrides the run method of OETestRunner
it recieves a list of suites to be executed by a ThreadPool.
The new run method handles the ThreadPool creation and the
OETestResultThreaded fill.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The OETestResultThreadedInternal extends OETestResult to stores
results by Thread.
The OETestResultThreaded is a simple class that provides the
implementation of interfaces needed by outside like wasSuccesful,
stop, logSummary, logDetails.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This method is to assign results into the TestContext, create
an internal one to support change implementation in Thread version.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The OEStreamLoggerThreaded overrides OEStreamLogger to redirect
the PyUnit output to a logger.
Instead of log every line when comes the OEStreamLoggerThreaded
will buffer the PyUnit output and write everything at end of every
suite execution to don't have mixed suite outputs.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The threded module will implement Thread support into the OEQA
framework.
The OETestLoaderThreaded overrides discover to return a list of
suites every suite will be run by a separate Thread.
Some design considerations are:
- All the tests of a certain module needs to be run at one thread
because unittest framework needs to keep track of setUp{Module,
Class,} and tearDown{Module, Class,}.
- Tests that depends on other needs to be run at the same thread
because OEQA framework look at the status of dependant test to
decide if skip or not, this constraint can be change in the future
but a sync mechanishm is needed to implement between the Threads.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I override this method before for keep track of results and forget
to remove it, now isn't need.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The OEStreamLogger class is used for redirect PyUnit output
to a certain logger so there is not need to expose at level
of OETestContext because only OETestRunner needs to know.
[YOCTO #11450]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was found a case (see the bugzilla entry) where two runners were running at the same
second, creating identical test result folders, so one of them (the second runner)
was not able to create the folder because the other has already created it,
raising the following exception (many text was removed from log)
NOTE: Executing RunQueue Tasks
NOTE: Running task 1 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext)
NOTE: Running task 2 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-sato/images/core-image-sato.bb:do_testsdkext)
NOTE: recipe core-image-sato-1.0-r0: task do_testsdkext: Started
NOTE: recipe core-image-minimal-1.0-r0: task do_testsdkext: Started
.
.
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_testsdkext(d)
0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 188, function: do_testsdkext
0184:
0185:testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
0186:
0187:python do_testsdkext() {
*** 0188: testsdkext_main(d)
0189:}
0190:addtask testsdkext
0191:do_testsdkext[nostamp] = "1"
0192:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 171, function: testsdkext_main
0167: except Exception as e:
0168: import traceback
0169: bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
0170:
*** 0171: result = tc.runTests()
.
.
File: '/usr/lib64/python3.5/os.py', lineno: 241, function: makedirs
0237: cdir = bytes(curdir, 'ASCII')
0238: if tail == cdir: # xxx/newdir/. exists if xxx/newdir exists
0239: return
0240: try:
*** 0241: mkdir(name, mode)
0242: except OSError:
0243: # Cannot rely on checking for EEXIST, since the operating system
0244: # could give priority to other errors like EACCES or EROFS
0245: if not exist_ok or not path.isdir(name):
Exception: FileExistsError: [Errno 17] File exists: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/build/TestResults_20170409130114'
[YOCTO #11318]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Underscore modules are meant to be run only when manually added to the test
suite, so far another mechanisms are in place to make this happen with
runtime, sdk, and esdk (mostly in test* bbclasses).
This will add such functionality in the core framework so other specific
frameworks can take use this without adding something else.
[YOCTO #10980]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If importing a test with the same name as a built-in module,
it will silently import the built-in and check for tests in
built-in module. This happened with syslog module in debian
based machines, so add a raise to avoid this behavior.
[YOCTO #10978]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
With this we get the class that is actually having the problem,
not just a TypeError with an unknown class causing the error.
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The idea on getTarget is to use kwargs to send custom variables
to different targets, instead of this, a new variable was added
(just used for custom targets) and this broke testexport. So
in order to fix it, just add the custom variable to kwargs.
This fixes the use of getTarget() in testexport class that was
introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf.
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Without this an exception would occur
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add 'elf' as also supported by OEQemuTarget.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Change conditional to avoid losing stack traces when log
into the logger. The logic only needs to don't log empty lines.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The run-tests option is optional so if isn't specified set
to None instead of crash on split().
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a complementary fix to commit 7c552996:
[ meta: remove True option to getVar calls ]
it intended to remove all True option to getVar calls, but there are
still some remaining.
Search made with the following regex: getVar ?\((.*), True\)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.
Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.
[YOCTO #10964]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This allows to use the package install feature with
the new OEQA framework.
[YOCTO #10234]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
|
|
This will add functions to get module file path from a test
case or a complete suite.
[YOCTO #10234]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
|
|
This adds qemu target to be used in testimage.
It uses the current QemuRunner class in order
to boot and control qemu.
[YOCTO #10231]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
|
Robert Yang
Leonardo Sandoval
Aníbal Limón
Randy MacLeod
Mariano Lopez
Juro Bystricky
Ming Liu