| Age | Commit message (Collapse) | Author |
|---|
|
Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The current failure mode doesn't show us what the logs actually looked like
and later cleans can lose them. Show the whole log in case of failure
to aid debugging intermittent problems on the autobuilder.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids problems where shutil.remove will error with:
File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'
when there are races over file deletion (gpg agent may be slow to exit).
We already worked around speed and race issues in bb.utils.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This should speed the test up signficiantly without any loss of functionality
for the purposes of the test.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We ideally don't want to use gpg from the host. This is straightforward for package
management but not for sstate.
For sstate, create a second build directory to run the test in using gnupg-native
from the original build directory.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Raising an assertionError in the class setup isn't a particuarly good way to
indicate gpg isn't installed. Instead skip the tests if the required binary
isn't present. For the signing tests we do require it to be present and can't
use a prebuilt one.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip
and port attribute to enable connection with qemu running with slirp.
[YOCTO#10713]
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Enable qemurunner for slirp. Retrieved the ip & port from host machine
to connect to qemu from host machine.
[YOCTO#10713]
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Splits the SDK test implementation into configurable Python classes. The
classes used for the normal and extensible SDKs are
${TESTSDK_CLASS_NAME} and ${TESTSDKEXT_CLASS_NAME} respectively.
This allows SDK machines to override the classes used to implement the
tests. For the traditional SDK, a common "run()" function is provided by
the class (oeqa.sdk.testsdk.TestSDK), with several hook member functions
that can be overridden in child classes, making it easier to have
consistent behavior. The extensible SDK class
(oeqa.sdkext.testsdk.TestSDKEXT) also has a common "run()" function, but
no hooks have yet been added as there is not currently a known use case
for create derived classes.
These changes should be purely organizational; no functional changes
have been made to either the standard SDK or extensible SDK tests.
[YOCTO #13020]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).
Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.
Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
warnings
Use our own lsb function instead as used elsewhere by the codebase.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix the warnings:
meta/lib/oeqa/utils/qemurunner.py:250: DeprecationWarning: invalid escape sequence \.
ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
meta/lib/oeqa/utils/qemurunner.py:343: DeprecationWarning: invalid escape sequence \-
if re.search("root@[a-zA-Z0-9\-]+:~#", output):
poky/meta/lib/oeqa/utils/qemurunner.py:350: DeprecationWarning: invalid escape sequence \-
if re.search("root@[a-zA-Z0-9\-]+:~#", output):
meta/lib/oeqa/utils/qemurunner.py:448: DeprecationWarning: invalid escape sequence \-
if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data):
by correctly marking the regexs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid the warning:
meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
In this case importlib is a direct replacement.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid warnings such as:
meta/lib/oeqa/utils/commands.py:213: ResourceWarning: unclosed file <_io.BufferedReader name=4>
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking
the regexs correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Mark up these tests as needing a compiler, make and kernel source code
as appropriate, the image feature requirements can then be retired.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the OEHasPackage decorator to a variety of tests so they determine
automatically if they should run against a given image.
To ensure tests can do this we need to move target operations such
as scp commands into the tests and out of the class startup/teardown.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
execution
Usually skipped testcase output "SKIPPED"
[snip serial execution]
|RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: SKIPPED (0.01s)
|RESULTS - python.PythonTest.test_python3 - Testcase -1: SKIPPED (0.01s)
[snip serial execution]
But if enable multiprocess execution, skipped testcase output "UNKNOWN" status
[snip enable multiprocess execution]
|RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: UNKNOWN
|RESULTS - python.PythonTest.test_python3 - Testcase -1: UNKNOWN
[snip enable multiprocess execution]
Here is my investigation:
There is a class pairs TestProtocolClient and TestProtocolServer
provided by python3-subunit. The TestProtocolClient generates a
subunit stream of TestResult from a test run, and TestProtocolServer
parses the stream of subunit TestResult.
The class ProtocolTestCase is a unittest.TestCase adapter and it
uses TestProtocolServer to parse the stream of subunit TestResult.
In Yocto testsdk, it forks multiple processes to execute testcases
and use TestProtocolClient to generate TestResult stream; and then
it creates multiple threads to use ProtocolTestCase to parse stream
of subunit TestResult through pipe; finally it passes multiple
ProtocolTestCase as TestCase instance to main process and output
status result.
The problem point is TestProtocolServer parses `skip:' directive
after reading a `test:' directive. Without `test:' directive,
`skip:' directive will be ignored. All above requires SkipTest should
be raised inside a test method rather than setUpClass method.
Throwing SkipTest inside setUp works correctly
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without doing this, the code can and sometimes does try and do "rm ~/"
which I think we'd all prefer it didn't.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should only be wiping out things on target if the tests have
actually run.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The nightly-build-appliance target already tests this particular build.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should not be finding GStreamer issues right around
release time but that's what we seem to do.
Port gstreamer tests from Intel Ref Kit.
https://github.com/intel/intel-iot-refkit/blob/master/meta-iotqa/
lib/oeqa/runtime/multimedia/gstreamer/gstreamer_cli.py
Add tests for gstreamer commandline tools (gst-inspect-1.0
& gst-launch-1.0).
[YOCTO #11418]
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The upstream source tarball checksums changed. Use the copy from our source
mirror to avoid failures.
[YOCTO #12979]
Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
break tests
Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
['./',
'./etc/',
- './etc/build',
'./etc/default/',
'./etc/default/postinst',
Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.
Standardize selftest json output file to LOG_DIR.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've had a number of occasions where the Yocto Project source mirrors have not
been complete or functioning correctly. This adds a test so that if this happens
we find out out it sooner.
It also only works over http meaning we should be able to test that anyone behind
an http only proxy (no git protocol) also has functional fetches for OE-Core and
layers built by the core of the project.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The python git module may or may not be enabled, allow this code to
function without it, falling back to the same method as metadata_scm.bbclass
uses. This will be cleaned up in the next round of feature development.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no point in adding empty log entries to the json result files, only
add them if there is log data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tweak the preceeding commit to:
* Add STARTTIME to the identifier to make it unique
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.
By default, json testresult file will be written to "oeqa"
directory under the oe-selftest log directory.
To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.
The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.
Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.
Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
oe-selftest's devtool tests have been broken since commit 2457cd57
(oe-selftest: devtool: avoid parallel races by using temporary copy of
core) if meta is a symbolic link.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.
Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It appends TCLIBCAPPEND to TMPDIR in meta/conf/distro/defaultsetup.conf:
TMPDIR .= "${TCLIBCAPPEND}"
It affects some oe selftest cases in sstatetests.py. Reset TCLIBCAPPEND
for these cases.
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current error message is like:
kobject-example.kodoesn't exist
Add a space so that it looks like:
kobject-example.ko doesn't exist
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Building an image takes time and isn't necessary to test taskexp,
we can drop this step.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The autobuilder runs bitbake-selftest already so we don't need
to execute this manually.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia, manual test cases
need to be migrated to OEQA. These manual test case json files
will serve two use cases.
Use case#1: as input to the future commandline-based manual test
runner script, where this script will display actions and expected
result information in user friendly text, then it will capture
user input for test result and log, finally it will write test
result and log into existing standardize test result json format
from OEQA framework for automated tests.
Use case#2: QA will open and read these json file manually for
planning manual test execution. Any reader interested in
understanding manual test cases will open and read these files.
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After allarch was disabled in multilib, this test broke. Fix the test to account
for that change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sigchld handler in runqemu can raise a SystemExit when qemu shuts down.
Rather than backtracing, accept this as a successful test result.
ERROR: runqemu.QemuTest.test_qemu_can_shutdown (subunit.RemotedTestCase)
testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 183, in test_qemu_can_shutdown
qemu_shutdown_succeeded = self._start_qemu_shutdown_check_if_shutdown_succeeded(qemu, shutdown_timeout)
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 175, in _start_qemu_shutdown_check_if_shutdown_succeeded
time.sleep(1)
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/qemurunner.py", line 100, in handleSIGCHLD
raise SystemExit
SystemExit
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The qemu shutdown can race with the signal handler removal leading to
confusing tracebacks on slower/loaded systems.
Remove the signal handler first before shutting down.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The Navit recipe can depend on fribidi which is now part of oe-core. Since the
pkgconfig lookup is based on pkgdata the results can change depending on what
has been built, which leads to occasional failures.
Build gtk+ before the test to ensure that the pkgdata is populated with more of
the dependencies, and add fribidi to the generated DEPENDS checklist.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this, we see errors if gpg is missing from the host system
for "oe-selftest -r runtime_test.TestImage.test_testimage_dnf".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BPF Linux ELF objects are generated with kernel-selftests with
>= 4.18 kernel and when clang is enabled which packages BPF objects
into packages, therefore recongnise this as a valid ELF target
Add a selftest for BPF
Do not flag BPF objects in target, since they pretty much will be ok for
most of kernels architectures we care do support BPF
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Paul Eggleton
Richard Purdie
Yeoh Ee Peng
Joshua Watt
Hongxu Jia
Mohamad Noor Alim Hussin
Peter Kjellerstedt
Kai Kang
Chen Qi
Ross Burton
Khem Raj