diff options
author | Yeoh Ee Peng <ee.peng.yeoh@intel.com> | 2019-02-14 13:50:37 +0800 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2019-02-21 12:31:50 +0000 |
commit | 78a322d7be402a5b9b5abf26ad35670a8535408a (patch) | |
tree | 44d3c4c8288bf58307c5ed9508f83a4d373a0aae /meta/lib/oeqa/files | |
parent | c66f848938c04e133259c5b6903dc592866ab385 (diff) | |
download | openembedded-core-contrib-78a322d7be402a5b9b5abf26ad35670a8535408a.tar.gz |
resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'meta/lib/oeqa/files')
-rw-r--r-- | meta/lib/oeqa/files/testresults/testresults.json | 40 |
1 files changed, 40 insertions, 0 deletions
diff --git a/meta/lib/oeqa/files/testresults/testresults.json b/meta/lib/oeqa/files/testresults/testresults.json new file mode 100644 index 0000000000..1a62155618 --- /dev/null +++ b/meta/lib/oeqa/files/testresults/testresults.json @@ -0,0 +1,40 @@ +{ + "runtime_core-image-minimal_qemuarm_20181225195701": { + "configuration": { + "DISTRO": "poky", + "HOST_DISTRO": "ubuntu-16.04", + "IMAGE_BASENAME": "core-image-minimal", + "IMAGE_PKGTYPE": "rpm", + "LAYERS": { + "meta": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-poky": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + }, + "meta-yocto-bsp": { + "branch": "master", + "commit": "801745d918e83f976c706f29669779f5b292ade3", + "commit_count": 52782 + } + }, + "MACHINE": "qemuarm", + "STARTTIME": "20181225195701", + "TEST_TYPE": "runtime" + }, + "result": { + "apt.AptRepoTest.test_apt_install_from_repo": { + "log": "Test requires apt to be installed", + "status": "PASSED" + }, + "buildcpio.BuildCpioTest.test_cpio": { + "log": "Test requires autoconf to be installed", + "status": "ERROR" + } + } + } +}
\ No newline at end of file |