summaryrefslogtreecommitdiffstats
path: root/meta/lib
AgeCommit message (Collapse)Author
2021-02-12meta: drop _PYTHON_SYSCONFIGDATA_NAME hacksAlexander Kanavin
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d3a81dd0e72a3495bfc7cc969c2bb806b666023d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-09wic/selftest: test_permissions also test bitbake imageLee Chee Yang
existing test case test_permissions use Wic command as standalone tools to create wic image and check that wic image for permissions. add extra steps to the test case to also check against image build using bitbake do_image_wic. Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 551ce73a90757ba43501fe5cf9ac84a7b77de549) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-09oeqa: wic: Add tests for permissions and change-directoryRicardo Ribalda Delgado
Make sure that the permissions and username are respected when using all the rootfs modifiers. Add tests for change-directory command Cc: Paul Barker <pbarker@konsulko.com> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4aad9531df44d1b0637bd559161702ad86861b46) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-09sstatesig: Add descriptive error message to getpwuid/getgrgid "uid/gid not ↵Tomasz Dziendzielski
found" KeyError If path is not owned by any user installed on target it gives insufficient error "getpwuid(): uid not found" which may be misleading. This exception occurs if uid/gid of path was not found in PSEUDO_PASSWD files, which simply means the path is owned by host user and there is host user contamination. Add more information to the exception message to make it easier for user to debug. [YOCTO #14031] Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 38540b59ed4ec8632e30a5fd6364b010d9da8470) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-08lib/oe/patch.py: Ignore scissors line on applying patchTomasz Dziendzielski
The "devtool modify" could remove message body before scissors line, so patches re-generated from git tree were incorrectly modified. Adding --no-scissors to "git am" invocation to prevent this behaviour. [YOCTO #12674] Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 13ea33fbd197b9ee3cf913d9995617115f22798f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-04package.bbclass: hash equivalency and pr serviceMark Hatle
When the PR service is enabled a number of small changes may happen to variables. In the do_package step a call to package_get_auto_pr will end up setting PRAUTO and modifying PKGV (if AUTOINC is there). PRAUTO is then used by EXTENDPRAUTO, which is then used to generate PKGR. Since this behavior typically happens BEFORE the BB_UNIHASH is calculated for do_package, we need a way to defer the expansion until after we have the unihash value. Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded to placeholder values is the easiest way to deal with this. All other variables are expanded as expected. In the next task, typically do_packagedata, we will then use the UNIHASH from the do_package to get the PR (AUTOPR) as well as generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC). The do_packagedata then translates the placeholders to the final values when copying the data from pkgdata to pkgdata-pdata-input. Also update the prservice test case. With unihash, just changing the do_package (via a _append) will not change the PR. So write the date to a specific file that is incorporated into the unihash to ensure it is always different for the test. Various assert messages were also updated to make it easier to figure out where/why a problem occured. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2e32f37b0e4abc438c8f60e673cd18a5cc110768) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-04cve_check: add CVE_VERSION_SUFFIX to indicate suffix in versioningLee Chee Yang
add CVE_VERSION_SUFFIX to indicate the version suffix type, currently works in two value, "alphabetical" if the version string uses single alphabetical character suffix as incremental release, blank to not consider the unidentified suffixes. This can be expand when more suffix pattern identified. refactor cve_check.Version class to use functools and add parameter to handle suffix condition. Also update testcases to cover new changes. Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5dfd5ad5144708b474ef31eaa89a846c57be8ac0) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-02-04cve-check: replace Looseversion with custom version classLee Chee Yang
The way distutils.version.LooseVersion compare version are tricky, it treat all these ( "1.0-beta2", "1.0-rc1", "1.0A", "1.0p2" and "1.0pre1") as greater version than "1.0". This might be right for "1.0A" and "1.0p1" but not for the rest, also these version could be confusing, the "p" in "1.0p1" can be "pre" or "patched" version or even other meaning. Replace Looseversion with custom class, it uses regex to capture common version format like "1.1.1" or tag format using date like "2020-12-12" as release section, check for following known string/tags ( beta, rc, pre, dev, alpha, preview) as pre-release section, any other trailing characters are difficult to understand/define so ignore them. Compare release section and pre-release section saperately. included selftest for the version class. [YOCTO#14127] Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6ced85e9ddd3569240f1e8b82130d1ac0fffbc40) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28selftest/devtool: Add modify_localfiles_only test checking symlink pathTomasz Dziendzielski
If recipe uses only file:// fetcher devtool unpacks sources under oe-local-files/ and adds symlink in source root directory. We need to verify if the symlink in subdirectory has correct path. See [YOCTO #13738] for details. Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 044c90d152bfd1b2a23272df39327ba4cd862dff) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28oeqa/selftest/cases/tinfoil.py: increase timeout 10->60s test_wait_eventYi Fan Yu
The test would timeout on autobuilders. This patch increases the timeout to 60s The test will now also exit as soon as we receive the 2 expected events Expected runtime is around 1s if successful Bug 14158 Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit be02aa9283f805de718badd5ea12c4968da8774f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28oeqa/ptest: print a warning if ptests failedAlexander Kanavin
This allows spotting ptest regressions without having hard ptest failures (for that full ptest stability should be achieved). Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 79b54d902df2788b0822b6c9cc14705ab00e6a5b) Signed-off-by: Steve Sakoman <steve@sakoman.com> (cherry picked from commit 5232c0b480eba4537c8db71c95189950011fb64d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28bitbake.conf: Prevent pyc file generation in pseudo contextPaul Barker
This also effectively reverts commit b6d30c21b0: bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta The contents of ${COREBASE}/meta were ignored as pyc files could be generated for the contents of the lib subdirectory if python modules were imported within a pseudo context. However this doesn't protect us from pyc files being generated in the lib directories for other layers. It's far better to tell python not to produce pyc files when running under pseudo (by setting the PYTHONDONTWRITEBYTECODE variable) as this will cover any location where pyc files could possibly be created. This variable is set in FAKEROOTBASEENV so that it applies to the bitbake-worker instance for fakeroot tasks, preventing pyc files from being generated for imports in python tasks themselves. Also add a test case to ensure that pyc files are not created in tasks which are executed under pseudo. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 73d538f20743017a44cea4c20dbe09a0327cfc71) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28lib/oe/path: Add canonicalize()Peter Kjellerstedt
oe.path.canonicalize() is used to canonicalize paths (i.e., remove symbolic links and "..", and make them absolute). It takes a string with paths separated by commas, and returns the canonicalized path in the same format. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 282b19c0e27488ec119f00fb2542ffdc1af54e2a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28uninative: Don't use single sstate for pseudo-nativeRichard Purdie
pseudo-native is a bit special. It conditionally compiles in support for xattr, statx and statvfs amongst other options. If a pseudo-native binary is used on a system where these functions are present but it wasn't compiled in we see hard to debug permissions problems. An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target oe-selftest which shows a cryptic error: File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target self.assertEqual(filelist1, filelist2) File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual self.fail(msg) File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail raise self.failureException(msg) AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] First differing element 0: '-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor' '-rwxr-xr-x root root /etc/init.d/mdmonitor' This is due to a version of pseudo without statx being used on a system where ls uses statx, hence the files are displayed as 6000.6000 instead of root.root. Avoid this by always building pseudo-native for the specific distro in question rather than using a universal sstate feed. This hopefully fixes one of the mysterious AB-INT issues. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6e3785a3f1f3cf68f5fe101cd6bebe91db165973) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28oeqa/selftest/runtime_test: Exclude gpg directory from pseudo databaseRichard Purdie
Avoid pseudo abort()s like: path mismatch [1 link]: ino 6295376 db '/tmp/oeqa-feed-sign-2mw7z81v/S.gpg-agent.yocto-native' req '/tmp/jwkivmu6'. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 189630ca6cdf7ceb6cf9b8f9d86c58997f505efc) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-28base/bitbake.conf: Enable pseudo path filteringRichard Purdie
This is a pretty big change to the way pseudo operates when used in OpenEmbedded. Normally, pseudo monitors and logs (adds to its database) any file created or modified whilst in a fakeroot environment. There are large numbers of files we simply don't care about the permissions of whilst in that fakeroot context, for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control directories, This change uses new functionality in pseudo to ignore these directory trees, resulting in a cleaner database with less chance of "stray" mismatches if files are modified outside pseudo context. It also should reduce some overhead from pseudo as the interprocess round trip to the server is avoided. There is a possible complication where some existing recipe may break, for example, we found a recipe which was writing to "${B}/install" for "make install" in do_install and since we listed ${B} as not to be tracked, there were errors trying to chown root for files in this location. This patch fixes a few corner cases in OE-Core when used with this new ignore list: * The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed to something else since that directory does need its root permissions * The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR} * package_write_* task output (the debs/rpms/ipks) are now owned by the build user so we don't want the file ownership information in the hashequiv outhash calculation even if they are built under pseudo. * The fontcache postinstall intercept is run under qemu outside of pseudo context so delete files it may delete up front where pseudo can see this. * SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached by bitbake. We therefore need to trigger reparsing if this changes, which means SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST. Rework the variables to handle this. This otherwise breaks some of our sstate tests in oe-selftest. * Ignore the temp directory wic uses for rebuilding rootfs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ad8f5532ffaead9a5ad13e1034fe9e5e1b7979f4) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-19lib/oe/utils: Return empty string in parallel_makeTomasz Dziendzielski
In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make function and if PARALLEL_MAKE is set to empty string then this variable is exported as "None" causing cmake to fail with: "'CMAKE_BUILD_PARALLEL_LEVEL' environment variable invalid number 'None' given." Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2f790ded554a52ac18d1c28002142f9c62abec8b) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-08selftest: Add argument to keep build dirPaul Barker
The oe-selftest code already keeps the selftest build directory in place if any tests failed. By default the build directory is deleted if all tests pass but there may be cases where it's desirable to keep this directory around, for example to compare intermediate files between passing and failing test runs. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 67aa7069dbe8f5f5f186eb67708ece5c4bd42976) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-05oe-pkgdata-util: Added a test to verify oe-pkgdata-util without parametersMilan Shah
A test is implemented on poky/meta/lib/oeqa/selftest/pkgdata.py to test the scenario when oe-pkgdata-util is executed without parameters and help is displayed. See [YOCTO #10726] for detailed bug information. Signed-off-by: Milan Shah <mshah@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6b4e2eafa6eb71ca94ccc8a18d05b473b352367c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-04oeqa/commands: Ensure sync can be found regardless of PATHRichard Purdie
Avoid command not found errors shown in selftest logs due to changes to PATH settings which also risks intermittent problems due to IO load. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 40bcae01b0be2f293dea9ab42c6b7f8f47827cf5) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-04lib/oe/qa: handle the 'no specific instruction set' ELF e_machine valueRoss Burton
[RP: Update OEQA selftest to match change] Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b7cfc0f51cc0b4866f913f6eae4fcc6f72d2578c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-01-01oeqa/selftest/cases/devtool.py: fix typo in ignore_patterns callSteve Sakoman
Causes intermittent autobuilder errors: 2020-12-21 19:34:23,035 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/devtool.py", line 68, in setUpModule bb.utils.edit_bblayers_conf(bblayers_conf, None, None, bblayers_edit_cb) File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/bitbake/lib/bb/utils.py", line 1460, in edit_bblayers_conf (updated, newlines) = edit_metadata(newlines, ['BBLAYERS'], handle_bblayers) File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/bitbake/lib/bb/utils.py", line 1290, in edit_metadata if handle_var_end(): File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/bitbake/lib/bb/utils.py", line 1215, in handle_var_end (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines) File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/bitbake/lib/bb/utils.py", line 1425, in handle_bblayers res = edit_cb(layer, canonicalise_path(layer)) File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/devtool.py", line 60, in bblayers_edit_cb shutil.copytree(pth, destdir, ignore=ignore_patterns('*.pyc', '__pycache__')) NameError: name 'ignore_patterns' is not defined Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c2d9612279fce9cbcb738913b2042949f692c4a5) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-12-10meta/lib/oe/reproducible.py: gitsm:// works just as fine as git:// for ↵Alexander Kanavin
timestamps This in particular addresses vulkan-samples reproducibility which made me scratch my head for a while. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 4a2936126f12eeacecced051fa339c32c1f16576) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-12-10meta/lib/oeqa/manual/oe-core.json: Update test_bitbake_devshellsangeeta jain
Update command used for cross compilation to include ${CONFIGUREOPTS} to ensure right arguments are passed for cross compiling on any host. Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e995e67bc7a3d30adf9f8d0d2f4df92a941baeae) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-12-10oeqa/devtool: use Yocto mirror for pv-1.5.3 tarballRoss Burton
Some of the selftests failed over the weekend with "access denied" errors fetching this tarball. Instead of relying on upstream when fetching the tarball, use the Yocto source mirrors instead. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 988e0ff4131f46dfed14516ff5f61d72b9fb6941) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-14lib/oe/reproducible.py: Fix git HEAD checkJoshua Watt
The check for a git HEAD still wasn't quite correct because it was using the .git directory as the current working directory. Instead, it should be passed as the --git-dir argument when running git. Running `git rev-parse HEAD` in a .git directory with no HEAD reports 'HEAD' and exits with success but then 'git log' will fail, which is not what we want. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cdbd47dd7e1657b91b65a0940b7cbf119764240f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-14lib/oe/reproducible: Fix error when no git HEADJoshua Watt
Fixes an error that occurs when attempting to get the timestamp of the latest commit when there is no HEAD in the git repository. The easiest way to trigger this condition is to use the 'subdir=' option when specifying a 'git://' SRC_URI. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a64caca5b5dbe4a76acd0b5709b2c3e75b245863) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-14classes/reproducible: Move to library codeJoshua Watt
Moves most of the python code used for dealing with the source date epoch to library code. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a7ede90955bc0c8bec1cbb3cab498ef2583b2f4e) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-11oeqa/core/context: initialize _run_end_timeKonrad Weihmann
with _run_start_time as value. For partial results of interrupted runs, this info might be otherwise missing for at least one testcase Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1c5e8baf57fa2a33b9ef507b11d9ea9acaa77238) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-11oeqa/core/context: expose results as variableKonrad Weihmann
register an unittest handler for testresults and expose it as variable result. With this even partial results from an interrupted test suite run can be made available Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a97ae47525157871b6c098ffc352293e365a4335) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-02sstatesig: Log timestamps for hashequiv in reprodubile builds for do_packageRichard Purdie
Currently if a task generates the same output with different timestamps, hasequiv won't detect it but reproducibile builds will fail tests due to the different timestamps. Add do_package timestamps to the hash when reproducibile builds are enabled to avoid this. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 11e8200ccec765ff6a4263e06512e5751eca261a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-02Revert "lib/oe/patch: fix handling of patches with no header"Martin Jansa
* This reverts commit d9971f5dc8eb7de551fd6f5e058fd24770ef5d78. * With the missing Subject line fixed in GitApplyTree.prepareCommit() we should be able to revert, the fix which was trying to help it by parsing GitApplyTree.patch_line_prefix ("%% original patch:") also from Subject line, now GitApplyTree.patch_line_prefix should always end on separate line which is then skipped when copying the lines to resulting patch, see original commit message from Paul: lib/oe/patch: fix handling of patches with no header If a patch applied by a recipe has no header and we turn the recipe's source into a git tree (when PATCHTOOL = "git" or when using devtool extract / modify / upgrade), the commit message ends up consisting only of the original filename marker ("%% original patch: filename.patch"). When we come to do turn the commits back into a set of patches in extractPatches(), this first line ends up in the "Subject: " part of the file, but we were ignoring it because the line didn't start with the marker text. The end result was we weren't able to get the original patch name. Strip off any "Subject [PATCH x/y]" part before looking for the marker text to fix. This caused "devtool modify openssl" followed by "devtool update-recipe openssl" (without any changes in-between) to remove version-script.patch because that patch has no header and we weren't able to determine the original filename. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-02lib/oe/patch: GitApplyTree: save 1 echo in commit-msg hookMartin Jansa
* also remove the extra blank lines which is often added to patches when refreshed with devtool (GitApplyTree.patch_line_prefix lines are ignored when refreshing .patch files, but newly added blank lines aren't - the leading blank line wasneeded for patches with just the subject line (to prevent the GitApplyTree.patch_line_prefix line ending appended to the commit summary), but we can add it in prepareCommit instead Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-11-02lib/oe/patch: prevent applying patches without any subjectMartin Jansa
* this was discovered with $ devtool finish --force-patch-refresh where it was removing some patches and replacing them with patch in filename called "patch:" e.g. this .patch file: https://github.com/OSSystems/meta-browser/blob/311067d2d8a50cee5c836892606444f63f2bb3ab/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch confuses devtool which results to create new .patch file called "patch:" $ devtool finish --force-patch-refresh firefox meta-browser NOTE: Starting bitbake server... WARNING: Host distribution "ubuntu-20.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution. Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00 Loaded 2480 entries from dependency cache. Parsing recipes: 100% |#################################################################################################################################################################################################################################| Time: 0:00:00 Parsing of 1718 .bb files complete (1717 cached, 1 parsed). 2480 targets, 68 skipped, 0 masked, 0 errors. Summary: There was 1 WARNING message shown. INFO: Updating patch 0001-Bug-1554949-Fix-WebRTC-build-failure-with-newer-linu.patch ... INFO: Updating patch pre-generated-old-configure.patch INFO: Adding new patch patch: INFO: Updating recipe firefox_68.0esr.bb INFO: Removing file /OE/build/test-oe-build-time/poky/meta-browser/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch INFO: Cleaning sysroot for recipe firefox... INFO: Leaving source tree /OE/build/test-oe-build-time/poky/build/workspace/sources/firefox as-is; if you no longer need it then please delete it manually this looked like incorrect parsing of the git format-patch files exported from workspace/sources (the git format-patch version of fix-camera-permission-dialg-doesnot-close.patch starts like this: $ head 0008-original-patch-fix-camera-permission-dialg-doesnot-c.patch From 37dfa11961b48024bedcfb9336f49107c9535638 Mon Sep 17 00:00:00 2001 From: Takuro Ashie <ashie@clear-code.com> Date: Mon, 20 Aug 2018 10:16:20 +0900 Subject: [PATCH 08/34] %% original patch: fix-camera-permission-dialg-doesnot-close.patch so first I've modified GitApplyTree.extractPatches() to be able to parse the original patch name correctly even in this case where subject is wrapped, but then it still wasn't right, because we ended with correctly named .patch file, but all we could use for Subject line was the name of the original .patch file (instead of the Subject from metadata commit which introduced this .patch files as some other .patch files get when refreshed with devtool. In the end the issue happens even sooner in GitApplyTree.prepareCommit() where it correctly found the Subject from metadata commit, but then didn't apply it when there weren't any other outlines from patch headers. Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-10-22oeqa: Add sync call to command executionRichard Purdie
We previously put a sync call into devtool to try and combat the bitbake timeout issues on the autobuilder. It isn't enough as the timeouts occur mid test. They are also occurring on non-devtool tests. Add in sync calls around command execution instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ceca5ed121e2b54415a7ab3a217882e4ea86923a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-10-19bitbake-bblayers/create: Make the example recipe print its messageYoann Congal
The example recipe is setup to print a message using bb.plain() in the "do_build" task but this task is "noexec" so the message never prints. This might be confusing. This moves the message printing into another "do_display_banner" task and add it to the do_build "before" list. Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 252385bef9b226f32691b8513869ea3e41813b40) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-10-05oeqa/selftest/cases/devtool.py: avoid .pyc raceTim Orling
In certain conditions, most likely under heavy load on the AutoBuilder, the prebuilt .pyc files are attempting to be executed before they have been completely copied. Avoid this by not copying the .pyc files (nor the __pycache__ directory). The impact of python3-native recreating the .pyc files should hopefully be negligible. YOCTO#13421 YOCTO#13803 Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30wic: Add 512 Byte alignment to --offsetJoshua Watt
Allows the --offset argument to use the "s" or "S" suffix to specify that it is reporting the number of 512 byte sectors. This is required for some SoCs where the mask ROM looks for an item at a sector that isn't aligned to a 1KB boundary. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 938595d1dc4abaf5f7f3a7900add3f0492b805d0) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30wic: Fix --extra-space argument handlingJoshua Watt
467f84e12b ("wic: Add --offset argument for partitions") broke the --extra-space argument handling in wic. Fix the option and add a unit test for the argument. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 87722a92c18f94917c8f70afc8cd0763462a5c25) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30wic: Add --offset argument for partitionsJoshua Watt
Add support for an --offset argument when defining a partition. Many SoCs require that boot partitions be located at specific offsets. Prior to this argument, most WKS files were using the --align attribute to specify the location of these fixed partitions but this is not ideal because in the event that the partition couldn't be placed in the specified location, wic would move it to the next sector with that alignment, often preventing the device from booting. Unlike the --align argument, wic will fail if a partition cannot be placed at the exact offset specified with --offset. Changes in V2: * Fixed a small typo that prevented test_fixed_size_error from passing Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 467f84e12b96bc977d57575023517dd6f8ef7f29) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30oeqa/selftest/containerimage: Update to match assumptions in configurationRichard Purdie
Assumptions in the test mean it doesn't work with ssh-pregen-hostkeys. It also doesn't work with systemd. Update the configuration to make sure neither of these effect the test. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e2f277657f1db6a0f805ac07d318a3850535d695) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30oeqa/selftest/incompatible_lib: Fix append usageRichard Purdie
It's pure luck this has worked so far, add a missing space to the append. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 793e0575b1cebb953276b5f93ff31e48c19779c8) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-30oeqa/qemurunner: Increase serial timeoutRichard Purdie
Increase the serial login timeout from 60 to 120s. This seems like a long time, however for a qemumips image with systemd+PAM and openssh, (e.g. core-image-sato-sdk + DISTRO=poky-altcfg), the getty connects to systemd's pam module which waits on logind and 45s for all this to happen at the same time as things like ssh key generation happens is not unknown. Increase the timeout to match the longer times we know these things can take in the worst case scenarios since we're tired of intermittent issues related to the serial login affecting the autobuilder. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d8b4292db741de660f756dfb766210814d587b7a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-14oeqa/concurrencytest: Improve builddir path manipulationsRichard Purdie
Its possible some patterns may cause problems with the current path manipulations, make a small tweak to try and avoid potential pathname overlap issues. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 889005dc17d3e3b8eadee907ee2c05b8ff613285) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-14selftest/signing: Ensure build path relocation is safeRichard Purdie
Similarly to 04ee0e8b95cd8ed890374e0007f976684206b630, ensure only full build paths are replaced in the environment to avoid breaking buildtools. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fcd0a9683af1a9155eabbd9056e3b46d4a931b2e) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-02selftest/prservice: Improve test failure messageRichard Purdie
When failing, give more information about why exactly a failure is happening such as the PR values in question. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit bdd3abcc210e8f58b7b411da6bbd9c5314819908) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-09-02oeqa: sdk: Capture stderr outputJoshua Watt
Redirect stderr to stdout when running subcommands while doing the SDK tests. The tests will show stdout when CalledProcessError is raised, but any output to stderr was lost. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7cb4e9ab8c1596281060e94a216966060103956e) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-08-29oeqa/manual/bsp-hw.json : remove X_server_can_start_up_with_runlevel_5_boot testTeohJayShen
remove the test as graphical mode is already being validated Signed-off-by: TeohJayShen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ad5b5ca9ec1d9106911a610bb36fb98e3f17748d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-08-29oeqa/manual/bsp-hw.json : remove shutdown_system testTeohJayShen
remove the test as shutdown function is already being validated Signed-off-by: TeohJayShen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 58281082fbed57298247c019c3d8149e688de161) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-08-28oeqa: runtime_tests: Extra GPG debuggingJoshua Watt
This patch enables extra debugging for gpg to try and get more debugging information when [YOCTO #14003] occurs Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f24d7be1fb863006b5414fa8f9c9a37bb48d5a6f) Signed-off-by: Steve Sakoman <steve@sakoman.com>