aboutsummaryrefslogtreecommitdiffstats
path: root/scripts
AgeCommit message (Collapse)Author
2017-07-17recipetool: git reformat URI mangling & parameter strippedStanley Cheong Kwan, Phoong
recipetool seems to be mangling and stripping out the parameters for git URI. This will fix this issue as well as resolve the conflict of protocol parameter added by user. If a user adds their own protocol as an argument, it'll be honored. [YOCTO #11390] [YOCTO #11391] Signed-off-by: Stanley Cheong Kwan, Phoong <stanley.cheong.kwan.phoong@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17wic: isoimage-isohybrid: use grub-efi from deploy dirEd Bartosh
isoimage-isohybrid plugin tries to build grub-efi in its working directory if it can't find efi binary. Wic should avoid doing anything in working directories of other recipes. It should use artifacts from the image deployment directory instead. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17wic: isoimage-isohybrid: check result of glob()Ed Bartosh
isoimage-isohybrid plugin uses result of glob call to get path to initrd image. When glob returns empty list the plugin crashes with IndexError. Checking if result of glob call is not empty should fix the breakage. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17wic: build wic-tools only if wic is run manuallyEd Bartosh
When wic can't find native sysroot it tries to build wic-tools However, it's not possible when wic is run from bitbake. Moreover, it's not even feasible anymore as wic-tools should be used only when wic is run manually. Checked if wic is run manually before building wic-tools. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17wic: isoimage: do not remove temp directoryEd Bartosh
Removed isodir subrdirectory instead of removing temporary working directory as working directory can contain copy of rootfs partition and shouldn't be removed by any plugin. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17wic: get rid of using wic-toolsEd Bartosh
At the moment, when building images with IMAGE_FSTYPES=wic one ends up depending on wic-tools and thus syslinux and grub-efi even when not using those at all. Ideally, building an image with wic should only build the tools and components really needed. The problem is that "wic-tools" is needed also for the manual invocations of wic, in which case everything that might be needed has to be built in advance. Replaced dependency on wic-tools with dependency to a much shorter set of tools that wic uses almost for any image: 'parted', 'gptfdisk', 'dosfstools' and 'mtools'. [YOCTO #11552] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17runqemu: check exit code of 'ls -t'Ed Bartosh
Used check_output instead of Popen as it raises CalledProcessError exception when command exits with non-zero exit code. Catched the exception to produce user-friendly output. [YOCTO #11719] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-17runqemu: reworked exception handlingEd Bartosh
Introduced custom RunQemuException that script raises on known errors. This exception is handled in one place and prints error output without printing Python traceback. This shoud make error output less scary for the end user. Handling of unknown errors has not been changed - both error and traceback will be printed. Reimplemented OEPathError exception code to handle it similarly to RunQemuException. Moved exception handling code into main() to keep it in one place. [YOCTO #11719] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06oe-pkgdata-util: package-info: Allow extra variables to be displayedPeter Kjellerstedt
By specifying the -e <var> option one or multiple times, extra variables available in the pkgdata can be displayed, e.,g, `oe-pkgdata-util package-info -e SUMMARY -e LICENSE ...`. The extra variables displayed are quoted since some of them may contain whitespace. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: make signature check code reusablePatrick Ohly
This moves the main content of test_signature into a helper function. It can be reused by arbitrary tests that need to do a before/after signature comparison. Long-term this might even be useful in oeqa itself. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: allow README with suffixPatrick Ohly
It may be useful to append a suffix denoting the file format. For example, README.rst is rendered differently when viewed on Github, and also helps editors to switch to a mode more suitable for the format. The tests uses a file pattern to find the README file(s) and treats the one with the shortest name as the main one which must not be empty. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: add test_worldPatrick Ohly
"test_signatures" ignores wold build breakage for the sake of reporting differences also when a world build is broken. Therefore we need a dedicated test that a world build at least theoretically can proceed without obvious parse time problems (dependencies, parse errors, dangling .bbappends, etc.). This is similar to the BSP test_machine_world. The difference is that test_world doesn't change the MACHINE. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: apply test_signatures to all layersPatrick Ohly
Software layers were previously allowed to change signatures, but that's not desired for those layers either. The rule that a layer which is "Yocto Compatible 2.0" must not change signatures unless explicitly requested holds for all kinds of layers. However, as this is something that software layers might not be able to do right away, testing for signature changes in software layers can be disabled. It's on by default, as that was Richard's recommendation. Whether that should change needs further discussion as part of finalizing "Yocto Compatible 2.0". As it might still change, the tool now has both a with/without parameter so that users of the tool can choose the desired behavior without being affected by future changes to the default. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: tolerate broken world builds during signature diffPatrick Ohly
The "test_signatures" test ignored a broken world build when getting signatures, but the code which then tried to analyze a difference found by the test didn't, which prevented printing the difference. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-07-06yocto-compat-layer.py: avoid adding layers more than oncePatrick Ohly
add_layer_dependencies() might get called more than once, or one of the layer dependencies might already be present. The function should not add layers again because doing so can cause warnings like: WARNING: Duplicate inclusion for .../meta-openembedded/meta-oe/conf/distro/include/meta_oe_security_flags.inc in .../meta-openembedded/meta-oe/conf/layer.conf Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-30mkefidsk: fix bash/dash shell quoting problemSaul Wold
mkefidsk currently writes a startup.nsh with embedded control characters. This happens because \b etc are control sequences to the shell echo command when using dash. The resulting startup.nsh causes the bootup to fail, and the user is dropped into the EFI shell to manually run startup.nsh. Patch originally provided by Troy D. Hanson <troy.hanson@jhuapl.edu> [YOCTO #9665] Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-29scripts/contrib/patchreview: add new scriptRoss Burton
This script analyses the patches we apply and can sanity check or output statistics. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28scripts/test-reexec: fix several incomplete regexpMing Liu
Some "=" are missing in the regexp, which leads it match multiple results if the variables have overrides, for instance: ... SSTATE_DIR="xxxx" SSTATE_DIR_qemux86="yyyy" ... it will match both of them without "=". Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-28scripts/oe-find-native-sysroot: fix a incomplete regexpMing Liu
A "=" is missing in the regexp, which leads it match multiple results if STAGING_DIR_NATIVE has overrides, for instance: ... STAGING_DIR_NATIVE="xxxx" STAGING_DIR_NATIVE_qemux86="yyyy" ... it will match both of them without "=". Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-27runqemu: drop RPC portsRobert Yang
The following commit has removed rpc ports from runqemu-export-rootfs, so runqemu should also remove them, otherwise "runqemu nfs" doesn't work. And use abspath for nfsroot, otherwise it doesn't work when it is a relative path. commit 6bb9860ef7ba9c84fe9bd3a81aa6555f67ebd38e Author: Cody P Schafer <dev@codyps.com> Date: Tue Jun 6 18:30:49 2017 -0400 runqemu-export-rootfs: don't change RPC ports [YOCTO #11687] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23sstate-sysroot-cruft.sh: Extend the whitelistMartin Jansa
* add more php5 entries Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23wic: remove extra double-quote on documentation stringJean-Francois Dagenais
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23wic/runner.py: move runtool API to misc.pyEd Bartosh
Moved remaining API to misc.py. Removed runner.py. Now misc.py is ready to be moved to the scripts/lib/wic and utils directory can be removed. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23wic: fix restoring of original fstabEd Bartosh
Wic updates fstab if mount points are specified in .wks. After partition images are created the original fstab is restored. However, if exception is raised when partition image being prepared wic doesn't restore original fstab. This can cause duplication of added lines in fstab when 'wic create' runs next time. Wrapping call of 'prepare' method and restoring original fstab even if exception occurs should fix this. [YOCTO #11633] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23wic: code cleanupEd Bartosh
Split long lines. Removed unused imports. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23wic: flatten directory structureEd Bartosh
Moved misc.py from wic/utils/ to wic/ Removed wic/utils directory Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23scripts/contrib/patchtest: run patchtest on local branchLeonardo Sandoval
The script run patchtest on local branch commits, printing results into stdout. This script is useful to test patches before sending to the mailing list. Examples: $ git checkout master-next-1.9 Branch master-next-1.9 set up to track remote branch master-next-1.9 from origin. Switched to a new branch 'master-next-1.9' $ ~/scripts/contrib/patchtest.sh 166e70e: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: patch: fix CVE-2015-1196: FAIL Issue Missing or incorrectly formatted CVE tag in commit message [test_cve_presence_in_commit_message] Suggested fix Include a "CVE-xxxx-xxxx" tag in the commit message Issue Missing or incorrectly formatted CVE tag in included patch file [test_cve_tag_format] Suggested fix Correct or include the CVE tag on cve patch with format: "CVE: CVE-YYYY-XXXX" eaa4536: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: wget: 1.16.1 -> 1.16.2: OK 3c29ce3: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: git: 2.3.0 -> 2.3.1: OK 85491f6: Khem Raj: Thu Apr 2 12:01:38 2015 +0100: gdb: Upgrade 7.8.1 -> 7.9: OK f701142: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: binutils: upgrade to 2.25: OK 385d0b1: Khem Raj: Thu Apr 2 12:01:39 2015 +0100: binutils: Fix ICE in gold: OK [YOCTO #10720] Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-23buildhistory-diff: exclude paths from the outputEd Bartosh
Implemented -e/--exclude-path command line option to exclude paths from buildhistory-diff output. [YOCTO #11459] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-06-14python-3.5: Move bz2.py, lzma.py and _compression.py from python3-misc to ↵Martin Jansa
python3-compression * the /usr/lib/python3.5/_compression.py file is possibly incorrectly included in python3-misc. This runtime dependency is needed in order to use e.g. gzip.py in runtime: >>> import tarfile, zlib, gzip Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.5/gzip.py", line 12, in <module> import _compression ImportError: No module named '_compression' * at least python3-tests and lzma and bz2 still in python3-misc are using this as well: $ grep -R import.*_compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/ tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-compression/usr/lib/python3.5/gzip.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/lzma.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/bz2.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_bz2.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_lzma.py:import _compression and python3-tests are using it as well, so add new runtime dependency on python3-compression Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14runqemu: change terminal settings for valid tty'sMikko Ylinen
runqemu uses stty to change terminal settings to give users better control to qemu. However, stty does not work when runqemu is run directly or indirectly via oe-selftest in a Docker container (presumably some problems with Docker's pseudo-tty implementation). The error reported is: stty: 'standard input': Inappropriate ioctl for device As runqemu recently moved to subprocess.check_call() for stty calls we now get thrown an error and all runqemu runs fail. sys.stdin.isatty() does proper job in detecting if the stty calls can work so we use that check before running the stty subprocess operations. Signed-off-by: Mikko Ylinen <mikko.ylinen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: implement removing directoriesEd Bartosh
Added support for removing directories using mdeltree utility to Disk.del method [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: implement removing filesEd Bartosh
Added implementation of Disk.del method and wic_r function that removes files from the vfat partition using mdel utility. [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add 'wic rm' commandEd Bartosh
Added empty 'wic rm' command that does nothing. The functionality will be added by the next commits. [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add help and usage content for 'wic rm'Ed Bartosh
Added wic_rm_help and wic_rm_usage variables to help.py. These variables contain help content that will be used in 'wic rm help' and 'wic rm --help' output. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add wic_init_parser_rmEd Bartosh
Add parser for 'wic rm' subcommand. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: fully implement 'wic cp'Ed Bartosh
Added implementation of Disk.copy method and wic_cp function that copies files/directories to the vfat partition of the partitioned image. [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add Disk._put_part_image methodEd Bartosh
This method copies partition image into the wic image. It will be used in 'wic cp' and 'wic rm' subcommands to copy changed partition back into wic image. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14filemap: calculate dst size correctlyEd Bartosh
Fixed calculation of the dst file size using skip, seek and length parameters. Current code does it incorrectly which causes sparse_copy API to create unnecessary big output files. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14filemap: check if dest is written for every blockEd Bartosh
If lenght parameter is provided to sparse_copy call it's mandatory to check if the output file is fully written after reading unmapped block from input file. If it's not done then sparse_copy can write more data than specified length. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14filemap: change signature of sparse_copy functionEd Bartosh
Renamed parameter offset->skip to match names of dd parameters. Changed affected sparse_copy calls. Added explanation of the parameters to docstring. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add mcopy propertyEd Bartosh
Added property that points to the mcopy executable. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add Disk._prop helperEd Bartosh
Added generic helper to use in property methods to access commands in a lazy manner. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add 'wic cp' commandEd Bartosh
Added empty 'wic cp' command that does nothing. The functionality will be added by the next commits. [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add help and usage content for 'wic cp'Ed Bartosh
Added wic_cp_help and wic_cp_usage variables to help.py. These variables contain help content that will be used in 'wic cp help' and 'wic cp --help' output. [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add wic_init_parser_cpEd Bartosh
Add parser for 'wic cp' subcommand and a custom argument type. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14engine: implement listing wic imagesEd Bartosh
Implemented 'wic ls' functionality: - list image partitions - list directory content of vfat partitions [YOCTO #11283] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add 'wic ls' commandEd Bartosh
Added empty 'wic ls' command that does nothing. The functionality will be added by the next commits. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add help and usage content for 'wic ls'Ed Bartosh
Added wic_ls_help and wic_ls_usage variables to help.py. These variables contain help content that will be used in 'wic ls help' and 'wic ls --help' output. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14wic: add wic_init_parser_lsEd Bartosh
Added parser for 'wic ls' command. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14bootimg-pcbios: make boot image file uniqueEd Bartosh
Plugin code uses boot.img file name for an image file. If there are two partitions that use bootimg-pcbios wic breaks with an error "file already exists: boot.img" Made image file name unique by adding wks like number to it to fix the issue. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>