aboutsummaryrefslogtreecommitdiff
path: root/scripts
Commit message (Collapse)AuthorAgeFilesLines
* python3: Restructure python3 packaging and replace it with autopackagingAlejandro Hernandez8 hours1-442/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | See previous commit (python2 version) for more info, since mostly everything applies here as well. Old manifest file had several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained on previous commit), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained on previous commit). This patch has the same features as the python2 version but it differs in the following ways: - Python3 handles precompiled bytecode files (*.pyc) differently. for this reason and since we are cross compiling, wildcards couldnt be avoided on python3 (See PEP #3147 [1]). Both the manifest and the manifest creation script handle this differently, the manifest for python3 has an extra field for cached files, which is how it lets the user install the cached files or not via : INCLUDE_PYCS = "1" on their local.conf. - Shared libraries nomenclature also changed on python3, so again, we use wildcards to deal with this issue ( See PEP #3149 [2]): - Fixes python3 manifest, python3-core should be base and everything should depend on it, hence several packages were deleted: python3-enum, re, gdbm, subprocess, signal, readline. - When building python3-native it adds as symlink to it called nativepython3, which is then isued by the create_manifest task. - Fixes [YOCTO #11513] while were at it. References: [1] https://www.python.org/dev/peps/pep-3147/ [2] https://www.python.org/dev/peps/pep-3149/ Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* python: Restructure python packaging and replace it with autopackagingAlejandro Hernandez8 hours1-426/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reason we have a manifest file for python is that our goal is to keep python-core as small as posible and add other python packages only when the user needs them, hence why we split upstream python into several packages. Although our manifest file has several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. This patch fixes those issues, while adding some additional features. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained below), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained below). - Fixes the manifest in the following ways: * python-core should be base and all packages should depend on it, fixes lang, string, codecs, etc. * Fixes packages with repeated files (e.g. bssdb and db, or netclient and mime, and many others). - Sitecustomize was fixed since encoding was deprecated. - The JSON manifest file invalidates bitbake's cache, so if it changes the python package will be rebuilt. - It creates a solution for users that want precompiled bytecode files (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not. - Fixes python-core dependencies, e.g. When python is run on an image, it TRIES to import everything it needs, but it doesnt necessarily fails when it doesnt find something, so even if we didnt know, we had errors like (trimmed on purpose): # trying /usr/lib/python2.7/_locale.so # trying /usr/lib/python2.7/lib-dynload/_locale.so # trying /usr/lib/python2.7/_sysconfigdata.so while it didnt complain about _locale it should have imported it, after creating a new manifest with the automated script we get: # trying /usr/lib/python2.7/lib-dynload/_locale.so dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2); import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so How to use (after a new release of python, or maybe before every OE release): - A new task called create_manifest was added to the python package, which may be invoked via: $ bitbake python -c create_manifest This task runs a script on native python on our HOST system, and since the python and python-native packages come from the same source, we can use it to know the dependencies of each module as if we were doing it on an image, this script is called create_manifest.py and in a very simplistic way it does the following: 1. Reads the JSON manifest file and creates a dictionary data structure with all of our python packages, their FILES, RDEPENDS and SUMMARY. 2. Loops through all of them and runs every module listed on them asynchronously, determining every dependency that they have. 3. These module dependencies are then handled, to be able to know which packages contain those files and which should RDEPEND on one another. 4. The data structure that comes out of this, is then used to create a new manifest file which is automatically copied onto the user's python directory replacing the old one. Create_manifest script features: - Handles modules which dont exist anymore (new release for example). - Handles modules that are builtin. - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev) - Deals with packages which include folders. - Deals with packages which include FILES with a wildcard. - The manifest can be constructed on a multilib environment as well. - This method works for both python modules and shared libraries used by python. How to add a new package: - If a user wants to add a new package all that has to be done is modify the python2-manifest.json file, and add the required file(s) to the FILES list, the script should handle all the rest. Real example: We want to add a web browser package, including the file webbrowser.py which at the moment is on python-misc. "webbrowser": { "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"], "rdepends": [], "summary": "Python Web Browser support"} Run bitbake python -c create_manifest and the resulting manifest should be completed after a few seconds, showing something like: "webbrowser": { "files": ["${libdir}/python2.7/webbrowser.py"], "rdepends": ["core","fcntl","io","pickle","shell","subprocess"], "summary": "Python Web Browser support"} Known errors/issues: - Some special packages are handled differently: core, misc, modules,dev, staticdev. All these should be handled manually, because they either include binaries, static libraries, include files, etc. (something that we cant import). Specifically static libraries are not not supported by this method and have to be handled by the user. - The change should be transparent to the user, other than the fact that now we CANT build python-foo (it was pretty dumb anyway, since what building python-foo actually did was building the whole python package anyway), but doing IMAGE_INSTALL_append = " python-foo" would create an image with the requested package with no issues. [YOCTO #11510] [YOCTO #11694] [YOCTO #11695] Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* scripts/oe-depends-dot: add it to handle dot filesRobert Yang2 days1-0/+121
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add it to handle recipe-depends.dot and task-depends.dot. E.g.: * Print why rpm is built $ oe-depends-dot -k rpm --why/-w recipe-depends.dot Because: core-image-sato libdnf libsolv dnf * Print bzip2-native's depends $ oe-depends-dot -k bzip2-native --depends/-d recipe-depends.dot Depends: automake-native gnu-config-native libtool-native quilt-native autoconf-native * Remove duplicated dependencies to reduce the size of the dot files. For example, A->B, B->C, A->C, then A->C can be removed. The dot files are too big, we nearly couldn't use 'dot -T' to generate pictcures for target recipes, remove the duplicated dependencies makes is it possible. $ bitbake core-image-sato -g $ oe-depends-dot -r recipe-depends.dot Saving reduced dot file to recipe-depends-reduced.dot $ du -sh recipe-depends*.dot 608K recipe-depends.dot 32K recipe-depends-reduced.dot It has been recuded from 608K to 32K, now we can generate a picture, otherwise, it is too big: $ dot -Tpng recipe-depends-reduced.dot -O It also can handle task-depends.dot. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* buildhistory-diff: add support for colourising the outputRoss Burton3 days1-2/+7
| | | | | | | | | | | | Colour value removes in red and additions in green, making it easier to scan the output for relevant changes. This adds a --colour option to specify whether colouring should be on, off, or detected. The default is detected, and depends on whether stdout is a TTY (same behaviour as git). Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: rename the suggested license change tag to 'License-Update:'Alexander Kanavin3 days1-2/+2
| | | | | Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic: if we can't get from ioctl, try from os.stat()Dogukan Ergun7 days1-1/+9
| | | | | | | | | | | Under some conditions, ioctl FIGETBSZ can't return real value. We can try to use fallback via os.stat() to get block size. Source of patch: https://github.com/intel/bmap-tools/commit/17365f4fe9089df7ee9800a2a0ced177ec4798a4 Signed-off-by: Dogukan Ergun <dogukan.ergun@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* scripts/wic: explicitly set BUILDDIR within eSDKChang Rebecca Swee Fun9 days1-0/+2
| | | | | | | | | | | | | | | | | | | | | | | When we run wic within eSDK: $ wic create mkefidisk -e core-image-minimal ERROR: BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?) In order to figure out variable values, one must have sourced the OE build environment setup script. However, when we are in within the eSDK environment which isn't initialised like the normal OE build environment, we can't use wic utility with eSDK. Reference: https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#wic-requirements While wic ought to be fixed to be able to run without bitbake & native tools [YOCTO #11281], but this is a workaround to set BUILDDIR in the environment so that bitbake environment is setup for wic to build its required native tools. Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/wic: fix error of import wic module in eSDK environmentChang Rebecca Swee Fun9 days1-1/+1
| | | | | | | | | | | | | | | | | | wic modules in scripts/lib/ are needed for wic to work, but path to the python module is not exported in eSDK environment and we were using an absolutized path of wic script within the sysroots. We now changed to use real script path instead, where the wic modules are located. This will also resolved the tracebacks found when running wic from within the eSDK environment. Traceback (most recent call last): File "/tmp/deploy/sdk/poky_sdk/sysroots/x86_64-pokysdk-linux/usr/bin/wic", line 58, in <module> from wic import WicError ImportError: No module named 'wic' Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/wic: append bitbake executable file path in eSDK environmentChang Rebecca Swee Fun9 days1-0/+12
| | | | | | | | | | | | wic needs a set of tools to be available from sysroots. wic will find bitbake executable within the environment, and wic was unable to locate bitbake executable within eSDK because it wasn't setup with the OE build environment script. Hence, we need to add bitbake file path into the environment PATH for wic to be able to discover it and import bb modules. Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/wic: use scriptpath module to find bitbake path and oe lib pathChang Rebecca Swee Fun9 days1-4/+3
| | | | | | | | Use the scriptpath module in order to standardize the adding of bitbake and meta/lib path to sys.path. Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqemu: Add workaround for APIC hang on pre 4.15 kernels on qemux86Richard Purdie2018-01-021-0/+5
| | | | | | | | | | On pre 4.15 host kernels, an APIC window emulation bug can cause qemu to hang. On 64 bit we can use the x2apic, for 32 bit, we just have to disable the other timer sources and rely on kvm-clock. [YOCTO #12301] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic: Fix a path to a psuedo state directory (PSEUDO_LOCALSTATEDIR).Adrian Fiergolski2018-01-021-1/+1
| | | | | | | | | | | In case of 'new_rootfs' the psuedo directory is not copied. Thus PSEUDO_LOCALSTATEDIR should still point to the dsa 'native_sysroot'/../pseudo. Otherwise PSEUDO_LOCALSTATEDIR points to a not existing director ('new_rootfs'/../pseudo) and UID and GUID attributes are not applied to files of the image. Signed-off-by: Adrian Fiergolski <adrian.fiergolski@cern.ch> Signed-off-by: Ross Burton <ross.burton@intel.com>
* wic: Introduce --fsuuid and have --use-uuid make use of UUID tooTom Rini2017-12-186-19/+47
| | | | | | | | | | | First, allow for wic to be given a filesystem UUID to be used when creating a filesystem. When not provided, wic will generate the UUID to be used. Next, when --use-uuid is passed, we update the fstab to mount things via UUID (and if not found, then use PARTUUID) as UUID is more portable. Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* wic: partition.py: Update comments slightlyTom Rini2017-12-181-3/+1
| | | | | | | | | First, we support squashfs as root, so mention that. Second, the btrfs rootfs creation function had a copy/paste of the previous function comment, remove the irrelevant line. Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* wic: kparser.py: Check for SquashFS and use-uuidTom Rini2017-12-181-0/+5
| | | | | | | | The SquashFS filesystem does not support UUIDs so make this combination be an error. Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: add license checksum change handling to 'devtool upgrade'Alexander Kanavin2017-12-181-2/+73
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Specifically, 'devtool upgrade' will now do these things: 1) determine if any of the license checksums need updating; if so, write the new checksums into the LIC_FILES_CHKSUM value in the recipe that is written to the workspace; 2) print a notice to the standard output: NOTE: New recipe is /home/ak/development/poky/build/workspace/recipes/puzzles/puzzles_git.bb NOTE: License checksums have been updated in the new recipe; please refer to it for the difference between the old and the new license texts. 3) and the cool part: devtool will create a diff of the old and new licenses, and write the diff into the workspace recipe as a comment, like this: ====== FIXME: the LIC_FILES_CHKSUM values have been updated by 'devtool upgrade'. The following is the difference between the old and the new license text. Please update the LICENSE value if needed, and summarize the changes in the commit message via 'License-checksum-change:' tag. (example: 'License-checksum-change: copyright years updated.') The changes: --- LICENCE +++ LICENCE @@ -1,8 +1,9 @@ This software is copyright (c) 2004-2014 Simon Tatham. Portions copyright Richard Boulton, James Harvey, Mike Pinna, Jonas -Klker, Dariusz Olszewski, Michael Schierl, Lambros Lambrou, Bernd -Schmidt, Steffen Bauer, Lennard Sprong and Rogier Goossens. +Kölker, Dariusz Olszewski, Michael Schierl, Lambros Lambrou, Bernd +Schmidt, Steffen Bauer, Lennard Sprong, Rogier Goossens and Michael +Quevillon. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files ====== Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
* devtool: provide useful defaults for version/commit when upgrading recipesAlexander Kanavin2017-12-181-4/+12
| | | | | | | Specifically, 'devtool upgrade' will use the latest upstream release if available or latest commit if upstream never makes releases. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
* devtool: add a 'latest-version' commandAlexander Kanavin2017-12-181-0/+26
| | | | | | | | | | | | | | | | | | | | | | This command queries the upstream server for what the latest release is and prints the output; it is a much neater way to find out these things than fumbling with distrodata, 'bitbake -c checkpkg' and awkward to read csv output in a file. Examples: python3 (tarballs): NOTE: Current version: 3.5.3 NOTE: Latest version: 3.6.3 rpm (git): NOTE: Current version: 4.13.90 NOTE: Latest version: 4.14.0 NOTE: Latest version's commit: da3720f62e57648fb1dc2a632744d38866139971 puzzles (git without version tags): NOTE: Latest commit: ee8ea9b9785964694cb2b3ad77c3fb2460f49510 Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
* recipetool: create: fix conflict between SRCREV and tagChang Rebecca Swee Fun2017-12-181-4/+4
| | | | | | | | | | | | | | | | | | | If you specify 'tag=' for a git URL and passed to recipetool create, you will get into Bitbake expansion error shown below: ----- snip ----- $ devtool add --version 2.4.2 mbedtls "git://github.com/ARMmbed/mbedtls;tag=mbedtls-2.4.2" ... bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Conflicting revisions (abeccb9dbd7e19ae91ac50e1edd3803111c5f9b6 from SRCREV and mbedtls-2.4.2 from the url) found, please specify one valid value ----- snip ----- Assuming the tag is valid, we should get the tag commit hash and drop the usage of 'tag=' from SRC_URI. By using a commit hash corresponding to the tag will prevent bitbake from accessing remote repository in order to expand SRCPV. Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* python-*-manifest/generators: add runpy; python3-plistlibTim Orling2017-12-132-0/+10
| | | | | | | | | * runpy allows running modules/scripts with 'python -m foo' * python3-setuptools RDEPENDS on plistlib (present in python2) * pip3 RDEPENDS on _markupbase (add to python3-core) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* python-*-manifest/generators: fix long line lengthsTim Orling2017-12-132-22/+26
| | | | | | | | | | | The generators create python-*-manifest.inc files with lines over 2500 characters long which breaks sending patches via git send-email (because of smtp limitation). This patchset formats all the long lines into multiple lines. Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* wic: support filesystem label for rawcopyMartin Hundebøll2017-12-091-0/+22
| | | | | | | | | The '--label' argument should work for '--source rawcopy' as it does for '--source rootfs', so add a method in RawCopyPlugin to update the label on the temporary filesystem images. Signed-off-by: Martin Hundebøll <mnhu@prevas.dk> Signed-off-by: Ross Burton <ross.burton@intel.com>
* runqemu: Allow the user to override the device tree optionAlistair Francis2017-12-021-1/+14
| | | | | | | | | | | | | Update the runqemu script to allow the user to specify a device tree to boot when calling runqemu. This involves creating a seperate check_dtb() function incase the user has specified 'none' for the kernel but still wants a device tree. Signed-off-by: Alistair Francis <alistair.francis@xilinx.com> Reviewed-by: Ben Levinsky <ben.levinsky@xilinx.com> Cc: Ben Levinsky <ben.levinsky@xilinx.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* runqemu: Allow the user to override the Kernel optionAlistair Francis2017-12-021-1/+5
| | | | | | | | | Update the runqemu script to allow the user to specify a Kernel to boot when calling runqemu. Signed-off-by: Alistair Francis <alistair.francis@xilinx.com> Cc: Ben Levinsky <ben.levinsky@xilinx.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* scripts/contrib/bbvars.py: Remove dead codeAmanda Brindle2017-12-021-50/+2
| | | | | | | | | Removed the -m option since this script now searches through all recipes in the configuration. Also removed dead code, which includes the functions recipe_bbvars() and collect_bbvars(). Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* runqemu: print command search result when not foundRobert Yang2017-11-291-0/+3
| | | | | | | This makes debug easier. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* verify-bashisms: use argparse, add verbose optionRoss Burton2017-11-291-4/+11
| | | | Signed-off-by: Ross Burton <ross.burton@intel.com>
* runqemu: Also specialcase resolution of '.' to the file's locationRichard Purdie2017-11-211-0/+2
| | | | | | | Similarly to handling "../", handle "." to resovle to the qemuconf file's current directory. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqemu: Improve relative path handling in qemuconf filesRichard Purdie2017-11-211-0/+2
| | | | | | | | | If a variable starts with "../", its likely its a path and we want to set it to an absolute path relative to the qemuconf file. This means we don't have to use bitbake as often to figure out variables. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqemu: Ensure we process all tap devicesRichard Purdie2017-11-211-1/+1
| | | | | | | | | | | | | | | | | | The regexp in the script misses some tap devices, e.g. we see output like: runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap25.lock failed: [Errno 11] Resource temporarily unavailable runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap26.lock failed: [Errno 11] Resource temporarily unavailable runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap27.lock failed: [Errno 11] Resource temporarily unavailable runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap28.lock failed: [Errno 11] Resource temporarily unavailable runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap40.lock failed: [Errno 11] Resource temporarily unavailable runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap41.lock failed: [Errno 11] Resource temporarily unavailable What happened to tap29 to tap39? The issue is was we were missing devices with '0' in the number, like "10:" and so on in the output from "ip link". Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* recipetool: create: fix failure handling included dictsPaul Eggleton2017-11-211-0/+2
| | | | | | | | | | | | | | If a setup dict in a python setup.py file pulled in the contents of another dict (e.g. **otherdict), then we got an error when mapping the keys because the key is None in that case. Skip those keys to avoid the error (we pick up the values directly in any case). A quick reproducer for this issue: recipetool create https://files.pythonhosted.org/packages/source/p/pyqtgraph/pyqtgraph-0.10.0.tar.gz Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: implement conditional patch handlingPaul Eggleton2017-11-102-26/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | If you have a recipe that uses overrides to conditionally extend SRC_URI to add additional patches, then you will often need to update those patches if you're making other changes to the source tree (for example if you're upgrading the underlying source). Make this possible with devtool by creating devtool-override-* branches for each override that conditionally appends/prepends SRC_URI, and have devtool update-recipe / finish check each branch out in turn and update the corresponding patches. A current example of a recipe that does this is the quota recipe - it applies an additional patch if musl is the selected C library (i.e. libc-musl is in OVERRIDES). Note that use of this functionality does require some care - in particular, updates to patches that appear on the main branch (named "devtool" by default) should be made there and not only on one of the specific devtool-override-* branches that are created for each override. The recommended procedure is to make the changes you want to make to the main branch first, then check out and rebase each devtool-override-* branch, testing each one by activating the corresponding configuration, and then finally run devtool finish. Fixes [YOCTO #11516]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: finish: add dry-run optionPaul Eggleton2017-11-101-71/+199
| | | | | | | | | | | | If you're not sure what changes devtool finish is going to make, or you're not sure you're finished with your modifications, it is useful to be able to see what devtool finish is going to do beforehand, so add a -N/--dry-run option to make that possible. (It's also very useful for debugging devtool finish itself.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: finish: improve reporting for removed filesPaul Eggleton2017-11-101-14/+18
| | | | | | | | | | | | | | | | | | | | | | If a file is going to be effectively removed from the destination by devtool finish, we should report that rather than just reporting that we're removing files from the workspace. This is a little tricky because the way we actually operate when finishing is to: (1) remove all original files (as recorded by devtool upgrade, if that was used) (2) as part of updating the recipe file, remove the files from next to the new recipe (i.e. in the workspace for an upgrade, real recipe otherwise) corresponding to commits not in the git tree (3) copy over remaining files from the workspace to the destination To report the files removed with respect to what was originally there, we need to swap steps 1 and 2 so we can see what no longer exists after the deletion, and suppress the reporting currently done in step 2 - however, we still want to report removal in step 2 for the non-upgrade case, so the latter is conditional. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: show a better error message if meta-files aren't foundPaul Eggleton2017-11-101-4/+7
| | | | | | | | | If the files that the devtool-source class is supposed to create in the source tree aren't found in the temporary directory then we know that the class hasn't worked properly - say that explicitly. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: reformat --no-patch warning messagePaul Eggleton2017-11-101-4/+2
| | | | | | | | | * Only log one warning message instead of one per line * Be a bit more verbose * "if list" is more pythonic than "if len(list)" Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: automatically handle changes to source subdirectoryPaul Eggleton2017-11-102-9/+28
| | | | | | | | | | | | | | If the directory where the source code extracts to changes (for example, when upgrading iucode-tool from 1.5 to 2.1.1, the subdirectory in the tarball changed from "iucode_tool-${PV}" to "iucode-tool-${PV}") then handle this automatically. Also handle when it changes to match the default S value (i.e. "${WORKDIR}/${BP}") in which case we just drop setting S in the recipe. Fixes [YOCTO #10939]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: show messages before source extraction stepsPaul Eggleton2017-11-101-0/+2
| | | | | | | Give the user a little more insight into what's being done. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* recipetool: create: show a warning for github archive URLsPaul Eggleton2017-11-101-0/+3
| | | | | | | | | | | github archive URLs are not guaranteed to be stable [1] and thus we should show a warning if a user specifies one to recipetool create (or devtool add). [1] http://lists.openembedded.org/pipermail/openembedded-core/2017-September/142519.html Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: stop always moving workspace to end of BBLAYERSPaul Eggleton2017-11-101-1/+5
| | | | | | | | | | | | | | | | | I noticed that using bitbake-layers add-layer followed by a devtool command resulted in bitbake re-parsing all of the recipes, which is annoying. Upon closer inspection I could see that devtool was moving the workspace layer path to the end of BBLAYERS if it happened to be somewhere in the middle - there's no need for it to be doing this. This occurred because we were passing the current workspace path to remove and the "new" path to add even if the path is not being changed, and I think earlier versions of bb.utils.edit_bblayers_conf() didn't move the existing entry under these circumstances as it clearly does now. Fix it so we only pass the path to be removed if we're actually changing the path. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* recipetool: create: drop debug printPaul Eggleton2017-11-101-1/+0
| | | | | | | This looks like some debug printing that was left in by accident. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: handle recipes that use named SRC_URI checksumsPaul Eggleton2017-11-101-13/+40
| | | | | | | | | | | | | | | | | | devtool upgrade did not properly handle setting SRC_URI checksums for recipes that use named SRC_URI entries and also use those names in the SRC_URI checksums. A further complication was where the name contained an expression that changed with the version e.g. ${PV} (probably quite rare, but the dnsmasq recipe in meta-networking is currently one such recipe.) All of these are now handled properly. Additionally, drop the _get_checksums() function that wasn't being called from anywhere in the code. Note that this now turns nowrap_vars in recipeutils.py to be a list of regexes, hence things such as [ and ] need to be appropriately escaped. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: finish: fix "layer not in bblayers.conf" warning when path specifiedPaul Eggleton2017-11-101-2/+4
| | | | | | | | | | | | | | devtool finish will check if the destination layer is part of bblayers.conf so that we avoid the user getting confused about the recipe vanishing from their configuration if it isn't. devtool finish also accepts a path underneath a layer so that you have a bit more control over where it ends up. However if you used a path underneath a layer then it wasn't converting this to the base of the layer before checking it against BBLAYERS, thus the warning was being shown erroneously in that case. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: finish: ensure repository is clean before proceedingPaul Eggleton2017-11-102-2/+32
| | | | | | | | | | | | | | | If the git repository for a recipe in the workspace has uncommitted changes in it then it's possible that the user has forgotten to commit something, so check and exit if there are any. Provide a -f/--force option to continue in the case where the uncommitted changes aren't needed. Separately, if the repository is in the middle of a rebase or git am / apply then error out (without the opportunity to force) since the user really needs to sort this out before finishing. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: reset: print source tree base pathPaul Eggleton2017-11-101-5/+5
| | | | | | | | | | If S points to a subdirectory of the source rather than the "base" of the source tree then print that rather than the subdirectory path when telling the user they need to remove the source tree, since that is the directory that they will need to remove. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: make find-recipe and edit-recipe always work with any recipePaul Eggleton2017-11-101-12/+14
| | | | | | | | | | | After some reconsideration I think it's a bit annoying for users to be forced to use an option to work with recipes where the file isn't in the workspace, so let's just have these commands check the workspace first for the recipe, and if it isn't there then load the cache and get it that way. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: show some warnings for upgrade versionsPaul Eggleton2017-11-103-3/+23
| | | | | | | | | | | | | * Show a warning in devtool upgrade if the version is less than the current version suggesting that the user may need to bump PE in the recipe * Show a warning in devtool add and devtool upgrade if the version looks like a pre-release version suggesting using a version number that won't mess up the progression when you come to upgrade to the final release version. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: fix handling of oe-local-files when source is in a subdirectoryPaul Eggleton2017-11-102-13/+26
| | | | | | | | | | | | | | | If S points to a subdirectory of the source rather than the "base" of the source tree then we weren't handling the oe-local-files directory properly - it got extracted to the base of the tree but devtool update-recipe and devtool finish assumed it would be under S which would be the subdirectory, thus it would be missing and devtool would assume the files had been deleted and remove them from the recipe. Record the base of the source tree in the bbappend and read it into the in-memory workspace so we can use that to find out where oe-local-files should be found. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: improve performance and show progress when adding filesPaul Eggleton2017-11-101-2/+9
| | | | | | | | | | | | | | | | | | | | When devtool upgrade is upgrading to a new version where the source is fetched as an archive (e.g. a tarball), we create a single commit in the git repository that is the upgrade from the old version to the new. We do this by extracting the old source, committing it, deleting all files, copying in the new files, running git add on each new/changed/deleted file, and then committing the result. When a lot of files have changed in an upgrade (such as QEMU 2.8.1.1 -> 2.10.0) the penultimate step of running git add it can take quite a long time; in order to reduce this and show some feedback to the user, run git add with batches of 100 files at once and also show a progress bar. In a local test with the aforementioned QEMU upgrade it took the time down from over 7 minutes down to about 13 seconds. Fixes [YOCTO #11948]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* devtool: upgrade: fix not committing deleted files with older git versionsPaul Eggleton2017-11-101-1/+1
| | | | | | | | | | | With versions of git older than 2.0, "git add" on a deleted file (i.e. in this case a file that was removed between versions) will not add the delete to be committed by default, with the result that the rebase of patches on top of the new branch will fail. We need to use the -A option in order to force that for older git versions. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>