summaryrefslogtreecommitdiffstats
path: root/lib
AgeCommit message (Collapse)Author
2020-03-06bblayers: query: Add multiconfig optionJoshua Watt
Adds an option to the show-recipes subcommand that allows the user to specify which multiconfig should be shown. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06tinfoil: Add multiconfig supportJoshua Watt
Adds support for the Tinfoil cache adaptor to be bound to a specific multiconfig and invoke the appropriate commands for that multiconfig instead of the default. The cooker adapter now creates a cache adapter for each multiconfig specified in BBMULTICONFIG so that each multiconfig is present. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06cooker: Respect multiconfig parameterJoshua Watt
The cooker had a multiconfig parameter for the findProviders() and findBestProviders() API, but it was being ignored. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06command: Add mc parameter to findProviders commandJoshua Watt
Adds a multiconfig selection parameter to the findProviders command. This allows a client to find the providers for a specific multiconfig instead of the base configuration. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-02cooker: Reset parse status unpon clientCompleteRichard Purdie
If for example a tinfoil connection edits the datastore, a subsequent connection can be "corrupted" by those changes. By setting the parse status of the caches as False at exit, the behaviour becomes the same as a newly setup server as a new data store is setup. This avoids problems in tests when BB_SERVER_TIMEOUT is set as the server is properly reset between connections. [YOCTO #13812] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-02fetch2: Limit shown checksums to sha256Richard Purdie
Currently bitbake will list many checksums for a recipe when none are present, encouraging users to add them all to a recipe. We don't need/want them all. We used to show md5 and sha256 but given the concerns about md5, switch to showing just sha256 going forward which seems like the sensible one to standardise upon. There will be no change to existing recipe functionality. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-22lib/bb/utils.py: Preserve ownership of symlinkDaisuke Yamane
Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-20fetch2/svn: Avoid UnboundLocalError exceptionRichard Purdie
The update codepath would trigger: Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment Fix this so the code functions as intended in both fetch and update cases. [YOCTO #13798] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19layerindex: allow clones to be shallowJan-Simon Moeller
When bitbake-layers fetch-layerindex clones the repositories, these are full clones. Allow the user to specify '-s' and do shallow clones instead for faster downloads. Signed-off-by: Jan-Simon Moeller <dl9pf@gmx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: svn: care for path_specJens Rehsack
Documentation says: "path_spec": A specific directory in which to checkout the specified svn module. but existing svn fetcher uses "module" always as path of checked out svn-module, regardless whether path_spec was given or not. Signed-off-by: Jens Rehsack <sno@NetBSD.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19event: Remove duplicated items from close matchesRobert Yang
It printed duplicated ones when there are multiple similar recipes in differrent layers, for example, if python-lockfile in different layers, and there is no python3-lockfile: $ bitbake python3-lockfile ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches: python-lockfile python-lockfile python3-aiofiles Remove the duplicated ones to fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19cooker: Reset loghandlerRichard Purdie
When parsing, reset the loghandler when finished, else the messages can be misleading. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19data: Don't allow renameVar calls with equivalent keysKyle Russell
While usually a programming error, the behavior can cause a Parser instance to eventually gobble up a significant amount of memory, greatly affecting system performance. Try to avoid getting into that situation and alert the user about what they attempted to do. Signed-off-by: Kyle Russell <bkylerussell@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache"Peter Kjellerstedt
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if BB_DONT_CACHE is set for the recipe. Remove any such keys upon the next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will make sure the srcrev is updated as expected if ${AUTOREV} is used. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: Make fetcher_compare_revisions() workPeter Kjellerstedt
This seems to have been broken for a very long time. Now it also works regardless of BB_SRCREV_POLICY. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19knotty: Make the bb.command.CommandExit event terminate bitbakePeter Kjellerstedt
This matches the other bb.command.Command* events and without it, running `bitbake --revisions-changed` will hang indefinitely if there are changed revisions. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-17cooker/siggen: Empty siggen cache during parsingRichard Purdie
When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15bitbake: layerindex: use branch when specifiedJon Mason
When currently specified, the branch is used to verify the versioning of the meta layer, but the master branch is checked out. This change allows for the branch to be specified. Now it is easy to specify all of the meta layers being added are of the same version, without having to do it in each individual git tree. Also, it will error if there are branches without a matching version. Finally, this allows for meta layer git trees without a master branch. Signed-off-by: Jon Mason <jdmason@kudzu.us> Minor rework of the patch to use the layerBranch actual_branch since the layerindex referenced branch may be different then the overall release branch. Also adjust the patch to use the default git checkout branch instead of master if no branch was specified. (Some repositories don't have a master branch.) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Avoid cache mismatch issues with locked sigsRichard Purdie
If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Cache unihash values to avoid cache lookupRichard Purdie
Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Optimise get_unihash disk based cache handlingRichard Purdie
Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-06cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews
Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27tests/fetch: add npmsw testsJean-Marie LEMETAYER
This commit adds some tests to validate the npmsw fetcher: - bb.tests.fetch.NPMTest.test_npmsw - bb.tests.fetch.NPMTest.test_npmsw_bad_checksum - bb.tests.fetch.NPMTest.test_npmsw_destsuffix - bb.tests.fetch.NPMTest.test_npmsw_dev - bb.tests.fetch.NPMTest.test_npmsw_mirrors - bb.tests.fetch.NPMTest.test_npmsw_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npmsw_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npmsw_npm_reusability - bb.tests.fetch.NPMTest.test_npmsw_premirrors Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: add the npmsw fetcherJean-Marie LEMETAYER
This commit adds a new npmsw fetcher that fetches every npm dependencies described in a npm shrinkwrap file: https://docs.npmjs.com/files/shrinkwrap.json.html The main package must be fetched separately: SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \ npmsw://${THISDIR}/npm-shrinkwrap.json" Since a separation has been created between the package and its dependencies, the package can also be fetched with a non npm fetcher without impacting the general behavior: SRC_URI = "git://github.com/foo/bar.git;protocol=https \ npmsw://${THISDIR}/npm-shrinkwrap.json" Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27tests/fetch: add npm testsJean-Marie LEMETAYER
This commit adds some tests to validate the npm fetcher: - bb.tests.fetch.NPMTest.test_npm - bb.tests.fetch.NPMTest.test_npm_bad_checksum - bb.tests.fetch.NPMTest.test_npm_destsuffix_downloadfilename - bb.tests.fetch.NPMTest.test_npm_mirrors - bb.tests.fetch.NPMTest.test_npm_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npm_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npm_package_invalid - bb.tests.fetch.NPMTest.test_npm_package_none - bb.tests.fetch.NPMTest.test_npm_premirrors - bb.tests.fetch.NPMTest.test_npm_registry_alternate - bb.tests.fetch.NPMTest.test_npm_registry_invalid - bb.tests.fetch.NPMTest.test_npm_registry_none - bb.tests.fetch.NPMTest.test_npm_version_invalid - bb.tests.fetch.NPMTest.test_npm_version_latest - bb.tests.fetch.NPMTest.test_npm_version_none Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2/npm: refactor the npm fetcherJean-Marie LEMETAYER
This commit refactors the npm fetcher to improve some points and fix others: - The big change is that the fetcher is only fetching the package source and no more the dependencies. Thus the npm fetcher act as the other fetchers e.g git, wget. The dependencies will be handled later. - The fetcher only resolves the url of the package using 'npm view' and then forwards it to a proxy fetcher. - This commit also fixes a lot of issues with the package names (exotic characters, scoped packages) which were badly handled. - The validation files - lockdown.json and npm-shrinkwrap.json - are no longer used by the fetcher. Instead, the downloaded tarball is verified with the 'integrity' and 'shasum' provided in the 'npm view' of the package [1][2]. 1: https://docs.npmjs.com/files/package-lock.json#integrity 2: https://www.w3.org/TR/SRI Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2/wget: fix downloadfilename parameterJean-Marie LEMETAYER
When using a download filename with characters which can be interpreted by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the filename fixes the issue. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the done conditionJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the done condition to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the mirrors managementJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "try_mirrors" functions to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the donestamp managementJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "verify_donestamp" and "update_stamp" functions to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: add more hash functions for checksum verificationJean-Marie LEMETAYER
This commit enables the "sha1", "sha384" and "sha512" hash functions in the supported checksum list. This allows to use more SRC_URI checksums functions for a url: SRC_URI[sha1sum] = "..." SRC_URI[sha384sum] = "..." SRC_URI[sha512sum] = "..." The npm fetcher needs this to support subresource integrity: https://www.w3.org/TR/SRI/ Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: refactor checksum verificationJean-Marie LEMETAYER
This commit refactors the way checksums are verified to be more generic. The support of new hash functions is now limited to the update of the CHECKSUM_LIST variable. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27utils: add is_semver functionJean-Marie LEMETAYER
This function checks if a string is a semantic version: https://semver.org/spec/v2.0.0.html The npm fetcher needs this function to validate its version parameter. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27utils: add sha384_file and sha512_file functionsJean-Marie LEMETAYER
The npm fetcher needs these functions to support the subresource integrity: https://www.w3.org/TR/SRI/ Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22tests/fetch: Allow wget upgrade tests to run against a local serverRichard Purdie
Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19fetch2/git: _revision_key: collapse adjacent slashesChris Laplante
>From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19fetch2: do not suffix srcrev cache key with PNChris Laplante
Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19tests/utils: add tests for bb.utils.get_referenced_varsChris Laplante via bitbake-devel
Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bb.utils: add get_referenced_varsChris Laplante via bitbake-devel
Given a start expression, bb.utils.get_referenced_vars returns the referenced variable names in a quasi-BFS order (variables within the same level are ordered aribitrarily). For example, given an empty data store: bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d) returns either ["A", "B", "C"], ["A", "C", "B"], or another permutation. If we then set A = "${F} ${G}", then the same call will return a permutation of [A, B, C] concatenated with a permutation of [F, G]. This method is like a version of d.expandWithRefs().references that gives some insight into the depth of variable references. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19lib: amend code to use proper singleton comparisons where possibleFrazer Clews
amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19lib: remove unused importsFrazer Clews
removed unused imports which made the code harder to read, and slightly but less efficient Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11runqueue: Use a set for the setscene tasks listRichard Purdie
This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11knotty: Be consistent when creating/updating progress barsPeter Kjellerstedt
When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11knotty: Hide the footer if a process progress bar is shownPeter Kjellerstedt
With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-08cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt
When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01cache: Lower debug level for wold build messagesRichard Purdie
These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01siggen: Test extra cross/native hashserv methodRichard Purdie
Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01runqueue: Fix task dependency corner case in sanity testRichard Purdie
A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01runqueue: Ensure task dependencies are run correctlyRichard Purdie
We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-06-15 12:38:53 +0000