aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2020-02-14bitbake: layerindex: use branch when specifiedmgh/layerindex-fetchJon Mason
When currently specified, the branch is used to verify the versioning of the meta layer, but the master branch is checked out. This change allows for the branch to be specified. Now it is easy to specify all of the meta layers being added are of the same version, without having to do it in each individual git tree. Also, it will error if there are branches without a matching version. Finally, this allows for meta layer git trees without a master branch. Signed-off-by: Jon Mason <jdmason@kudzu.us> Minor rework of the patch to use the layerBranch actual_branch since the layerindex referenced branch may be different then the overall release branch. Also adjust the patch to use the default git checkout branch instead of master if no branch was specified. (Some repositories don't have a master branch.) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
2020-02-08doc: minor tweaks to ch 1 of BB user manualRobert P. J. Day
Tweaks include: - hyphenation - rewording for brevity or clarification - adding <firstterm> markup where appropriate Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08doc: Standardize spelling on 'BitBake' throughout docsRobert P. J. Day
Since the proper spelling is, in fact, 'BitBake', might as well make it consistent throughout the user manual. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Avoid cache mismatch issues with locked sigsRichard Purdie
If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Cache unihash values to avoid cache lookupRichard Purdie
Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Optimise get_unihash disk based cache handlingRichard Purdie
Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-06cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews
Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27tests/fetch: add npmsw testsJean-Marie LEMETAYER
This commit adds some tests to validate the npmsw fetcher: - bb.tests.fetch.NPMTest.test_npmsw - bb.tests.fetch.NPMTest.test_npmsw_bad_checksum - bb.tests.fetch.NPMTest.test_npmsw_destsuffix - bb.tests.fetch.NPMTest.test_npmsw_dev - bb.tests.fetch.NPMTest.test_npmsw_mirrors - bb.tests.fetch.NPMTest.test_npmsw_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npmsw_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npmsw_npm_reusability - bb.tests.fetch.NPMTest.test_npmsw_premirrors Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: add the npmsw fetcherJean-Marie LEMETAYER
This commit adds a new npmsw fetcher that fetches every npm dependencies described in a npm shrinkwrap file: https://docs.npmjs.com/files/shrinkwrap.json.html The main package must be fetched separately: SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \ npmsw://${THISDIR}/npm-shrinkwrap.json" Since a separation has been created between the package and its dependencies, the package can also be fetched with a non npm fetcher without impacting the general behavior: SRC_URI = "git://github.com/foo/bar.git;protocol=https \ npmsw://${THISDIR}/npm-shrinkwrap.json" Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27tests/fetch: add npm testsJean-Marie LEMETAYER
This commit adds some tests to validate the npm fetcher: - bb.tests.fetch.NPMTest.test_npm - bb.tests.fetch.NPMTest.test_npm_bad_checksum - bb.tests.fetch.NPMTest.test_npm_destsuffix_downloadfilename - bb.tests.fetch.NPMTest.test_npm_mirrors - bb.tests.fetch.NPMTest.test_npm_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npm_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npm_package_invalid - bb.tests.fetch.NPMTest.test_npm_package_none - bb.tests.fetch.NPMTest.test_npm_premirrors - bb.tests.fetch.NPMTest.test_npm_registry_alternate - bb.tests.fetch.NPMTest.test_npm_registry_invalid - bb.tests.fetch.NPMTest.test_npm_registry_none - bb.tests.fetch.NPMTest.test_npm_version_invalid - bb.tests.fetch.NPMTest.test_npm_version_latest - bb.tests.fetch.NPMTest.test_npm_version_none Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2/npm: refactor the npm fetcherJean-Marie LEMETAYER
This commit refactors the npm fetcher to improve some points and fix others: - The big change is that the fetcher is only fetching the package source and no more the dependencies. Thus the npm fetcher act as the other fetchers e.g git, wget. The dependencies will be handled later. - The fetcher only resolves the url of the package using 'npm view' and then forwards it to a proxy fetcher. - This commit also fixes a lot of issues with the package names (exotic characters, scoped packages) which were badly handled. - The validation files - lockdown.json and npm-shrinkwrap.json - are no longer used by the fetcher. Instead, the downloaded tarball is verified with the 'integrity' and 'shasum' provided in the 'npm view' of the package [1][2]. 1: https://docs.npmjs.com/files/package-lock.json#integrity 2: https://www.w3.org/TR/SRI Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2/wget: fix downloadfilename parameterJean-Marie LEMETAYER
When using a download filename with characters which can be interpreted by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the filename fixes the issue. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the done conditionJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the done condition to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the mirrors managementJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "try_mirrors" functions to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: allow fetchers to forward the donestamp managementJean-Marie LEMETAYER
This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "verify_donestamp" and "update_stamp" functions to a proxy fetcher. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: add more hash functions for checksum verificationJean-Marie LEMETAYER
This commit enables the "sha1", "sha384" and "sha512" hash functions in the supported checksum list. This allows to use more SRC_URI checksums functions for a url: SRC_URI[sha1sum] = "..." SRC_URI[sha384sum] = "..." SRC_URI[sha512sum] = "..." The npm fetcher needs this to support subresource integrity: https://www.w3.org/TR/SRI/ Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27fetch2: refactor checksum verificationJean-Marie LEMETAYER
This commit refactors the way checksums are verified to be more generic. The support of new hash functions is now limited to the update of the CHECKSUM_LIST variable. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27utils: add is_semver functionJean-Marie LEMETAYER
This function checks if a string is a semantic version: https://semver.org/spec/v2.0.0.html The npm fetcher needs this function to validate its version parameter. Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27utils: add sha384_file and sha512_file functionsJean-Marie LEMETAYER
The npm fetcher needs these functions to support the subresource integrity: https://www.w3.org/TR/SRI/ Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22tests/fetch: Allow wget upgrade tests to run against a local serverRichard Purdie
Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19fetch2/git: _revision_key: collapse adjacent slashesChris Laplante
>From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19fetch2: do not suffix srcrev cache key with PNChris Laplante
Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19tests/utils: add tests for bb.utils.get_referenced_varsChris Laplante via bitbake-devel
Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19bb.utils: add get_referenced_varsChris Laplante via bitbake-devel
Given a start expression, bb.utils.get_referenced_vars returns the referenced variable names in a quasi-BFS order (variables within the same level are ordered aribitrarily). For example, given an empty data store: bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d) returns either ["A", "B", "C"], ["A", "C", "B"], or another permutation. If we then set A = "${F} ${G}", then the same call will return a permutation of [A, B, C] concatenated with a permutation of [F, G]. This method is like a version of d.expandWithRefs().references that gives some insight into the depth of variable references. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19lib: amend code to use proper singleton comparisons where possibleFrazer Clews
amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19lib: remove unused importsFrazer Clews
removed unused imports which made the code harder to read, and slightly but less efficient Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11runqueue: Use a set for the setscene tasks listRichard Purdie
This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11knotty: Be consistent when creating/updating progress barsPeter Kjellerstedt
When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-11knotty: Hide the footer if a process progress bar is shownPeter Kjellerstedt
With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-08cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt
When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01cache: Lower debug level for wold build messagesRichard Purdie
These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01siggen: Test extra cross/native hashserv methodRichard Purdie
Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01runqueue: Fix task dependency corner case in sanity testRichard Purdie
A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01runqueue: Ensure task dependencies are run correctlyRichard Purdie
We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-01runqueue: Fix equiv hash handling build failuresRichard Purdie
Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-30bitbake-layers: Keep loglevel in colored loggerOla x Nilsson
Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-30prserv/serv: Use with while reading pidfileOla x Nilsson
Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-30lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directoriesAníbal Limón
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-28base.bbclass: fix 'python should use 4 spaces indentation' warningsMasahiro Yamada
Even if there are not so many users of base.bbclass bundled in bitbake, it is helpful as sample code at least. If this base.bbclass is used, bitbake emits the following warnings: WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 41 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 42 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 43 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 44 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 45 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 46 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 47 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 53 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 54 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 55 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 56 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 63 WARNING: python should use 4 spaces indentation, but found tabs in base.bbclass, line 64 Let's fix the indentation as suggested. While I was touching these lines, I also fixed "isnt" into "isn't" in the comment line. Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-28fetch2/clearcase: Fix tar command working directoryCHerzig@Gauselmann.de
We need to ensure the correct working directory is used for this tar command to succeed. Signed-off-by: Christian Herzig <cherzig@gauselmann.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16lib/bb: Optimise out debug messages from cookerRichard Purdie
We have bb.debug(2, xxx) messages in cooker which are useful for debugging but have really bad effects on performance, 640,000 calls on recent profile graphs taking tens of seconds. Rather than commenting out debug which can be useful for debugging, don't create events for debug log messages from cooker which would never be seen. We already stop the messages hitting the IPC but this avoids the overhead of creating the log messages too, which has been shown to be signficiant on the profiles. This allows the code to perform whilst allowing debug messages to be availble when wanted/enabled. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16runqueue: Only call into the migrations function if migrations activeRichard Purdie
This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15runqueue: Optimise task filteringRichard Purdie
We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15runqueue: Optimise out pointless loop iterationRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15runqueue: Optimise task migration code slightlyRichard Purdie
Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15runqueue: Fix sstate task iteration performanceRichard Purdie
Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15siggen: Split get_tashhash for performanceRichard Purdie
There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15runqueue: Fix task mismatch failures from incorrect logicRichard Purdie
The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-14runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie
The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-26 06:20:51 +0000