aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2020-11-20doc/conf.py: add missing import syssphinxMert Kirpici
Due to the calls to sys.stderr.write() and sys.exit() in exception handling in case of sphinx_rtd_theme not being installed, the following exception is raised by Python due to the fact that sys module not being imported. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5ddf0e5bee0de59d07295fc5693e20b1a0380fde) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-11-13sphinx: partial undo (bitbake-user-manual: update perforce fetcher docs)Nicolas Dechesne
This change is not applicable to 1.46/dunfell branch, and was brought by the automatic backport of Sphinx docs, so let's undo it. Note that this patch also included a general statement which applies to 1.46, so this snippet was not removed. Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-11-13sphinx: undo (bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST ↵Nicolas Dechesne
example) This change is not applicable to 1.46/dunfell branch, and was brought by the automatic backport of Sphinx docs, so let's undo it. Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-11-13sphinx: import sphinx docsNicolas Dechesne
The Bitbake manual was migrated to Sphinx in Yocto Project 3.2. Since the docs between 3.2 and 3.1 are "similar", and since 3.1 is an LTS release, we agreed to backport the documentation onto 3.1. If we look at all docs changes in 3.1 and 3.2, we have the following: === Changes in 3.1 git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/1.46 -- doc/ ==== 324aaa7f bitbake-user-manual-metadata.xml: fix a minor error f92e19a3 doc: More explanation to tasks that recursively depend on themselves e4695176 doc: Clarify how task dependencies relate to RDEPENDS 25c5c79b user manual: properly tag content as <replaceable> be367887 docs: delete reference to obsolete recipe-depends.dot === Changes in 3.2/master git log --oneline b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/ ==== a7c47f1e sphinx: rename Makefile.sphinx 427721d8 sphinx: remove DocBook files d52190ea docs: static: theme_overrides.css: fix responsive design on <640px screens 9ae5cce7 docs: sphinx: report errors when dependencies are not met ec4c481a docs: update README file after migrationg to Sphinx c87cc35a docs: sphinx: replace special quotes with double quotes ebdeef2c docs: ref-variables: add links to terms in glossary 29081375 bitbake-user-manual: fix bad links a0f37789 sphinx: theme_override: Use bold for emphasis text cbc5ca48 sphinx: theme_override: properly set font for verbatim text 08b1ae23 sphinx: remove leading '/' 99ba6fe9 sphinx: update style for important, caution and warnings d99760cc sphinx: last manual round of fixes/improvements 4f94633a sphinx: bitbake-user-manual: insert additional blank line after title 63adcaa5 sphinx: add releases page 3e940d93 sphinx: conf: enable extlinks extension 9921c652 sphinx: index: move the boilerplate at the end of the page 4e461224 sphinx: add SPDX headers cb19159c sphinx: Enhance the sphinx experience/nagivation with: 10a54678 sphinx: tweak html output a bit 219b2348 sphinx: Makefile.sphinx: add clean and publish targets 35fdc185 sphinx: fixes all remaining warnings e11d2dd1 sphinx: fix links inside notes 57300955 sphinx: fixup for links fa304c01 sphinx: override theme CSS 29af1cd2 sphinx: switch to readthedocs theme e8359fd8 sphinx: bitbake-user-manual: use builtin sphinx glossary 6bf6c8d6 sphinx: initial sphinx support 84ccba0f sphinx: add initial build infrastructure 44b57216 bitbake-user-manual: update perforce fetcher docs 9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example 06b5cf0a bitbake-user-manual-metadata.xml: fix a minor error c92a266c doc: More explanation to tasks that recursively depend on themselves caf42243 doc: Clarify how task dependencies relate to RDEPENDS 647c13d4 user manual: properly tag content as <replaceable> 2effbb6e docs: delete reference to obsolete recipe-depends.dot We can conclude the following commits exist in 3.2 and not in 3.1 (if we filter out sphinx changes) 44b57216 bitbake-user-manual: update perforce fetcher docs 9186ca47 bitbake-user-manual: Add BBFILES_DYNAMIC 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example Out of these 3 changes, the following patches are for 3.2 only: 44b57216 bitbake-user-manual: update perforce fetcher docs 7689fa78 bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example To backport the Sphinx docs, we then need to cherry-pick all docs patches from 3.2/1.48 and 'undo' the two patches above. This first patch is the first step that imports all Sphinx files, and remove Docbook files. It was done with the following command: git cherry-pick -n \ $(git log --reverse --oneline \ b94dec477a8d48ebceec91952ba290798c56c1f5..origin/master -- doc/ \ | cut -f1 -d' ') Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2020-10-26tests/fetch: Update upstream master->main branchname transition2020-04.4-dunfell1.46.4Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-13bitbake: tests/fetch: add unit tests for SRC_URI with spaces in urlCharlie Davies
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e7dab75c8d1923abcbbc7c9ac7de215d720ccf26) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-13bitbake: fetch/git: add support for SRC_URI containing spaces in urlCharlie Davies
Microsoft's TFS VCS system allows for spaces in a git repository url. An example of a valid url is: ssh://tfs-my-company.org:22/tfs/My Projects/FooBar This commit adds support for such urls by implementing two changes. Firstly, when bitbake makes a git command line call the url is surrounded by quotes so that the url, regardless of spaces, is treated as one argument. Secondly, additional parsing of various filepath variables, which are based off of the url, are now completed with any spaces in the url replaced with underscores. Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eb38b6f0935763f7ba19e5618f376fcae1dac41a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-13bitbake-user-manual: Add BBFILES_DYNAMICKonrad Weihmann
- add missing entry for BBFILES_DYNAMIC, ported from yocto-docs - add description for the new inverse mode Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9186ca47ce73b4d1c87eb69163698a04679fb55c) Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17tests/fetch: Move away from problematic freedesktop.org urls2020-04.3-dunfell1.46.3Richard Purdie
We're either hitting rate limiting with freedesktop.org or the servers have intermittent network connections. Use our own mirror of these repositories instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-08-24fetch2: Drop cups.org from wget status checksRichard Purdie
Its becomming clear the upstream server doesn't like this, drop these two urls from the test, not sure we need them here anyway. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ab2ef942dc21f9639793c972f2e546edf9444783) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-08-24siggen: clean_basepath: remove recipe full path when virtual:xyz presentJean-Francois Dagenais
Before this fix, this example basepath (a): virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb:do_compile would get incorrectly "cleaned" into: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native:/full/path/to/recipes-example/helloworld/helloworld_1.2.3.bb When searching backwards in `a` trying to isolate the 'virtual:xyz' to add it to the end of the string, we need to consider `a` still has the recipe path and taskname. So stoping the rsplit after only 1 split is not enough. We want to reach the second ':' from the end. This way, we obtain: helloworld/helloworld_1.2.3.bb:do_compile:virtual:native reviewed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d193d93422a0ad62aa35b5d4ca5da8d422f72180) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-29server/process: Account for xmlrpc connectionsRichard Purdie
UI control can happen via the xmlrpc connection. Account for this when timing out UI connections. This was causing issues for toaster on systems where it couldn't parse the metadata within the timeout. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fa85a8263971c25e67fa3b421c686a90e46acd87) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-29server/process: Fix UI first connection trackingRichard Purdie
We're only meant to be doing UI connection timeouts on the first connection but haveui changes for each connection. We need to add a specific variable to track this correctly and get the intended behaviour. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e7c387c2e2fb2cc3ca1dc9d2029362909c326d72) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-23cooker: Handle multiconfig name mappings correctly2020-04.2-dunfell1.46.2Richard Purdie
"bitbake mc:arm:bash mc:arm:busybox" works but "bitbake multiconfig:arm:bash multiconfig:arm:busybox" does not. The reason is the list is modified whilst iterating. Don't do that. [YOCTO #13607] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-21server/process: Fix note reference -> infoRichard Purdie
Its bb.note or logger.info, this avoids a backtrace. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-13server/process: Ensure UI-less servers don't sit in infinite loopsRichard Purdie
If server startup is broken for some reason (e.g. lockfile issues) and no UI connection is made, the server will just sit inifinitely waiting. Add a timeout upon startup in the non-memory resident case so that such infinite waits are avoided. In the memory resident case, the server wouldn't have shut down in the first place or will timeout according to configuration. Since any race may mean the socket file is no longer present, ensure the unlink doesn't fault upon exit, thus ensuring any hashequiv or PRServ is removed from memory, allowing all processes to exit cleanly in such scenarios. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39888b750df12478e8bdea6727cca112dce1df85) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13server/process: Fix a rare lockfile raceRichard Purdie
We're seeing rare occasional races on the autobuilder as if two server processes have the lockfile at the same time. We need to be extremely careful this does not happen. I think there is a potential race in this shutdown code since we delete the lockfile, then call unlockfile() which also tries to delete it. This means we may remove a lock file now held by another process if we're unlucky. Since unlockfile removes the lockfile when it can, just rely on that and remove any possible race window. An example cooker-deamonlog: --- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 --- Started bitbake server pid 2266 Entering server connection loop Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) Processing Client Connecting Client Running command ['setFeatures', [2]] Running command ['updateConfig', XXX] Running command ['getVariable', 'BBINCLUDELOGS'] Running command ['getVariable', 'BBINCLUDELOGS_LINES'] Running command ['getSetVariable', 'BB_CONSOLELOG'] Running command ['getSetVariable', 'BB_LOGCONFIG'] Running command ['getUIHandlerNum'] Running command ['setEventMask', XXXX] Running command ['getVariable', 'BB_DEFAULT_TASK'] Running command ['setConfig', 'cmd', 'build'] Running command ['getVariable', 'BBTARGETS'] Running command ['parseFiles'] --- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 --- Started bitbake server pid 8252 --- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 --- Started bitbake server pid 13278 Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}] Running command ['getRecipes', ''] Running command ['clientComplete'] Processing Client Disconnecting Client No timeout, exiting. Exiting where it looks like there are two server processes running which should not be. In that build there was a process left sitting in memory with its bitbake.sock file missing but holding the lock (not sure why it wouldn't timeout/exit). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e1a7c1821483031b224a1570bfe834da755219cc) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13pyshyacc: allow double COMMA statementsKonrad Weihmann
this allows shell statements like '; ;' to pass the parser. As it may be bad code but still valid enough to execute Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13fetch2: Change git fetcher not to destroy old referencesRichard Purdie
It looks like we're about to see a lot of changes in branch names in repos. If we have the prune option here, those old names are lost, the changes propagate to our source mirrors and our old releases break. We have the force option so any replaced references should be replaced, its only orphaned branches which will now be preserved. I believe this behaviour will cause us fewer problems given the changes that look likely to happen. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 820ab886e79eea516560c0c008e4cf059c6e11a3) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13server/process: Increase timeout for commandsRichard Purdie
We're running into this timeout on loaded autobuilders in situations where things should otherwise succeed. Log a note in these cases and continue to try for longer. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e567743e70f426786ae54dcb5ab550748d9266e4) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13ui/teamcity: don't use removed logging classesChris Laplante
Allows the TeamCity frontend to be used again. Signed-off-by: Chris Laplante <mostthingsweb@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c5477ba79fcad4a887808dd0df9cfe3554e2c17a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13lib/ui/taskexp: Fix missing Gtk importDavid Khouya
Adding back gtk objects import. Fix bug introduce when adding validation on gtk import. Signed-off-by: David Khouya <dakhouya@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 765be5ef60668f8a1cfbcba248f4995725807196) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-13lib/ui/taskexp: Validate gi importDavid Khouya
When running bitbake -g -u taskexp without having gi python module or and invalid gtk version, bitbake fails with a stack trace. In case of import or version error, bitbake should exit with an error message instead of a stack trace. Signed-off-by: David Khouya <dakhouya@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2a2c507f239b047f34765312df4168030e38b90d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-01msg: Avoid issues where paths have relative componentsRichard Purdie
The autobuilder can end up using build/../ syntax which is an issue if the build directory is cleaned. Avoid this by using normpath() on the file path passed in. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 41988fec47eb196ab7195a75330a6d98de19101b) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-01runqueue: Avoid unpickle errors in rare casesRichard Purdie
In rare cases the pickled data from a task contains "</event>" which causes backtrace. This can be reproduced with something like: do_unpack_prepend () { bb.warn("</event>") } There are several solutions but the easiest is to catch this exception and look for the next marker instead as this should be the only way such an unpickle error could occur. This fixes rare exceptions seen on the autobuilder. Also add in other potential exceptions listed in the pickle manual page so that better debug is obtained should there be an error in this code path in future. exitcode doesn't need the same handling since we control what is in that data field and it could never contain </exitcode> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5ada512d6f9cbbdf1172ff7818117c38b12225ca) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-01siggen: Fix error when hash equivalence has an exceptionJoshua Watt
The code that handled exceptions from the hash equivalence client was raising an exception itself because hashserv.client wasn't imported Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a76290dfc6f34ff9f6efdb13a6db74b6b4759daf) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-01hashserv: Chunkify large messagesJoshua Watt
The hash equivalence client and server can occasionally send messages that are too large for the server to fit in the receive buffer (64 KB). To prevent this, support is added to the protocol to "chunkify" the stream and break it up into manageable pieces that the server can each side can back together. Ideally, this would be negotiated by the client and server, but it's currently hard coded to 32 KB to prevent the round-trip delay. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e27a28c1e40e886ee68ba4b99b537ffc9c3577d4) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-07-01test/fetch: change to better svn sourceakuster
fixes: svn: warning: W175002: Unexpected HTTP status 504 'Gateway Timeout' on '/openembedded/bitbake/!svn/vcc/default' svn: E205011: Failure occurred processing one or more externals definitions picked pcre2 [Yocto #13948] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1483d17108da02f5d615e83403d5fd6288ca957c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-22tests/fetch: Switch from git.infradead.org to a YP mirrorRichard Purdie
Upstream is unavailable, breaking tests. Switch to a YP mirror since if we can't reach that there are bigger problems. This should remove a source of intermittent failures on the autobuilder. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-01toaster-requirements.txt: require Django 2.22020-04.1-dunfell1.46.1Tim Orling
In commit 9730f95686b2ac72cf1fa513c555f7c7787e2667 Django 2.2 was enabled. Django 1.11 was EOL on April 1, 2020 Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ee15e78c6f9b59c221b1e43973ee4db20c5b443b) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-01bitbake-user-manual-metadata.xml: fix a minor errorKai Kang
In the '_remove' example in bitbake-user-manual-metadata.xml, there is no 'jkl' in the original value of FOO2. So remove it from result. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 06b5cf0ab6c6e518ac780d081fab5546334c5c7d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-01doc: More explanation to tasks that recursively depend on themselvesJacob Kroon
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c92a266c8e452833f2a590721aa1c2bd6fbeb2e0) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-01doc: Clarify how task dependencies relate to RDEPENDSJacob Kroon
Clarify that BitBake knows how to map entries defined in the runtime dependency namespace back to build-time dependencies (recipes) in which tasks are defined. Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit caf422435ad64aacbdab8a94da3115599dd0938b) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-01user manual: properly tag content as <replaceable>Robert P. J. Day
Tag a couple fields as replaceable to be consistent with rest of manual. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 647c13d4ae746a1bb9bd76ff318477dadb4d292f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-06-01docs: delete reference to obsolete recipe-depends.dotRobert P. J. Day
Given that generation of recipe-depends.dot was removed: commit 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30 Author: Chen Qi <Qi.Chen@windriver.com> Date: Thu Aug 22 15:52:51 2019 +0800 cooker.py: remove generation of recipe-depends.dot The information of recipe-depends.dot is misleading. delete mention of it from the user manual. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2effbb6e10b07dc12e4ecdf449ca29fc20968c59) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-04-24bitbake.conf: Drop unneeded variables from bitbake.confRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24data_smart: Handle hashing of datastores within datastores correctlyRichard Purdie
If there is a datastore within a datastore (e.g. BB_ORIGENV) then get-hash() doesn;t correclty handle the contents using the memory address instead of the contents. This is a patch from dominik.jaeger@nokia.com which addresses this problem. Its been low priority since we don't include BB_ORIGENV anywhere this would cause an issue as standard. [YOCTO #12473] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24cache: Fix performance problem with large numbers of source filesRichard Purdie
Some companies are using large numbers of patch files in SRC_URI. Rightly or wrongly that exposes a performance problem where the code does not handle the large string manipulations in a way which works efficienty in python. This is a modified version of a patch from z00539568 <zhangyifan46@huawei.com153340508@qq.com which addresses the performance problem. I modified it to use a more advanced regex, retain the "*" check and cache the regex. [YOCTO #13824] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24fetch2/wget: Set User-Agent when checking status of a URLPaul Barker
When a website is behind a CDN like Cloudflare there may be a "Browser Integrity Check" or other test applied to requests before they are allowed through to the server. Downloading via wget passes these tests as headers are set appropriately, however the Python urllib module may fail these tests unless additional headers are set. This causes Wget.checkstatus() to fail where Wget.download() would actually succeed. For Cloudflare in particular a valid User-Agent is needed, it's easy to add this to the headers in Wget.checkstatus(). The user agent string is copied from Wget._fetch_index(). Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24tinfoil: fix config_data mess up insane check while parsing multiple recipesHongxu Jia
Since commit [tinfoil: Simplify remote datastore connections][1] and [tinfoil: Add back ability to parse on top of a datastore][2] applied, bitbake run command parseRecipeFile with param config_data.dsindex rather than config_data. While calling tinfoil.parse_recipe_file() with one config_data (with the same config_data.dsindex) to parse multiple recipes, it will mess up insane check. It broke update_layer.py on layerindex, here are the simplified steps: [snip] t= bb.tinfoil.Tinfoil() t.prepare() data = bb.data.createCopy(t.config_data) fn = "path_to/oe-core/meta/recipes-graphics/images/core-image-clutter.bb" t.parse_recipe_file(fn, appends=False, config_data=data) fn = "path_to/oe-core/meta/recipes-graphics/packagegroups/packagegroup-core-x11-base.bb" t.parse_recipe_file(fn, appends=False, config_data=data) | File "path_to/oe-core/meta/classes/insane.bbclass", line 1303, in __anon_1304__path_to_oe_core_meta_classes_insane_bbclass | bb.fatal("Fatal QA errors found, failing task.") [snip] In above failure, RDEPENDS is assigned `${PACKAGE_INSTALL} ${LINGUAS_INSTALL} ${IMAGE_INSTALL_DEBUGFS}' in core-image-clutter.bb, but it broke insane check on packagegroup-core-x11-base.bb >From commit [remotedata: enable transporting datastore from the client to the server][3], it create a new DataSmart to save receive_datastore's remote_data Similarly, make a copy of config_data(with different config_data.dsindex) could fix the issue. [1] http://git.openembedded.org/bitbake/commit/?id=85e03a64dd0a4ebe71009ec4bdf4192c04a9786e [2] http://git.openembedded.org/bitbake/commit/?id=4618da2094189e4d814b7d65672cb65c86c0626a [3] http://git.openembedded.org/bitbake/commit/?id=784d2f1a024efe632fc9049ce5b78692d419d938 Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06tinfoil: Add back ability to parse on top of a datastoreyocto-3.12020-04-dunfell1.46.0Richard Purdie
This option was removed recently as we didn't realise the layerindex relies upon it. Add back the API which it turns out can be supported as long as we assume the datastore passed in is a remote datastore which it usually would be unless created locally. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06data/siggen: Don't expand ignored variablesRichard Purdie
If a variable is in the signature whitelist, we'd currently expand it, then later ignore the data. This is problemtic for code which has effects when expanded, recently source date epoch in OE-Core for example. We don't actually need to do this, if we pass the whitelist into the earlier function it can avoid the expansion. This also also give a small performance boost since we avoid running code in some cases. [YOCTO #13581] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06bitbake: Update to version 1.46Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29fetch2: Fix urldata_cache key issuesRichard Purdie
Upon inspection its clear the way the keys for this cache were being handled would break it and cause the cache to never be used. Fix this. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29toaster: fix for import build directory with var refs in BBLAYERSPaul Eggleton
Update importing a build directory to support where bblayers.conf sets BBLAYERS to a value that includes a variable reference e.g.: BBLAYERS = "${TOPDIR}/../meta \ ${TOPDIR}/../meta-selftest" [YOCTO #13707] Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29toaster: FieldError loading Reverse runtime dependenciesReyna, David
When selecting reverse runtime dependencies, filter on simply 'depends_on' instead of the obsolete 'depends_on__name' or 'depends_on__size'. [YOCTO #13717] Signed-off-by: Ahmed.Hossam <Ahmed.Hossam@opensynergy.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29toaster: support environment-safe manage commandsReyna, David
Directly support the various 'manage' commands from the Toaster executable, so that users do not have to manually set up the required environment and paths. Examples: $ . toaster manage createsuperuser $ . toaster manage lsupdates [YOCTO #13170] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29toaster: update to DunfellReyna, David
Toaster needs update to Dunfell [YOCTO #13847] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29toaster: migrate to Django-2.2David Reyna
Toaster migration to Django-2.2. Django-1.x has been deprecated. [YOCTO #13207] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24Revert "fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache""Khem Raj
As per mailing list discussion, the cache policy was behaving correctly before and wouldn't expect to update after the initial fetch even for AUTOREV. This reverts commit ba093a38539960e645e994a66ed7872a604c00a9. Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>