summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2020-02-18cooker/siggen: Empty siggen cache during parsing2019-10.2-zeus1.44.2Richard Purdie
When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Avoid cache mismatch issues with locked sigsRichard Purdie
If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 27ad9c1d468fba858a4adeb56b605227b415ae0f) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08siggen: Cache unihash values to avoid cache lookupRichard Purdie
Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5c9cc45b60904a1c355db9bf9c4495f1b25aca37) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08siggen: Optimise get_unihash disk based cache handlingRichard Purdie
Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews
Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a194f275235f22411cb2368f06a44f61ceb6a0f3) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante
To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit feb43e7c30f5bfab75d718896c45df621810d06f) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08fetch2/git: _revision_key: collapse adjacent slashesChris Laplante
>From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 425e21c14955dd38868c6e97637df3bbe0f89fac) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-02-08fetch2: do not suffix srcrev cache key with PNChris Laplante
Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6c938e6fd29beebe09b32be839dae008fe6491d2) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-18runqueue: Use a set for the setscene tasks listRichard Purdie
This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-18knotty: Be consistent when creating/updating progress barsPeter Kjellerstedt
When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-18knotty: Hide the footer if a process progress bar is shownPeter Kjellerstedt
With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-18cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt
When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02cache: Lower debug level for wold build messagesRichard Purdie
These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1a9247c468cf09da60e5d396ccb81e950841c99e) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02siggen: Test extra cross/native hashserv methodRichard Purdie
Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0a09b0fa03d1afc08037964dc63a18ef7cff9c78) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02runqueue: Fix task dependency corner case in sanity testRichard Purdie
A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02runqueue: Ensure task dependencies are run correctlyRichard Purdie
We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7a92b7f58ab187eddfe550bd6fb687240c7b11bb) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02runqueue: Fix equiv hash handling build failuresRichard Purdie
Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 40928f6991436cf687821015324483b205abfcb1) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02prserv/serv: Use with while reading pidfileOla x Nilsson
Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6fa8a18ea4994031fdd1253fe363c5d8eeeba456) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2020-01-02lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directoriesAníbal Limón
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-12-17lib/bb: Optimise out debug messages from cookerRichard Purdie
We have bb.debug(2, xxx) messages in cooker which are useful for debugging but have really bad effects on performance, 640,000 calls on recent profile graphs taking tens of seconds. Rather than commenting out debug which can be useful for debugging, don't create events for debug log messages from cooker which would never be seen. We already stop the messages hitting the IPC but this avoids the overhead of creating the log messages too, which has been shown to be signficiant on the profiles. This allows the code to perform whilst allowing debug messages to be availble when wanted/enabled. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f04cd931091fb0508badf3e002d70a6952700495)
2019-12-17runqueue: Only call into the migrations function if migrations activeRichard Purdie
This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d446fa89d206fbc6d098215163c968ea5a8cf4a9)
2019-12-17runqueue: Optimise task filteringRichard Purdie
We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17)
2019-12-17runqueue: Optimise out pointless loop iterationRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 105d1f0748edde7753a4063e6fdc758ffc8a8a9e)
2019-12-17runqueue: Optimise task migration code slightlyRichard Purdie
Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e28ec69356f1797de3e4e3fca0fef710bc4564de)
2019-12-17runqueue: Fix sstate task iteration performanceRichard Purdie
Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit de18824996841c3f35f54ff5ad12f94f6dc20d88)
2019-12-17siggen: Split get_tashhash for performanceRichard Purdie
There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3)
2019-12-17runqueue: Fix task mismatch failures from incorrect logicRichard Purdie
The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 608b9f821539de813bfbd9e65950dbc56a274bc2)
2019-12-17runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie
The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c9c68d898985cf0bec6fc95f54c151cc50255cac)
2019-12-17bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6beddf6214e22b4002626761031a9e9d34fb04db)
2019-12-17siggen: Fix performance issue in get_unihashRichard Purdie
There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e561672d039ebfb8cd0e0654a44dcf48513317c)
2019-12-17runqueue: Batch scenequeue updatesJoshua Watt
Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes). Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit db033a8f8a276d864bdb2e1eef159ab5794a0658)
2019-12-17siggen: Ensure new unihash propagates through the systemRichard Purdie
Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit abcaa1398031fa5338a43859c661e6d4a9ce863d)
2019-12-17siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies ↵Richard Purdie
rehash An example: NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset. Change the code so that nostamp taints are preserved to avoid the issue. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 61624a3fc38e8546e01356d5ce7a09f21e7094ab)
2019-12-17knotty/uihelper: Switch from pids to tids for Task event managementRichard Purdie
We've seen cases where a task can execute with a given pid, complete and a new task can start using the same pid before the UI handler has had time to adapt. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main helper.eventHandler(event) File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler del self.running_tasks[event.pid] KeyError: 13490 This means using pids to match up events on the UI side is a bad idea. Change the code to use task ids instead. There is a small amount of fuzzy matching for the progress information since there is no task information there and we don't want the overhead of a task ID in every event, however since pid reuse is unlikely, we can live with a progress bar not quite working properly in a corner case like this. [YOCTO #13667] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e427eafa1bb04008d12100ccc5c862122bba53e0)
2019-12-17runqueue: Add extra debugging when locked sigs mismatches occurRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3aad9978be2a40d4c535a5ae092f374ba2a5f627)
2019-12-17runqueue/siggen: Allow handling of equivalent hashesRichard Purdie
Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen. This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
2019-12-17hashserv: Add support for equivalent hash reportingRichard Purdie
The reason for this should be recorded in the commit logs. Imagine you have a target recipe (e.g. meta-extsdk-toolchain) which depends on gdb-cross. sstate in OE-Core allows gdb-cross to have the same hash regardless of whether its built on x86 or arm. The outhash will be different. We need hashequiv to be able to adapt to the prescence of sstate artefacts for meta-extsdk-toolchain and allow the hashes to re-intersect, rather than trying to force a rebuild of meta-extsdk-toolchain. By this point in the build, it would have already been installed from sstate so the build needs to adapt. Equivalent hashes should be reported to the server as a taskhash that needs to map to an specific unihash. This patch adds API to the hashserv client/server to allow this. [Thanks to Joshua Watt for help with this patch] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 674692fd46a7691a1de59ace6af0556cc5dd6a71)
2019-11-29runqueue.py: not show warning for deferred multiconfig taskKai Kang
When follow the instructions of multiconfig from Yocto dev manual that set in core-image-sato recipe: do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" it show too many annoying warnings look like: | WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot | after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot Treat them as infomations rather than warnings. Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-27tests/runqueue: Fix to match recent task migration fixes2019-10.1-zeus1.44.1Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25runqueue: Ensure failed harddependencies in scenequeue are accounted for in ↵Richard Purdie
migrations Setscene hard dependencies were not being correctly handled during task migration. For example, do_package of recipe X might become valid due to hashequiv yet we're still rebuilding pseudo-native, a harddep of do_package. This would mean it would try to execute that setscene task despite pseudo not being present. Fix this by ignoring tasks with failed harddependencies. This does mean stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately the new runqueue model can likely better handle these cases than the older codebase could but that is for another more invasive patch. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25runqueue: Improve sstate rehashing outputRichard Purdie
Bibake is currently too 'chatty' when hash equivalence is enabled. Fix this by only printing the log output if a rehash happens and it matches an sstate object. Also, pass a summary option to the hash checking function. This was already changed to a mechanism which allows addition of new parameters so this should be backwards and forwards compatible. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25siggen: Fix hashequiv bug where new hash wasn't referenced correctlyRichard Purdie
If a hash is reported to the hash server, the stamp written out by the current task didn't account for any new hash the server may have provided. Fix this so the correct stamp is written. This means "bitbake X; bitbake X" no longer rebuilds lots of things when hashequiv is active. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-25prserv/serv: Only restart the server if settings changeRichard Purdie
The server is now restarting when running commands which doesn't make sense. Only restart if its configuration has changed. This should potentially fix various memory resident bitbake usages too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-23runqueue: Fix hash equivalence duplicate tasks runningRichard Purdie
The key problem is that currently running setscene tasks are not accounted for when processing task migrations. This means can allow two of the same task to execute at the same time with unpredictable effects. This change allows us to stop doing that and refactor the code slightly to make it clearer that these conditions don't arrive even with deferred tasks. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23fetch2/clearcase: Fix warnings from python 3.8Richard Purdie
bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="? if command is 'mkview': bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'rmview': bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'setcs': Python 3.8 is quite correct and we so mean "==" here, fix it to avoid the warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23utils: also use mmap for SHA256 and SHA1, for performanceRoss Burton
md5_file() uses a mmap() window to improve performance when hashing files, so refactor the code and do the same for SHA1 and SHA256. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23tests: add test for the hashing functionsRoss Burton
Add a basic test for bb.utils.md5_file() etc. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23cooker: Remove a left-over comment about expanded_dataPeter Kjellerstedt
This should have been removed together with expanded_data in commit e3694e73 (cooker/command: Drop expanded_data). Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23prserv: fix ResourceWarning due to unclosed socketGavin Li
With PRSERV_HOST = "localhost:0", this message would occasionally pop up during the initial cache read: WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)> value = pickled.load() The file location stated is irrelevant; it just happens to be wherever CPython decides to run the garbage collector. The issue is that after we fork off a PRServer, self.socket is also duplicated. The parent side of it also needs to be closed. Signed-off-by: Gavin Li <gavin@matician.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>
2019-11-23fetch2/hg: Fix various runtime issuesVolker Vogelhuber
Fix mercurial fetching after breakage from changes to the core fetcher. Fix username and password usage and setting moddir needed by setup_revisions. Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com>