| Age | Commit message (Collapse) | Author |
|---|
|
Greatly simply the unpack rule by copying the general functionality of
update_submodules as unpack_submodules. This will recursively construct
a set of urls and unpack them using the standard system behaviors.
The overall code may be slightly bigger, but this ensures that all of the
standard locks are inplace, ensuring the code doesn't change out from
under the unpack function. (This could have happened before due to using
'cp' instead of further unpacks on submodules. This may still happen in
shallow clones.)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was reported that a race condition on a shared download directory could
occur with the gitsm fetcher, the result happened with a call to
git config
that occured within the update_submodules. Since the fetch is locked by the
upper level, it was probably the prior need_update(...) function causing this
because of some old code.
The gitsm class inherits the git class. The need_update was overridding the
version in gitsm, so that it forceably checked the submodules.
It's clear we can optimize the code by only updating if the primary repository
needs updating. Since we don't care if the submodule repository has changed
because if the primary hasn't, references to the submodule won't change.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In order to test the ssh processing in gitsm, we add an alternative
testcase that can be downloaded from git.yoctoproject.org. However,
this test case requries (read) access, via ssh, to git.yoctoproject.org.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The following appear to be the git supported formats:
proto://user:pass@host/path (URI format)
user@host:path (SSH format)
/path or ./path or ../path (local file format)
We adjust the parsing to find out if we have a URI format or not.
When we are NOT in URI format, we do our best to determine SSH or
file format by looking for a ':' in the overall string. If we find
a ':' we assume SSH format and adjust accordingly.
Note, in SSH format we simply replace the ':' with a '/' when constructing
the URL. However, if the original path was ":/...", we don't want '//' so
we deal with this corner case as well.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is possible for a submodule to be defined in the .gitmodules file, but
never initialized in the repository itself. This shows itself when searching
for the defined module hash you will get back a empty value.
Similarly we need to identify and skip defined but not initialized submodules
during the unpack stages as well.
Thanks to raphael.lisicki@siemens.com for their help is figuring out how
to resolve this issue.
Additionally a problem was found where, while unlikely, it may be possible
for the wrong revision to have been searched using ls-tree. This has been
resolved in the update_submodules function by keeping the correct revision
along with the submodule path.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
./lib/bs4/element.py:1565: DeprecationWarning: Using or importing the
ABCs from 'collections' instead of from 'collections.abc' is deprecated,
and in 3.8 it will stop working
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
Create a new build
$ bitbake quilt-native -ccleansstate -Snone
$ bitbake quilt-native -ccleansstate -Sprintdiff
[snip]
latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
> prevh = __find_md5__.search(latestmatch).group(0)
output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
AttributeError: 'NoneType' object has no attribute 'group'
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The unihash should be fetched using the task filename that includes the
multiconfig prefixes.
[YOCTO #13124]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Similarly to the codeparser change, change to sha256 hashes due to worries
over collisions. The main impact of this change is slightly slower parsing
time as well as longer sstate file names.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've reports of hash collision with codeparser. Looking at the way collision
problems occur with md5 and the way our function templating works, I can believe
we may run into issues.
This patch therefore switches to sha256.
Performance wise, parse time could appear to rise by 4s in 374s
Before:
384329 in 2.966s (md5)
After:
349743 in 2.340s (sha256)
34723 in 1.245s (md5)
since we still have md5 used elsewhere in the code, something we should look at
next (using sha256 everywhere is around 5.3s in total)
Unfortunately this does nearly double the size of the codeparser cache file
due to the hash size change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have two choices, split the recipes amongst the parsing threads in
blocks ahead of time, or have a queue which parsers pull from when idle.
The optimum approach depends on how similar the pieces are. For the single
recipe reparse case, there is currently a significant wait for the feeder
thread to start (around 0.25s in a 2s command).
Its possible splitting into blocks in advance may be unluckly for some other
workloads but experimentally it seems to work better overall for me at least.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There used to be many bugs in multiprocessing and we implemented our own
feeder process to avoid them. Now that we have python 3.x, these are fixed
and just using the standard Queue mechanism appears to work fine. We can
therefore drop the unneeded code and simplify.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a test for x86_64 in overrides where is was being incorrectly handled.
There was a previous fix (3a3be518536acc868c7eeb3c1111ad1b321480b7) but
this ensures we don't regress.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Implements a reference implementation of the hash equivalence server.
This server has minimal dependencies (and no dependencies outside of the
standard Python library), and implements the minimum required to be a
conforming hash equivalence server.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Under heavy load with process delays, sqlite can issues a "locking
protocol" error (SQLITE_PROTOCOL). Unfortunately, it is impossible to
distinguish between actual locking protocol errors and this race
condition, so they best that can be done is to retry the operation when
the error is detected.
[YOCTO #13108]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the unique hash is being used to track task dependencies, the hash
validation function needs to know about it in order to properly validate
the hash.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The unique hash is now passed to the task in the BB_UNIHASH variable
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Requests the task unique hash from siggen and tracks it
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Abstracts the function to get the unique hash for a task. This hash is
used as in place of the taskhash for the purpose of determine how other
tasks depend on this one. Unless overridden, the taskhash is the same as
the unique hash, preserving the original behavior.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When an indirect multiconfig dependency exists, such as:
A depends on B, B has a multiconfig dependency to C,and our build
target is A, the multiconfig dependency to C is not processed on
time, hence no providers are added for it, causing an exception in
the runqueue because the dependency does exist in it.
Call add_unresolved() for all available multiconfigs before processing
providers for multiconfig dependencies, detecting mcdepends on time so
providers for them can be added correctly.
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Follow dependent hash changes recursively also when specifying two
signature files explicitly. Previously this was only done when using the
--task option.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The functionalities of bitbake-diffsigs and bitbake-dumpsig are so
similar that they can be merged into one. Add an option --dump to make
bitbake-diffsigs dump the last signature data instead of comparing it.
Keep bitbake-dumpsig as a symbolic link to bitbake-diffsigs. When it is
called as bitbake-dumpsig, it behaves as if --dump was specified.
Also make -D the short option for --debug again (the way it used to be,
and still was for bitbake-dumpsig), so that -d can be used as the short
option for --dump.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #12399]
The bug was to get the BitBake User Manual into the YP Mega-manual.
All the changes here create unique tags used with variables in the
BitBake Manual. Prior to the fix, tags were identical between like
variables in the YP reference manual and the BitBake User Manual.
The reason for this is because when I created the BitBake manual's
glossary, it was a cut-and-paste operation to get the bulk of
the work started. At the time, the BitBake User Manual was not
a part of the Mega-manual. Once we decided to include the
BitBake User Manual as part of the Mega-Manual, building the
mega-manual produced warnings for all these duplicate links.
To fix, I have updated the variable tags in the BitBake User
Manual to use the following form:
'var-bb-<variable_name>'
The tags used in the YP ref-manual follow this form (original):
'var-<variable_name>'
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.
Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.
This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.
Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.
This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.
If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current bitbake output makes it hard to know which retry is being
attempted. Add this information to the output to make it clearer.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current value of 2 seconds has shown to be short in
wider testing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds a test suite for testing the persistent data cache
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With aarch64 hosts coming into use, set the syscall number to
avoid ioprio warnings on that platform.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing problems due to the way x86-64 is handled (or not handled)
as an override. Relax the containts on overrides from being lowercase
to being lowercase or numeric. This fixes problem where MACHINE=qemux86
would work but MACHINE=qemux86-64 would fail the same tests.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix some further python3 warnings about unescaped regexs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The p_error() is used for printing errors when parse shell code, but it can't
the EOF error correctly
- Add the following lines to quilt.inc
do_configure_prepend () {
find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \;
if [ hello ]; then
}
- Before the patch:
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb
[snip]
bb.pysh.sherrors.ShellSyntaxError: None
followed by:
We can see that this isn't easy to debug, let p_error() check wheter it is EOF
and print appropriate errors can improve the error message. And don't let
codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error()
only raise ShellSyntaxError), but make it print the last 5 lines which might be
useful for debuging.
- After the patch
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are:
find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
if [ hello ]; then
autotools_do_configure
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile
[snip]
File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None):
w('Unexpected EOF')
> raise sherrors.ShellSyntaxError(''.join(msg))
bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
>>> from Set import Set
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'Set'
So remove related code.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The only two that we need are ShellSyntaxError and ShellError, others are not
used, so remove them.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We don't use these modules in bitbake, so remove them.
And pysh's author declared that pysh is no longer maintained:
http://pysh.sourceforge.net/
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The configuration of the sqlite database can timeout due to locking
under heavy load and should be subject to the same retry logic as the
other statements.
[YOCTO #13069]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've already tweaked remove() for speed and not to error if it
races for deletion. Therefore use this for prunedir() which was
starting to show the same bug reports.
[YOCTO #13003]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
os.errno used to happen to work but is invalid. Correct to use errno.
[YOCTO #13068]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This could clearly never have worked since the python3 migration
but as its in an error path, it doesn't get tested/used much.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing slow startup in bitbake, add some timeing debug messages so
the logs are more useful for debugging when its slow.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.
The delays come from setting up the inotify watches (31s) so can't
really be avoided.
After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove a deprecated warning and stop using our own deprecated API!
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The mechanism used to get the hash for a stamp file is split out so that
it can be overridden by derived classes
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Turns off the shared cache. It isn't a significant factor in performance
(now that WAL is enabled), and is a really bad idea to have enabled in
processes that fork() (as bitbake it prone to do).
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Enabling the write ahead log improves database reliability, speeds up
writes (since they mostly happen sequentially), and speeds up readers
(since they are no longer blocked by most write operations). The
persistent database is very read heavy, so the auto-checkpoint size is
reduced from the default (usually 1000) to 100 so that reads remain
fast.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Constructs the "key" column in the persistent database as a non-NULL
primary key. This significantly speeds up lookup operations in large
databases.
[YOCTO #13030]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Mark Hatle
Richard Purdie
Robert Yang
Joshua Watt
Alejandro Enedino Hernandez Samaniego
Peter Kjellerstedt
Scott Rifenbark
Yong, Jonathan