| Age | Commit message (Collapse) | Author |
|---|
|
These don't need to access recipe information, so let's not waste the
user's time parsing all recipes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
This module provides two components, layerindex, which is used to talk a
layerindex, such as layers.openembedded.org.
The other module is the 'manager'. This module will handle downloading,
re-writing the bblayers to match the downloads and other related layer
management tasks.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
|
|
Update stray usage of deprecated logger.warn(), which was introduced
to fetch2/__init__.py after all other instances had been replaced by
logger.warning():
http://git.openembedded.org/bitbake/commit/?id=5a53e7d7b017769a6eb0f0a6335735a1fe51a5ec
http://git.openembedded.org/bitbake/commit/?id=676a5f592e8507e81b8f748d58acfea7572f8796
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When wget is fetching a listing for a directory over FTP it writes to a
temporary file called .listing in the current directory. If there are many such
operations happening in parallel - for example during 'bitbake world -c
checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to
write to, read, and delete the same file.
This results in various failures such as the file disappearing before wget has
processed it or the file changing contents, which causes checkpkg to randomly
fail.
Mitigate the race condition by creating a temporary directory to run wget in
when doing directory listings.
[ YOCTO #11828 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The server shutdown is currenlty laggy and race prone. This patch:
* adds a waitpid so that no zombie server is left around if its not
running in daemon mode.
* adds a quit "sentinal" using a pipe so that we're not sitting in
a socket poll() until timeout in order just to quit.
* use a select() call to poll the socket and the pipe for a quit signal.
The net result of this change is that the prserv exits with the cooker server
and it does so immediately and doesn't wait for the select/poll calls to
timeout. This makes bitbake a lot more responsive for startup/shutdown and
doesn't cause UI timeout errors as often when prserv is used.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the sstate is accessed via HTTP, the existence check can fail due
to network issues, in which case bitbake silently continues without
sstate.
One such network issue is an HTTP server like Python's own SimpleHTTP
which closes the TCP connection despite an explicit "Keep-Alive" in
the HTTP request header. The server does that without a "close" in the
HTTP response header, so the socket remains in the connection cache,
leading to "urlopen failed: <urlopen error [Errno 9] Bad file
descriptor>" (only visible in "bitbake -D -D" output) when trying to
use the cached connection again.
The connection might also get closed for other reasons (proxy,
timeouts, etc.), so this is something that the client should be able
to handle.
This is achieved by checking for the error, removing the bad
connection, and letting the check_status() method try again with a new
connection. It is necessary to let the second attempt fail
permanently, because bad proxy setups have been observed to also lead
to such broken connections. In that case, we need to abort for real
after trying twice, otherwise a build would just hang forever.
[YOCTO #11782]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Trim the entered built target value so that Toaster is not
confused with no real targets nor a ghost second target.
[YOCTO #11727]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Set the clone progress to be off by default for the benefit of
command line projects and 'Local Yocto' builds. For Toaster managed
projects that do use the clone feature the clone progress status
is already explicitly set by the existing code and thus displayed.
[YOCTO #11744]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Change the task progress counting from the task order field
which excludes setscene to the task completion field which
counts all completed tasks regardless of type.
[YOCTO #9971]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not
going to raise an error, then fire an event so that scripts can listen
for it and get the checksums.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning
if no checksums are specified in the recipe. This is not intended to be
used from recipes - it is needed when we move to using more standard
code paths to fetch new files from scripts i.e. where we don't know what
the checksums are in advance.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want this API to be easier to use, so add missing function
documentation to help with that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).
(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is useful for the caller to know whether the uihelper has handled the
event passed so that it can skip other event handling code if so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Make it easy to determine if recipes are parsed (and thus information
about available recipes is in memory).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
$ bitbake --observe-only
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
And even:
$bitbake --observe-only -B localhost:-1
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
This was beucase the brackets were in a wrong position.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed 1:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
$ bitbake -m # shutdown it again
$ bitbake -m
Starting bitbake server...
Terminated bitbake server.
It starts the server and kill it which is redundant.
$ bitbake -m
$ bitbake --status-only
Starting bitbake server...
It starts the server which is not what we need.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
$ bitbake --server-only -B localhost:-1
Infinite loop
Reconnecting to bitbake server...
[snip]
Break the loop when server-only can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no global var "configuration", so the old code hang at
self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't
figure out why it hang (but not print errors).
Reproducer:
$ bitbake -B localhost:-1 world -k
Check bitbake.log, there was nothing, now fixed.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if this code is used with something like oeqa's xml logging
it fails as sys.stdout is an io stream. Add in try/except to handle
this case.
Add a waitpid() call to remove a zombie whilst forking.
Also, append to the logfile, don't overwrite it (otherwise
debugging can be a real pain when the server is restarting for
unknown reasons).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Finding the top level build directory is currently hard and relies on
having a complete cooker being setup. Add a helper function which
does the same thing without all the extra overhead. This is needed
to be able to locate the bitbake lockfile and hence the socket
for connecting clients in the new server model.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In preparation for rewriting this code, expand the relatively useless
base classes into the code itself.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.
Use the new command in the process server backend.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix one incorrect indentation and several lines with trailing spaces.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a signal is sent like SIGWINCH the select could be interrupted
so ignore the InterruptError like in XMLRPC server [1].
[1]
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/server/xmlrpc.py#n307
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To quote Paul:
Not that long ago we added a prefix to logged messages to allow us to see
where the message has been generated (recipe / task). This is undoubtedly
useful for errors and warnings, however, I'm not sure it's really appropriate
for bb.plain(). As an example, see the output for -c listtasks now:
...
NOTE: Executing RunQueue Tasks
nodejs-native-4.5.0-r0 do_listtasks: do_addto_recipe_sysroot
nodejs-native-4.5.0-r0 do_listtasks: do_build Default task for a recipe - depends on all other normal tasks required to 'build' a recipe
nodejs-native-4.5.0-r0 do_listtasks: do_checklicense
nodejs-native-4.5.0-r0 do_listtasks: do_checklicenseall
nodejs-native-4.5.0-r0 do_listtasks: do_checkpkg
...
This patch excludes PLAIN messages from this prefixing making the log output
neater.
[YOCTO #11457]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes
except bb.parse.SkipRecipe:
> bb.data.setVar("__SKIPPED", True, d)
if include == 0:
AttributeError: module 'bb.data' has no attribute 'setVar'
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When heartbeat event support was added it was only added to process.py. Add
it to server/xmlrpc too. There is duplicated code however since we're likely
to combine the server abstractions soon its not worth worrying about now.
This ensures the backends have the same event support.
[YOCTO #10741]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
You might think Queue.Queue.get(True, 0) would return an event immediately
if present and otherwise return. It doesn't, it immediately "times out"
and returns with nothing from the queue.
The behaviour we want is not to wait but return anything present which is
what .get(False) does so map to this.
This fixes some odd behaviour observed in some of the tinfoil selftests.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Paul Eggleton
Mark Hatle
Andre McCurdy
Ross Burton
Richard Purdie
Patrick Ohly
David Reyna
Robert Yang
Aníbal Limón
Enrico Scholz