| Age | Commit message (Collapse) | Author |
|---|
|
I realised only the first logging message was being displayed in a given
parsing process. The reason turned out to be the UI handler failing
with a "pop from empty list". The default handler was then lost and
no further messages were processed.
Fix this by catching the exception correctly in the connection writer code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3e64f64525187f1409531a0bd99df576e627f7f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 543315e6463f15ca7ab2b4ef3e8ed41bb4207ccf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.
Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.
I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139
Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This debug is useful but the cooker shutdown or post_serve() may have cleanup
left so run after those.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids backtraces when starting toaster or using bitbake in
remote mode.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the code can just show nothing as the exception if there was a double
fault, which in this code path is quite likely. This leads to an error log
which effectively says "it failed" with no information about how.
Improve things so we get a nice verbose traceback left in the logs/output
which is preferable to no logs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:
INHERIT += "this-class-does-not-exist"
Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've seen cases where an OOM error causes bitbake server to hang:
9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
server.run()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
ret = self.main()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
ready = self.idle_commands(.1, fds)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
bb.event.fire(heartbeat, self.cooker.data)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
fire_class_handlers(event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
ret = handler(event)
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are autobuilder logs where the server commands are failing
but we have no debug info in the server log. Improve this to try and
understand what is failing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its hard to tell from the server logs whether commands complete or not
(or how long they take). Add extra info to allow more debugging of
server timeouts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Knowing which process printed which messages and the timestamp of the
message is useful for debugging, so add this. Ensure the log parsing
isn't affected by using search() instead of match().
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Showing "leftover process" messages when a new server has started and is being
used by some UI is horrible. Compare the PID data from the lockfile to
avoid this (and the ton of confusing log data it generates).
Also, move the time.sleep() call to be after the first lock attempt, which
reduces noise in the logs significantly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
lsof/fuser error if the file doesn't exist. It can be deleted by something
else so ignore this if it happens and loop.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The cookerlog output goes to a file and its misleading to look at it and
not have it up to date with what the cooker is actually doing. Ensure
written data is flushed.
Ultimately this should be using python's logging but that is for another
day, we need simple fixes right now.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The binary string printed into the output is ugly, parse this so the
linebreaks come out in the logs and make them much more readable (I
was misssing the information initially despite looking for it).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With an invalid layer, the desired error output should be:
ERROR: The following layer directories do not exist:
ERROR: /this_path_does_not_exist
ERROR: Please check BBLAYERS in .../build-invalid-layer/conf/bblayers.conf
Instead we were met with a backtrace:
Traceback (most recent call last):
File "/scratch1/fray/xilinx/poky/bitbake/bin/bitbake", line 36, in <module>
cookerdata.CookerConfiguration()))
...
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 267, in parseBaseConfiguration
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 358, in parseConfigurationFiles
sys.exit(1)
SystemExit: 1
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.
Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.
In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.
[YOCTO #14022]
[YOCTO #14033]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I have a suspicion based on process traces that the flock() call is no
longer interrupted by SIGALRM and hence the timeout doesn't work. We
were relying on EINTR triggering around syscalls but python is likely
protecting us from that in modern versions.
Re-implement this code with a different mechanism which doesn't have
that potential issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using sys.executable ensures we're using the same python binary for the server
as the client, which is probably advisable. "bitbake-server" is left as the process
name as its more distinctive in process listings.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.
Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Dump info into the logs if there are extra threads left at process exit
time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sock object isn't used client side so we can just created it in
the server process and save passing around the fd/object.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.
The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is the possibility data is being lost from the logfile due to
data buffering. Add in a couple of extra flush calls to ensure data
is being written out before the lock file is dropped.
Possible fix for [YOCTO #14000]?
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Further cleanup, just pass these settings directly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We already call bb.daemonize.createDaemon() in BitBakeServer so the extra
multiprocessing.Process() appears to be totally unneeded and just an extra layer
of forking which confuses things. Remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
UI control can happen via the xmlrpc connection. Account for this when timing
out UI connections. This was causing issues for toaster on systems where it
couldn't parse the metadata within the timeout.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're only meant to be doing UI connection timeouts on the first connection
but haveui changes for each connection. We need to add a specific variable
to track this correctly and get the intended behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its bb.note or logger.info, this avoids a backtrace.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.
Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.
Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.
I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.
This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.
An example cooker-deamonlog:
--- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
--- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
--- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting
where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
removed unused imports which made the code harder to read, and slightly
but less efficient
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
exceptions
The parseBaseConfiguration() raises bb.BBHandledException(), but
BitBakeServer() didn't handle it, so we always got unexpected exceptions when
there were errors. For example:
=== Case 1:
* Add "print "hello"' in base.bbclass' def oe_import() function
def oe_import(d):
print "hello"
[snip]
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21
print "hello"
^
SyntaxError: Missing parentheses in call to 'print'
<The first exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
<The second exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
<The third exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
[snip]
Now it looks like:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21:
The code lines resulting in this error were:
0001:def oe_import(d):
*** 0002: print "hello"
0003: import sys
0004:
0005: bbpath = d.getVar("BBPATH").split(":")
0006: sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21)
=== Case 2:
* Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
<Tracebacks>
[snip]
During handling of the above exception, another exception occurred:
[snip]
<Tracebacks>
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
hello
The error message is printed by bb.fatal() which raises bb.BBHandledException(),
but BitBakeServer() doesn't handle it, so we got it.
Now it looks like:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
hello
No unexpected exceptions anymore.
[YOCTO #13267]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.
This explains some mystery hangs we've been seeing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.
Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.
This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.
Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.
This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.
If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current value of 2 seconds has shown to be short in
wider testing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Peter Kjellerstedt
Alexander Kanavin
Martin Jansa
Joshua Watt
Ross Burton
Mark Hatle
Frazer Clews
Frazer Clews
Robert Yang