aboutsummaryrefslogtreecommitdiffstats
path: root/lib/prserv
AgeCommit message (Collapse)Author
3 daysprserv: add bitbake selftestsMichael Opdenacker
Run them with "bitbake-selftest prserv.tests" Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: import simplificationMichael Opdenacker
Simplify the importone() hook: - to make it independent from the "history" mode which is client specific. - remove the "history" parameter - we want all values to be imported for binary reproducibility purposes. - using the store_value() function (which warrants you don't save the same value twice and doesn't write when you're using a read-only server) is enough. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: store_value() improvementsMichael Opdenacker
Add a test_checksum_value() to test whether a (version, pkgarch, checksum, value) entry already exists in the database. This is used to protect the store_value() function from an error when trying to store a duplicate entry in the database. Also check whether the current database is open in read-only mode. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: avoid possible race condition in database codeMichael Opdenacker
Remove a possible race condition by allowing a read-only server to create the PR table anyway. This avoids a failure if both a read-only and read-write server try to access an empty database at the same time. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <jpewhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: enable database sharingMichael Opdenacker
sqlite3 can allow multiple processes to access the database simultaneously, but it must be opened correctly. The key change is that the database is no longer opened in "exclusive" mode (defaulting to shared mode). In addition, the journal is set to "WAL" mode, as this is the most efficient for dealing with simultaneous access between different processes. In order to keep the database performance, synchronous mode is set to "off". The WAL journal will protect against incomplete transactions in any given client, however the database will not be protected against unexpected power loss from the OS (which is a fine trade off for performance, and also the same as the previous implementation). The use of a database cursor enabled to remove the _execute() wrapper. The cursor automatically makes sure that the query happens in an atomic transaction and commits when finished. This also removes the need for a "dirty" flag for the database and for explicit database syncing, which simplifies the code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: add "upstream" server supportMichael Opdenacker
Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one. This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y"). Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format." Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision. This is done by going on supporting the "history" mode that wasn't used so far (revisions can return to a previous historical value), in addition to the default "no history" mode (revisions can never decrease). Rather than storing the history mode in the database table itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode is now passed through the client requests. As a consequence, the table name is now "PRMAIN", which is incompatible with what was generated before, but avoids confusion if we kept the "PRMAIN_nohist" name for both "history" and "no history" modes. Update the server version to "2.0.0". Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: move code from __init__ to bitbake-prservMichael Opdenacker
This script was the only user of this code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
3 daysprserv: declare "max_package_pr" client hookMichael Opdenacker
Add missing declaration for the max_package_pr client hook Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: remove unnecessary codeMichael Opdenacker
In db.py, the ifnull() statement guarantees that the SQL request will return a value. It's therefore unnecessary to test the case when no value is found. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: correct error messageMichael Opdenacker
according to db.py, prserv.NotFoundError is returned here when adding a new value to the database failed Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: remove redundant exception handlerMichael Opdenacker
This exception handler is already present in db.py's get_value() code. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: add extra requestsMichael Opdenacker
Useful for connecting a PR server to an upstream one - "test-package" checks whether the specified package version and arch is known in the database. - "test-pr" checks a specified output hash is found in the database. Otherwise it returns 'None' instead of a new value. - "max-package-pr" returns the highest PR number for (version, arch) entries in the database, and None if not found Add new DB functions supporting the above, plus test_value() which tells whether a given value is available for the specified package and architecture. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: capitalization and spacing improvementsMichael Opdenacker
Choosing only one style of capitalization Add extra space after some commas too Remove idle spaces Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: use self.logger instead of logger directlyMichael Opdenacker
In both the PRServerClient and PRClient objects. This aligns with what is done in hashserv/server.py and makes it possible to benefit from possible specializations of the logger in the corresponding super classes, instead of using always the global logger. Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: use double quotes by defaultMichael Opdenacker
To aligh with the hashserv code Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14prserv: simplify the PRServerClient() interfaceMichael Opdenacker
serv.py: simplify the PRServerClient() interface by passing the server object instead of multiple arguments, and then retrieving the data through this object. This replicates what is done for ServerClient() in hashserv/server.py Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-01prserv/serv: Fix a PID file removal race on prserv stopYoann Congal
A race condition has happened where the exiting server removed the PID file between the existence check and the removal, resulting in a FileNotFoundError exception. The fix is to ignore the FileNotFoundError exception, the existence check is now redundant so remove it to simplify. Fixes [YOCTO #14341] Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-14asyncrpc: Add option to set log level when running as a processJoshua Watt
When running an asyncrpc server as a subprocess, it is often desired to run it with a lower logging level since the normal logging of clients connecting and disconnecting is not desired. As such, add an option to set the logging level of the server when running as a subprocess and set the level to WARNING when starting a local hashserver or PRserver Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2023-11-09asyncrpc: Add context manager APIJoshua Watt
Adds context manager API for the asyncrcp client class which allow writing code that will automatically close the connection like so: with hashserv.create_client(address) as client: ... Rework the bitbake-hashclient tool and PR server to use this new API to fix warnings about unclosed event loops when exiting Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-09asyncrpc: Abstract socketsJoshua Watt
Rewrites the asyncrpc client and server code to make it possible to have other transport backends that are not stream based (e.g. websockets which are message based). The connection handling classes are now shared between both the client and server to make it easier to implement new transport mechanisms Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12bitbake: Add copyright headers where missingRichard Purdie
Where copyright headers were not present, add them to make things clear. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-02prserv: make localhost workMingli Yu
After [1] introduced, the PR server doesn't work in docker when use below setting like before. PRSERV_HOST = "localhost:0" And it's because the localhost is resolved to an ipv6 address ::1 as the below bitbake-prserv shows. bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address So add the extra logic to make the localhost resolved as expected to make the PR service work especially in docker. [1] 6a2b23e2 prserv: Replace XML RPC with modern asyncrpc implementation Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-01prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"Martin Jansa
* When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23prserv: Add read-only modePaul Barker
[YOCTO #13659] Signed-off-by: Paul Barker <pbarker@konsulko.com> [updated for asyncrpc changes] Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-23prserv: Replace XML RPC with modern asyncrpc implementationPaul Barker
Update the prserv client and server classes to use the modern json and asyncio based RPC system implemented by the asyncrpc module. Signed-off-by: Paul Barker <pbarker@konsulko.com> [updated for asyncrpc changes, client split to separate file] Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06prserv: Handle requests in main threadPaul Barker
The prserver process is cleanly separated from the main bitbake process so requests can be handled in the main thread. This removes the need for a request queue and a separate request handling thread. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06prserv: Extract daemonization from PRServer classPaul Barker
The code to start the prservice process as a daemon is extracted out of the PRServer class and simplified. This makes the PRServer class easier to modernise as it no longer needs to worry about process management. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06prserv: Use multiprocessing to auto start prserverPaul Barker
We can use the modern multiprocessing support in Python instead of manually using fork to start the prserver process. To do this we need to set up the signal handlers for the prserver process in the work_forever function (which is now used as the main function for this process). The old code to start the prserver process using fork is not removed in this commit as it is tightly intertwined with the daemonization code which will be refactored in a following commit. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27prserv: Add connect functionPaul Barker
This function abstracts the setup of a PR service client connection so that openembedded-core doesn't need to be updated any time the details are changed. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27prserv: Drop unused dump_db methodPaul Barker
Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27prserv: Drop obsolete python version checkPaul Barker
Bitbake no longer supports Python 2 so this version check is obsolete. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19lib: amend code to use proper singleton comparisons where possibleFrazer Clews
amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-30prserv/serv: Use with while reading pidfileOla x Nilsson
Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-23prserv/serv: Only restart the server if settings changeRichard Purdie
The server is now restarting when running commands which doesn't make sense. Only restart if its configuration has changed. This should potentially fix various memory resident bitbake usages too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-13prserv: fix ResourceWarning due to unclosed socketGavin Li
With PRSERV_HOST = "localhost:0", this message would occasionally pop up during the initial cache read: WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)> value = pickled.load() The file location stated is irrelevant; it just happens to be wherever CPython decides to run the garbage collector. The issue is that after we fork off a PRServer, self.socket is also duplicated. The parent side of it also needs to be closed. Signed-off-by: Gavin Li <gavin@matician.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-23bitbake: prserv/serv: Use with while reading pidfileOla x Nilsson
Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06prserv: Use a memory journalRichard Purdie
We've seen PR Server timeouts on the autobuilder, this is likely from the journal being blocked on disk IO generated by the build. Since we're running with synchronous off, we may as well put the journal into memory and avoid any IO related stalls. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31prserv/serv: Gracefully handle the PR server exiting quicklyRichard Purdie
If the server exits quickly its PID may no longer exist. Handle this gracefully. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31prserv/serv: Rename self.quit -> self.quitflagRichard Purdie
self has a quit function and a variable. Separate this to two different things as the current setup is prone to breakage. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31prserv/serv: Send sentinel to stop handler threadRichard Purdie
Shutdown from SIGTERM currently has to wait for the handler thread to timeout. Add a sentinel value which triggers it to loop and allows for a quick exit. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31prserv/serv: Shut down any existing server before restartingRichard Purdie
This allows for cleaner code in cooker as any existing server is dealt with before a new one is started. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31prserv/cooker: Drop unused paramRichard Purdie
Drop pointless unused function parameter. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24prserv/serv: Improve process exit handlingRichard Purdie
The server shutdown is currenlty laggy and race prone. This patch: * adds a waitpid so that no zombie server is left around if its not running in daemon mode. * adds a quit "sentinal" using a pipe so that we're not sitting in a socket poll() until timeout in order just to quit. * use a select() call to poll the socket and the pipe for a quit signal. The net result of this change is that the prserv exits with the cooker server and it does so immediately and doesn't wait for the select/poll calls to timeout. This makes bitbake a lot more responsive for startup/shutdown and doesn't cause UI timeout errors as often when prserv is used. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21server: Rework the server API so process and xmlrpc servers coexistRichard Purdie
This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-06prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie
These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-05prserv/serv: Tweak stdout manipulation to be stream safeRichard Purdie
We've been seeing oe-selftest failures under puzzling circumstances. It turns out if you run oe-selftest on a machine with xmlrunner installed and have the recent tinfoil2 changes, the launching of PR server crashes leading to selftest hanging if using an autoloaded PR server. The reason is that xmlrunner uses an io.StringIO object as stdout/stderr instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method. We have to deal with both cases and in the python way, we try and then seek forgivness if we see an AttributeError or UnSupportedOperation exception. Unfortunately we have to deal with both cases as we may be performing a traditiional double fork() from the commandline, or a larger python program. [YOCTO #10866] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01bitbake: Convert to python 3Richard Purdie
Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-12daemonize/prserv/tests/fetch: Convert file() -> open()Richard Purdie
Use python3 compatible functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-24 07:48:26 +0000