aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* utils: Add aarch64 support to ioprio_setHEADmaster-nextmasterRichard Purdie4 hours1-0/+2
| | | | | | | With aarch64 hosts coming into use, set the syscall number to avoid ioprio warnings on that platform. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Allow numeric characters in overridesRichard Purdie4 hours1-1/+2
| | | | | | | | | We're seeing problems due to the way x86-64 is handled (or not handled) as an override. Relax the containts on overrides from being lowercase to being lowercase or numeric. This fixes problem where MACHINE=qemux86 would work but MACHINE=qemux86-64 would fail the same tests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Add missing regexp markupRichard Purdie4 hours1-2/+2
| | | | | | Fix some further python3 warnings about unescaped regexs. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-user-manual: Added "usehead" parameter.Yong, Jonathan4 days1-0/+8
| | | | | Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: pysh: Improve error handling for shell codeRobert Yang6 days2-9/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The p_error() is used for printing errors when parse shell code, but it can't the EOF error correctly - Add the following lines to quilt.inc do_configure_prepend () { find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \; if [ hello ]; then } - Before the patch: $ rm -fr cache/ tmp/cache/; bitbake -p [snip] WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb [snip] bb.pysh.sherrors.ShellSyntaxError: None followed by: We can see that this isn't easy to debug, let p_error() check wheter it is EOF and print appropriate errors can improve the error message. And don't let codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error() only raise ShellSyntaxError), but make it print the last 5 lines which might be useful for debuging. - After the patch $ rm -fr cache/ tmp/cache/; bitbake -p [snip] ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are: find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \; if [ hello ]; then autotools_do_configure sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile [snip] File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None): w('Unexpected EOF') > raise sherrors.ShellSyntaxError(''.join(msg)) bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pysh/pyshlex.py: Set is not importableRobert Yang6 days1-5/+0
| | | | | | | | | | | | >>> from Set import Set Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'Set' So remove related code. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pysh/sherrors.py: Remove unused classesRobert Yang6 days1-26/+0
| | | | | | | | The only two that we need are ShellSyntaxError and ShellError, others are not used, so remove them. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pysh: Remove unused modulesRobert Yang6 days5-2437/+0
| | | | | | | | | | We don't use these modules in bitbake, so remove them. And pysh's author declared that pysh is no longer maintained: http://pysh.sourceforge.net/ Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Retry database setupJoshua Watt6 days1-35/+48
| | | | | | | | | | | The configuration of the sqlite database can timeout due to locking under heavy load and should be subject to the same retry logic as the other statements. [YOCTO #13069] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: Use utils.remove() within prunedir()Richard Purdie7 days1-9/+1
| | | | | | | | | | We've already tweaked remove() for speed and not to error if it races for deletion. Therefore use this for prunedir() which was starting to show the same bug reports. [YOCTO #13003] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: Fix os.errno referencesRichard Purdie7 days1-2/+2
| | | | | | | | os.errno used to happen to work but is invalid. Correct to use errno. [YOCTO #13068] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/wget: Fix typo in exception nameRichard Purdie7 days1-1/+1
| | | | | | | This could clearly never have worked since the python3 migration but as its in an error path, it doesn't get tested/used much. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Add some timing debug messages to the server startupRichard Purdie7 days1-0/+19
| | | | | | | We're seeing slow startup in bitbake, add some timeing debug messages so the logs are more useful for debugging when its slow. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Increase server startup time delayRichard Purdie7 days1-1/+4
| | | | | | | | | | | | | | | On loaded production systems we've seen bitbake server take over 40s to start up. Increase the timeout to 90s which tries to avoid failures in selftests. The delays come from setting up the inotify watches (31s) so can't really be avoided. After 5s delay we now warn the user we're waiting for 90s so the interactive exeperience shouldn't be much changed and its very unlikely the user would see that anyway. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: Cleanup deprecated function usageRichard Purdie7 days1-1/+1
| | | | | | Remove a deprecated warning and stop using our own deprecated API! Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* siggen: Split out stampfile hash fetchJoshua Watt7 days1-4/+9
| | | | | | | | | | The mechanism used to get the hash for a stamp file is split out so that it can be overridden by derived classes [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-worker: Pass taskhash as runtask parameterJoshua Watt7 days2-13/+10
| | | | | | | | | | | | | Pass the task hash as a parameter to the 'runtask' message instead of passing the entire dictionary of hashes when the worker is setup. This is possible less efficient, but prevents the worker taskhashes from being out of sync with the runqueue in the event that the taskhashes in the runqueue change. [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: Disable enable_shared_cacheJoshua Watt7 days1-6/+0
| | | | | | | | | | | Turns off the shared cache. It isn't a significant factor in performance (now that WAL is enabled), and is a really bad idea to have enabled in processes that fork() (as bitbake it prone to do). [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: Enable Write Ahead LogJoshua Watt7 days1-0/+5
| | | | | | | | | | | | | | Enabling the write ahead log improves database reliability, speeds up writes (since they mostly happen sequentially), and speeds up readers (since they are no longer blocked by most write operations). The persistent database is very read heavy, so the auto-checkpoint size is reduced from the default (usually 1000) to 100 so that reads remain fast. [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: Add key constraintsJoshua Watt7 days1-1/+1
| | | | | | | | | | | Constructs the "key" column in the persistent database as a non-NULL primary key. This significantly speeds up lookup operations in large databases. [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: Fix leaking cursors causing deadlockJoshua Watt7 days1-53/+135
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The original implementation of persistent data executed all SQL statements via sqlite3.Connection.execute(). Behind the scenes, this function created a sqlite3 Cursor object, executed the statement, then returned the cursor. However, the implementation did not account for this and failed to close the cursor object when it was done. The cursor would eventually be closed when the garbage collector got around to destroying it. However, sqlite has a limit on the number of cursors that can exist at any given time, and once this limit is reached it will block a query to wait for a cursor to be destroyed. Under heavy database queries, this can result in Python deadlocking with itself, since the SQL query will block waiting for a free cursor, but Python can no longer run garbage collection (as it is blocked) to free one. This restructures the SQLTable class to use two decorators to aid in performing actions correctly. The first decorator (@retry) wraps a member function in the retry logic that automatically restarts the function in the event that the database is locked. The second decorator (@transaction) wraps the function so that it occurs in a database transaction, which will automatically COMMIT the changes on success and ROLLBACK on failure. This function additionally creates an explicit cursor, passes it to the wrapped function, and cleans it up when the function is finished. Note that it is still possible to leak cursors when iterating. This is much less frequent, but can still be mitigated by wrapping the iteration in a `with` statement: with db.iteritems() as it: for (k, v) in it: ... As a side effect, since most statements are wrapped in a transaction, setting the isolation_level when the connection is created is no longer necessary. [YOCTO #13030] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* main: Don't use print() directly, use loggerRichard Purdie9 days1-1/+1
| | | | | | | | | | | | | | | | | Avoid failures like: 2018-12-04 03:30:56,033 - oe-selftest - INFO - ====================================================================== 2018-12-04 03:30:56,033 - oe-selftest - INFO - FAIL: pkgdata.OePkgdataUtilTests.test_find_path (subunit.RemotedTestCase) 2018-12-04 03:30:56,033 - oe-selftest - INFO - ---------------------------------------------------------------------- 2018-12-04 03:30:56,033 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): [...] AssertionError: 'Previous bitbake instance shutting down?,[77 chars]xist' != 'ERROR: Unable to find any package produci[14 chars]xist' - Previous bitbake instance shutting down?, waiting to retry... ERROR: Unable to find any package producing path /not/exist We need to use the logger so output is correctly handled in such cases. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Flush key output to logsRichard Purdie9 days1-0/+4
| | | | | | | | Small tweak to ensure these items are printed into the log since there is other logging code which looks for the header and this makes it clearer the server did start but is slow somewhere in startup. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Handle short readsRichard Purdie9 days1-3/+3
| | | | | | | Its possible the read may return a smaller number of characters. Remove the possibility by using a single character to signal the server is ready. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Make lockfile handling clearerRichard Purdie9 days1-4/+6
| | | | | | | This simplifies the code and makes it easier to read but has the same functionality. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Check tab indentation for python codeRobert Yang13 days2-2/+11
| | | | | | | | | | | | | | | | The previous check was in data.py which only can check code like "python funcname()" in the dependency chain, but there are 3 kinds of python functions: - python() - def py_funcname() - python funcname() Add the checking to BBHandler to check and warn for all of them. The warning looks like: WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75 Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/svn: Fix SVN repository concurrent update raceUlf Magnusson13 days1-28/+36
| | | | | | | | | | | | | | | | | | | | | | | The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN repositories from which tarballs are generated. These repositories were protected from concurrent update with a lock on the tarballs. However, the tarballs are specific to the SRCREV and module checked out (many tarballs can come from the same repository), meaning a repository could be modified concurrently if two recipes checked out two different SRCREVs or modules from it in parallel. This caused errors like the following: ERROR: Fetcher failure: Fetch command failed with exit code 1, output: svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details) svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked. svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked. Fix it by adding a per-repository lock that's independent of the module and SRCREV. Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de> Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: Ensure disk monitor is started when no setscene tasks are runRichard Purdie13 days1-5/+6
| | | | | | | | | Currently if there are no setscene tasks, the disk monitor isn't started. Move the startup code to somewhere to ensure it always is started. This issue would partially explain occasional selftest failures. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Fix ConnectionRefusedError tracebacksRichard Purdie13 days1-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | Improve connetion refused error handling: NOTE: Retrying server connection... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 471, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 502, in connectProcessServer os.close(i) TypeError: an integer is required (got type NoneType) ) WARNING: /home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: Avoid warning about incorrect character escaping in regexRichard Purdie2018-11-271-1/+1
| | | | | | | | | Fixes: lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w re.compile("^\w+:(?!//)").match(uri): Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Show the last 60 log lines, not the last 10Richard Purdie2018-11-271-2/+2
| | | | | | | 10 log lines may not capture any full traceback, increase the number of lines to 60 which covers most tracebacks. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Show last 60 lines of the log if the server didn't startRichard Purdie2018-11-271-0/+6
| | | | | | | | We're seeing issues where the server doesn't start with no logs as to why. Allow the server to print the last 60 log lines just in case this shows us something useful about what is failing. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch: don't use shell=True when listing ar filesRoss Burton2018-11-231-1/+1
| | | | | Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* layerindex: don't use shell=True when cloningRoss Burton2018-11-231-1/+1
| | | | | Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/fetch: Update after recent gitsm message changesRichard Purdie2018-11-221-3/+2
| | | | | | Also use assetIn instead of assertTrue which aids debugging failures. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/git: fix AttributeError in shallow extraction logicChristopher Larson2018-11-221-5/+8
| | | | | | | | | | | | | This code checks to see if shallow is either disabled or the tarball is missing, but the else block tries to print the tarball filename, and this attribute doesn't exist at all when shallow is disabled. Handle the two cases separately to give sane errors for both cases without the exception: Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow' Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Fix multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2018-11-221-8/+1
| | | | | | | | | | | | | | | | | | | | | | | When multiconfig is enabled the cooker adds providers for all the targets to be built on all the multiconfig variables that were set, regardless if there is a dependency to it or not. This causes an issue when a certain target is incompatible with one or more of the multiconfigs, e.g. the target is not in COMPATIBLE_MACHINE for one of the MACHINEs being built, causing the cooker to error out since no providers can be found for that certain target on that multiconfig. This patch modifies the behavior to only look for PROVIDERS for a target on the multiconfig that was selected to be built, PROVIDERS are then looked for in other multiconfigs only when there is a defined dependency to them. [YOCTO #12985] Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/npm.py: Allow shrinkwrap resolved relative URL which startswith ↵Parthiban Nallathambi2018-11-201-1/+1
| | | | | | | | | | | | | | | 'http' (e.g http-proxy) shrinkwrap resolved relative URL can start with http. For example, "resolved: http-proxy/-/http-proxy-${PV}.tgz" is still relative URL to npm registry, but starts with http. Current if statement compares the startswith 'resolved' to 'http', which makes impossible to use npm download. Condtional comparison now strictly checks for "http://" and "https://" Signed-off-by: Parthiban Nallathambi <pn@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* BBHandler: Fix __python_func_regexp__ for comment linesRobert Yang2018-11-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - Add a comment in base.bbclass: def oe_import(d): import sys # Comment bbpath = d.getVar("BBPATH").split(":") [snip] Note, '# Comment' is started with '#', it is legal in python's syntax (though maybe not a good style), but bitbake reported errors: $ bitbake -p ERROR: ParseError at /path/to/base.bbclass:20: unparsed line: ' bbpath = d.getVar("BBPATH").split(":")' This error report would mislead people, the real problem is that '# Comment' is not supported, but it reports the next line, this may make it hard to debug the code are complicated. We can make __python_func_regexp__ handle '^#' to fix the problem, since it already can handle blank line "^$" in a python function, so it would be pretty safe to handle "^#" as well. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: print a message when no logfileRobert Yang2018-11-161-0/+4
| | | | | | | | | [YOCTO #12898] There might be no bitbake-cookerdaemon.log, print a message for debugging. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: fix filename for compile()Robert Yang2018-11-161-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: Add the following two lines to conf/local.conf: FOO = "${@foo = 5}" HOSTTOOLS += "${FOO}" * Before the patch $ bitbake -p Check the first lines of bitbake bitbake-cookerdaemon.log [snip] File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 125, in python_sub codeobj = compile(code.strip(), self.varname or "<expansion>", "eval") File "FOO", line 1 [snip] There isn't a file named 'FOO', but a variable name. * After the patch $ bitbake -p [snip] File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 129, in python_sub codeobj = compile(code.strip(), varname, "eval") File "Var <FOO>", line 1 foo = 5 Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data_smart: Add original traceback to ExpansionErrorRobert Yang2018-11-161-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can make it print clearer errors, for exmaple: Add Runtime_error to 'def oe_import(d)" 16 def oe_import(d): 17 import sys 18 Runtime_error [snip] * Before the patch: $ bitbake -p ERROR: Unable to parse /buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py Traceback (most recent call last): File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 430, in DataSmart.expandWithRefs(s='${@oe_import(d)}', varname='OE_IMPORTED[:=]'): except Exception as exc: > raise ExpansionError(varname, s, exc) from exc bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined This error message has two problems: - "Unable to parse data_smart.py": This isn't the real cause. - It pionts to "raise ExpansionError(varname, s, exc) from exc" which isn't clear enough. * After the patch: $ bitbake -p ERROR: Unable to parse OE_IMPORTED[:=] Traceback (most recent call last): File "OE_IMPORTED[:=]", line 1, in <module> File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 18, in oe_import(d=<bb.data_smart.DataSmart object at 0x7f9257e7a0b8>): import sys > Runtime_error bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined This one is more clearer than before. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* parse/ast: fix line number for anonymous functionRobert Yang2018-11-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - Define an error anonymous function in base.bbclass: 15 16 python() { 17 Compile error 18 } $ bitbake -p ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18: The code lines resulting in this error were: 0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d): *** 0002: Compile error 0003: SyntaxError: invalid syntax (base.bbclass, line 18) The lineno should be 17, but it reported 18, this would mislead people a lot when there more lines. - Now fix it to: ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 17: The code lines resulting in this error were: 0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d): *** 0002: Compile error 0003: SyntaxError: invalid syntax (base.bbclass, line 17) This is because the anonymous function is constructed by: text = "def %s(d):\n" % (funcname) + text The len(self.body) doesn't include the "def " line, the length of the function should be "len(self.body) + 1", so we need pass "self.lineno - (len(self.body) + 1)" which is the same as 'self.lineno - len(self.body) - 1' to bb.methodpool.insert_method() as we already had done to named function. Otherwise, the lineno is wrong, and would cause other problems such as report which line is wrong, but the line is not what we want since it reports incorrect line. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: better_compile(): Fix line number when report errorsRobert Yang2018-11-161-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - Add an error line in base.bbclass, e.g.: 15 16 def oe_import(d): 17 import sys 18 Compile error 19 bbpath = d.getVar("BBPATH").split(":") [snip] Note the "Compile error" line, I added it for reporting errors. $ bitbake -p ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 15: The code lines resulting in this error were: 0014: import oe.data 0015: for toimport in oe.data.typed_value("OE_IMPORTS", d): 0016: imported = __import__(toimport) 0017: inject(toimport.split(".", 1)[0], imported) *** 0018: 0019: return "" 0020: SyntaxError: invalid syntax (base.bbclass, line 18) There are 2 problems: - The "line 15" is incorrect, it is a blank line, not the error line. - The "*** 0018" points to incorrect position. These two problems would mislead people a lot sometimes. - Now fix it to: $ bitbake -p ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18: The code lines resulting in this error were: 0001:def oe_import(d): 0002: import sys *** 0003: Compile error 0004: bbpath = d.getVar("BBPATH").split(":") [snip] SyntaxError: invalid syntax (base.bbclass, line 18) Please see comments in the code for more details on how it is fixed. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* siggen: Adapt colors used by bitbake-diffsigs to support light themesPeter Kjellerstedt2018-11-161-4/+4
| | | | | | | | | | | | | | | The colors specified for use with bitbake-diffsigs were adapted for a dark theme, e.g., by setting the background color to black, which made it look very bad when used with a light theme. To make it look good both with a dark or a light theme, it is better to drop the background color. It is also better to leave out the color altogether for the title and just use bold. Finally, dropping bold for the red and green texts indicating removed/added values better matches other colorized diff implementations as, e.g., git diff. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: Avoid regex value escape warningsRichard Purdie2018-11-131-5/+5
| | | | | | | | | | Avoid warnings such as: bitbake/lib/bb/utils.py:72: DeprecationWarning: invalid escape sequence \d numeric_regexp = re.compile('^(\d+)(.*)$') bitbake/lib/bb/utils.py:1165: DeprecationWarning: invalid escape sequence \( var_res[var] = re.compile('^(%s%s)[ \\t]*\([ \\t]*\)[ \\t]*{' % (var[:-2].rstrip(), override_re)) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: Avoid warnings about deprecated imp moduleRichard Purdie2018-11-131-8/+6
| | | | | | | | | The imp module is deprecated, port the code over to use importlib. bitbake/lib/bb/utils.py:30: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/bb/server: Avoid UnboundLocalError tracebackRichard Purdie2018-11-131-3/+3
| | | | | | | | | | | Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer if command_chan_recv: UnboundLocalError: local variable 'command_chan_recv' referenced before assignment Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process: Fix unclosed socket warnings upon server connection refusedRichard Purdie2018-11-061-8/+7
| | | | | | | | | | | | | | | | | | | Extend the server error handling to avoid: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 457, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused ) WARNING: /home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) NOTE: Starting bitbake server... Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* siggen: Correct format argument substitutionMike Crowe2018-11-051-1/+1
| | | | | | | | Use % operator to ensure that the dependency is substituted into the error message correctly. Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>