summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2020-03-12knotty: Treat verbconsole as a console outputJoshua Watt
The BitBake.verbconsole needs to be treated like a console output logger (meaning that the TerminalFilter attaches an InteractConsoleLogFilter to it), even if it's not directly attached to the root 'BitBake' logger. First, assign a special "is_console" property to the relevant handlers, then look for the property in the handlers from the configuration object return by bb.msg.setLoggingConfig(). Finally, pass the list of all handlers to the TerminalFilter object; it doesn't care about the difference between console and errconsole, so pass all the relevant handlers as a list. This fixes cases where the console output was corrupted when messages were sent to the 'BitBake.verbconsole' handler. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12msg: Return config objectJoshua Watt
Returns the configuration object from setLoggingConfig(). This object has a config dictionary that contains all of the created handlers, filters and loggers, which makes it much easier to pull out items with specific names. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Update hash equivalence loggingJoshua Watt
Updates hash equivalence logging so that the interesting VERBOSE messages are always logged to the consolelog file so that issues in individual user builds can be diagnosed. The autobuilder logging config then updates this so that they also are shown on stdout, since the consolelog file is not capture there. In order to facilitate this, 2 new logging handlers were added, "BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these handlers are attached to anything by default, but they will log any messages that wouldn't otherwise be logged by the normal console or consolelog handlers. Users can attach whatever loggers the desire to this handler to get them to appear on the console or in the consolelog, as demonstrated by the autobuilderlog.json file. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12contrib: Add autobuilder logging configurationJoshua Watt
Adds a configuration file that the autobuilder can use to capture interesting logging domains above the ones that show up for normal users on stdout/stderr. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12runqueue: Lower setscene complete logging levelJoshua Watt
Lowers the level of the log message when setscene tasks have completed. This message can occur multiple times when hash equivalence is enabled, since the runqueue switches between executing setscene tasks and normal tasks. Since this is primarily of use when debugging hash equivalence, use the hash equivalence logger at VERBOSE level. [YOCTO #13813] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12msg: Add helper to merge logging configsJoshua Watt
Adds a function that can be used by UI front ends to merge logging configuration fragments. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Add logging cleanupJoshua Watt
Adds code to close all loggers when bitbake exits. This prevents unclosed file ResourceWarnings. A form of this closing existed previously, but was removed in the new logging code. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12bitbake-user-manual: Add documentation for BB_LOGCONFIGJoshua Watt
Adds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12runqueue/siggen: Lower hash equivalence loggingJoshua Watt
Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12runqueue/siggen: Log hash equivalence with a different loggerJoshua Watt
Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12bitbake-worker: Remove unnecessary addDefaultLogFilterJoshua Watt
Adding the default log filter here is unnecessary because there are no defined logging domains when it is called, which means it does no actual filtering. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Setup logs with config helperJoshua Watt
Sets up logging in knotty to use python's structured logging config and the bb.msg.setLoggingConfig() helper to setup logging. This allows the user to specify additional logging mechanism in a config file with BB_LOGCONFIG Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Remove dependency on format variableJoshua Watt
Passing around the log formatter variable was unnecessary since the log levels of interest can be accesses as class members of bb.msg.BBLogFormatter. Switching to do this will make using the structured python logging much easier, since it can be difficult to extract out the formatter for a specific handler. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Add helper to set logging configJoshua Watt
Adds a helper function to setup the structure logging information in bitbake. The helper function takes a default configuration and an optional path to a user config file that can be merged into the default. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Remove unused filtersJoshua Watt
Now that the filter are described using the python logging structure, these classes are no longer needed. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Add filter utilitiesJoshua Watt
Adds generic filter utilities that can be used to filter when the log level is greater than or equal to a defined value, or below a defined value, as well as a generic function to translate a string to a logging level (or bitbake logging level) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Add commented logging_tree codeJoshua Watt
Adds a comment section that can be easily uncommented to enable dumping the logging tree. This module is extremely useful for debugging issued with logging configuration Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Add repr for BBLogFormatterJoshua Watt
Adds a __repr__ function for BBLogFormatter. This allows it to get a human readable string when printed using the logging_tree module Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Use log level instead of debug countJoshua Watt
Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12knotty: Handle logging messages with specific loggerJoshua Watt
Handles the log messages from the bitbake server with the specific logger that the event originated from. This allows hierarchical logging configurations to work as expected. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12lib/bb/msg: Convert default domains to a dictionaryJoshua Watt
Converts the default domain variable to a dictionary where the keys are the logging domains and the values are the logging level (instead of the debug count). This makes it easier to deal with the logging domains and the awkward conversion from a list to a dictionary only needs to be done once when logging is initialized. Finally, other code has been written that already assumes this variable is a dictionary, see: f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker") Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-11tests: Add test for gitsm fetcher with shallow mirror tarballsPaul Barker
Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-11fetch2/gitsm: Unpack shallow mirror tarballsPaul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked temporarily so that the .gitmodules file can be examined. Signed-off-by: Paul Barker <pbarker@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06build.py: augment TaskBase to capture PN and PVChris Laplante
PF is already captured, but it's a pain to try to parse out PN and PV, which can be helpful to have available. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06ui/teamcity: add a TeamCity service message frontendChris Laplante
Exposes build status via TeamCity service messages, see https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06runqueue: teach runonly/runall to accept "do_task" as well as "task"Chris Laplante
Previously --runonly=do_task would give a misleading error like: ERROR: Could not find any tasks with the tasknames ['do_task'] to run within the recipes of the taskgraphs of the targets... The problem is that BitBake tried to find "do_do_task". So teach it to only add the do_ prefix if it's not already there. Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06bblayers: query: Add multiconfig optionJoshua Watt
Adds an option to the show-recipes subcommand that allows the user to specify which multiconfig should be shown. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06tinfoil: Add multiconfig supportJoshua Watt
Adds support for the Tinfoil cache adaptor to be bound to a specific multiconfig and invoke the appropriate commands for that multiconfig instead of the default. The cooker adapter now creates a cache adapter for each multiconfig specified in BBMULTICONFIG so that each multiconfig is present. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06cooker: Respect multiconfig parameterJoshua Watt
The cooker had a multiconfig parameter for the findProviders() and findBestProviders() API, but it was being ignored. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06command: Add mc parameter to findProviders commandJoshua Watt
Adds a multiconfig selection parameter to the findProviders command. This allows a client to find the providers for a specific multiconfig instead of the base configuration. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-02cooker: Reset parse status unpon clientCompleteRichard Purdie
If for example a tinfoil connection edits the datastore, a subsequent connection can be "corrupted" by those changes. By setting the parse status of the caches as False at exit, the behaviour becomes the same as a newly setup server as a new data store is setup. This avoids problems in tests when BB_SERVER_TIMEOUT is set as the server is properly reset between connections. [YOCTO #13812] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-02fetch2: Limit shown checksums to sha256Richard Purdie
Currently bitbake will list many checksums for a recipe when none are present, encouraging users to add them all to a recipe. We don't need/want them all. We used to show md5 and sha256 but given the concerns about md5, switch to showing just sha256 going forward which seems like the sensible one to standardise upon. There will be no change to existing recipe functionality. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-22lib/bb/utils.py: Preserve ownership of symlinkDaisuke Yamane
Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-20fetch2/svn: Avoid UnboundLocalError exceptionRichard Purdie
The update codepath would trigger: Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment Fix this so the code functions as intended in both fetch and update cases. [YOCTO #13798] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19layerindex: allow clones to be shallowJan-Simon Moeller
When bitbake-layers fetch-layerindex clones the repositories, these are full clones. Allow the user to specify '-s' and do shallow clones instead for faster downloads. Signed-off-by: Jan-Simon Moeller <dl9pf@gmx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: svn: care for path_specJens Rehsack
Documentation says: "path_spec": A specific directory in which to checkout the specified svn module. but existing svn fetcher uses "module" always as path of checked out svn-module, regardless whether path_spec was given or not. Signed-off-by: Jens Rehsack <sno@NetBSD.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19event: Remove duplicated items from close matchesRobert Yang
It printed duplicated ones when there are multiple similar recipes in differrent layers, for example, if python-lockfile in different layers, and there is no python3-lockfile: $ bitbake python3-lockfile ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches: python-lockfile python-lockfile python3-aiofiles Remove the duplicated ones to fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19cooker: Reset loghandlerRichard Purdie
When parsing, reset the loghandler when finished, else the messages can be misleading. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19data: Don't allow renameVar calls with equivalent keysKyle Russell
While usually a programming error, the behavior can cause a Parser instance to eventually gobble up a significant amount of memory, greatly affecting system performance. Try to avoid getting into that situation and alert the user about what they attempted to do. Signed-off-by: Kyle Russell <bkylerussell@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache"Peter Kjellerstedt
Mark any keys used to cache the srcrevs for a recipe as "dontcache" if BB_DONT_CACHE is set for the recipe. Remove any such keys upon the next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will make sure the srcrev is updated as expected if ${AUTOREV} is used. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19fetch2: Make fetcher_compare_revisions() workPeter Kjellerstedt
This seems to have been broken for a very long time. Now it also works regardless of BB_SRCREV_POLICY. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19knotty: Make the bb.command.CommandExit event terminate bitbakePeter Kjellerstedt
This matches the other bb.command.Command* events and without it, running `bitbake --revisions-changed` will hang indefinitely if there are changed revisions. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-17cooker/siggen: Empty siggen cache during parsingRichard Purdie
When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-17doc: correct typo of 'BitBack'Robert P. J. Day
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15bitbake: layerindex: use branch when specifiedJon Mason
When currently specified, the branch is used to verify the versioning of the meta layer, but the master branch is checked out. This change allows for the branch to be specified. Now it is easy to specify all of the meta layers being added are of the same version, without having to do it in each individual git tree. Also, it will error if there are branches without a matching version. Finally, this allows for meta layer git trees without a master branch. Signed-off-by: Jon Mason <jdmason@kudzu.us> Minor rework of the patch to use the layerBranch actual_branch since the layerindex referenced branch may be different then the overall release branch. Also adjust the patch to use the default git checkout branch instead of master if no branch was specified. (Some repositories don't have a master branch.) Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08doc: minor tweaks to ch 1 of BB user manualRobert P. J. Day
Tweaks include: - hyphenation - rewording for brevity or clarification - adding <firstterm> markup where appropriate Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08doc: Standardize spelling on 'BitBake' throughout docsRobert P. J. Day
Since the proper spelling is, in fact, 'BitBake', might as well make it consistent throughout the user manual. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Avoid cache mismatch issues with locked sigsRichard Purdie
If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Cache unihash values to avoid cache lookupRichard Purdie
Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08siggen: Optimise get_unihash disk based cache handlingRichard Purdie
Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-30 23:16:07 +0000