| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
The debug in the comments was out of date. It is still useful so update the
code sample to the new code needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
process.signal_received is a list for signum and not iterable,
change a suitable method to handle sig.
Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The runtaskdep data in siginfo files was written out with full paths to
the bb files, matching bitbake's internal "unique key" ID for recipes/tasks.
When originally implemented this made sense.
Over time, the main use for the data in siginfo files has become to match
against other siginfo files to debug changes of hash calcuations. The
recipename data is not useful for this as the siginfo filenames use PN
instead which can often be derived from the recipe filename but not always.
It is time to throw away the 'tid' data format and switch over the use a
hybrid PN form which includes the multiconfig. That can be easily stripped
off in the find_siginfo code in oe-core.
The other purpose of having a sortable dependency ID is retained and the
multiconfig needs to be included to allow the taskhashes to be processed
and calculated correctly. PN is meant to be unique between recipes, only
one would ever be built so using PN in this location is fine.
The one risk of this change is there isn't any compatibility to the old
format. I'm not convinced we should spend time complicating the code
with it. This change will change the taskhashes everywhere so the only
mixing of old and new siginfo files will be either through hash equivalence
or through users using the tool against old and new info files manually
which will give some weird output but it should be clear they're in
different formats as there would be large paths from the old files not
present in the new ones.
We have options to add backwards compatibility if some issue is found
to need that.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add one more verification that checks if localpath is None, because
we can't compute checksum of a None.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
fetch them
When there are no dependencies, _foreach_proxy_method does not verify that there are
dependencies to fetch before fetching them.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update test_python_snippet_function_reference to not require the 'func'
flag, now that we know the real function will be returned for context
functions without the flag.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
We previously checked for the existence of the 'func' flag to determine
if we should avoid looking up in the metadata. This was done to ensure
the user gets the function for 'def' python functions rather than their
string contents. We can sidestep the metadata lookup and check our
function context directly, instead.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
This test makes sure that '${@eval()}' calls the eval builtin, even if an 'eval' variable is defined in the metadata.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
This avoids the need to hardcode a list of python builtins. This also slightly changes behavior, in a case like `${@eval("3")}`, this will ensure we always call the builtin, even if the metadata has an 'eval' variable defined.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
This ensures that any change to the presence of builtins in inline
python execs will be noticed.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
This is a test for an issue seen long ago, to avoid regressions, where a
reference to a def'd function in the metadata would return the string
value from the metadata rather than the function in inline python.
Signed-off-by: Christopher Larson <kergoth@gmail.com>
|
|
Fixes a few errors when the bitbake indent plugin is reloaded:
1) Define functions with "!" so that vim doens't issue a warning when
they are replaced
2) Rename GetPythonIndent -> GetBBPythonIndent to prevent potential
conflict with other plugins
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #13550]
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Using multiconfig to target baremetal pieces of the system and building
corresponding toolchains for them results in hundreds and hundreds of
"Deferring %s after %s" and "Deferred task %s now buildable".
To clean up the output and to reduce risk of missing important warnings,
convert these notice messages to debug messages.
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is unclear from the server logs when parsing starts. We know that timeouts
sometimes happen when parsing but it is unclear where in the code the delays
are from. Adding this debug message to the server log should help narrow
that down.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To learn more about the server timeout issues, be clear in the error
message about which command is showing the timeout. It is currently
unclear if this is the original command or a ping to the server.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"fetch2/npmsw: Add support for the new format of the shrinkwrap file"
added support for the new format shrinkwrap files but this regressed
our tests which still use the old format.
Similar to how npm handles this, support both for now until we can
migrate our tests.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Packages like @(._.)/execute causes problems because they generate names
that are not supported by yocto
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Dev dependencies should not be fetched only if it is specified in the
recipe.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Npm is a package manager that has its own manner to handle installation of packages.
But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation.
The shrinkwrap file changed its format over npm versions, but npm does not version
this file, so we can use it properly.
The actual changes make NPM depencies work with the actual shrinkwrap format.
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To support running the bitbake self-tests using pytest, add a
pytest.ini This enables these tests to be run using the following
command from the bitbake directory.
pytest
For now, testpaths only includes the bitbake selftests.
Add a note to the README.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By default, pytest captures all stdout and exposes it using its built-in
fixtures (capsys, caplog etc), so stdout does not support getvalue().
To support running tests using both unittest and pytest, use assertLogs
to capture logging and assert on the log output instead.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
git config returns an error code if user.email or user.name are not set,
so bb.process.run raises an ExecutionError, failing the test.
Improve the logic by catching the ExectionError and using this to set a
default value.
Change a direct invocation of 'git init' to use self.git_init so that
it follows the code path above.
Remove the related comment from the README now that git sets up the user
details automatically.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python
3.2, so rename to fix a warning during test execution.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
pytest test collection attempts to collect all classes containing
"Test". TestTimeout is not a test class so add the __test__ = False
attribute to indicate this.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
highlights for variables
Some variables are described in the glossary so add a term role to the
references. For the others, highlight them by surrounding them with
two backticks.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Partially reverting https://git.openembedded.org/bitbake/commit/?id=d1779e2f7789e516efae434b3781c981d0ef4ded
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Document the new variable.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I'd tested target configurations but in real world use, it became clear
the layername functionality in worker context was failing for virtual
class extensions. Fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #15128]
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This addresses [YOCTO #15127]
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
- Move "langdale" link to outdated section
- Add link to "mickledore" manual
- Fix typo in "kirkstone" name
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Include the hashfn (the value of BB_HASHFILENAME) in the task dependency
data. This allows tasks to get a specific unique hash for dependent
tasks when one is available.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes a crash when the server process attempts to check the PID of the
lock file that resulted because an integer (os.getpid()) was attempting
to be concatenated to a string
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the user configures the TMPDIR variable in a way that it contains $HOME (in local.conf), the DataSmart (d) object's PATH variable will also contain $HOME.
As the NpmEnvironment patches HOME with a temporary directory, further evaluation of PATH will become invalid an the build breaks with an error (npm not found).
As countermeasure PATH is evaluated *before* patching HOME.
Signed-off-by: Arno Baumfalk <a.baumfalk@astro-kom.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.
Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.
The idea is that this would make layer overrides possible:
OVERRIDES .= ":layer-${FILE_LAYERNAME}"
which then opens possibilities like:
WARN_QA:append:layer-core = " patch-fuzz"
as an example where OE-Core could enable specific QA tests only for that
specific layer.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you have a host where the git config for initial branch
name is something else than 'master' the unittest will fail
as they assume the default branch name is 'master'. Fix this
by explicitly set the intial branch name at 'git init'.
Signed-off-by: Peter Bergin <peter.bergin@windriver.com>
Signed-off-by: Peter Bergin <peter@berginkonsult.se>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current debug lists all the potential search paths but not the end result
which isn't helpful for debugging. Tweak to include the end result.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To avoid confusion, rename the local fetcher localpaths function to
something more specific so it doesn't appear similar to the localpaths
top level function.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Code changes for FILE_LAYERNAME exposed lifecycle issues around the collections
object in Cooker which only appeared in devtool usage in eSDK. Move
the collections setup to an earlier stage after parsing completes to
avoid any kind of race around it.
Also stop overwriting the code variable in MatchFiles. Ultimately we
need to combine these codepaths but that is for another patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The crate fetcher handles a crate with a name that matches the recipe's
name specially by placing the unpacked code in the current directory
(which typically is ${S}) rather than together with the sources for the
other crates. This broke when the URI names for all crates were changed
recently to include the version in the name.
Correct the crate fetcher to test against ${BP} instead of ${BPN}.
Also add a test case to the selftests to avoid this breaking again.
[Yocto #15012]
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Martin Jansa
Richard Purdie
Chen Qi
Emil Ekmečić
Yang Xu
BELOUARGA Mohamed
Christopher Larson
Joshua Watt
Yoann Congal
Dit Kozmaj
Denys Dmytriyenko
Peter Hoyes
Quentin Schulz
Michael Opdenacker
Arno Baumfalk
Bergin, Peter
Peter Kjellerstedt