| Age | Commit message (Collapse) | Author |
|---|
|
Just utilize the checksum cache in bb.fetch2 as it is not used for
anything else.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
|
|
BB_ORIGENV value on the datastore can be NoneType thus raising an AttributeError
exception when calling the getVar method. To avoid this, a check is done before
accesing it.
[YOCTO #9567]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some users may want to use authenticated SSH connections with credentials stored
in a keyring, such as gnome-keyring. These typically need a DBus session bus
connection, so pass DBUS_SESSION_BUS_ADDRESS into the fetcher environment.
To avoid the user needing to set it in their local.conf (which wouldn't be
usable) or adding it to the environment-cleansing whitelist (which would
potentially impact builds) allow the variables being passed to the fetchers to
come from the data store (first) or the original environment (second).
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was used for workaround git 1.7.9.2 which was released in 2012 which
should not be existed on nowadays host, so remove it to avoid
confusions.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake would fail to classify the following URL as belonging to a
allowed network, because of the port number in the url.
BB_ALLOWED_NETWORKS = "*.example.com"
SRC_URI = "http://git.example.com:8080/foo.tar.gz"
Since protocols aren't specified in the BB_ALLOWED_NETWORKS variable,
it's reasonable to believe that this should work regardless of protocol
being used.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are some codepaths where the file checksum is verified and can
be found to mismatch but the 'rename' logic doesn't kick in. If code
relies on the presence of a file for the checksum having been checked
(e.g. uninative.bbclass) then it can be used when the checksum hasn't
matched.
Therefore rename the file whenever an invalid checksum is encountered.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When prefix is part of the version directory it need to ensure that
only version directory is used so remove previous directories if exists.
Example: pfx = '/dir1/dir2/v' and version = '2.5' the expected result
is 'v2.5' instead of '/dir1/dir2/v2.5'.
[YOCTO #8778]
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Little improvement for reference tokens by names instead of index.
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We recently dropped lockfiles for file:// urls which in itself makes
sense.
If a file url redirects to something like an http:// mirror, we'd have
no lock taken for the original file and could race against others
trying to download the file. We therefore need to ensure there is a
lock taken in the mirror handling code.
This adds code to take such a lock, assuming it isn't the same lock
as the parent url.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes a problem when the repository contains multiple levels of submodules via a resursive submodule init.
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.
Use with care.
[YOCTO #8853]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes [YOCTO #9231]
npm when given an invalid registry URL with --registry actually goes and
fetches from the default registry, but this commit makes sure it goes to the
specified one.
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For some reason, the values for SRC_URI[md5sum] and SRC_URI[sha256sum]
were not being expanded. That lead to the following code not working
as expected:
SRC_URI = "http://.../${PN}-${PV}.tar.gz"
MD5SUM = "123abc..."
SHA256SUM = "abcd1234..."
SRC_URI[md5sum] = "${MD5SUM}"
SRC_URI[sha256sum] = "${SHA256SUM}"
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this you get a rather odd traceback instead of the proper
exception message.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.
Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.
As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
No code changes, just fix to use four spaces.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
dist.tarball
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.
These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files
Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For some reason the enablement piece of the patch went missing, add it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.
This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.
This tweaks the code so that unless its a file url, the host is returned
correctly.
This patch also adds test cases for these urls to the exist set of test
urls.
We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.
This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.
The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"
To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Drop unused 'd' argument from the cache save methods, simplifying the
API.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In order to use in other modules since is a common function
when needs to get proxies working.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since people do copy and paste these things, clean up old syntax styles.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This shouldn't be in here, use a variable instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fix a problem when checking out a commit that changes the submodules
previously checkout.
Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The recent change to verify_checksum() to only show checksum warnings
if no checksums are supplied made it possible to simplify the logic a
bit more.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If one checksum is supplied to a SRC_URI, we really don't want to show
warnings about the other type which isn't present as one checksum
is really good enough for most cases.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.
It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.
|
|
This reverts commit e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c.
This is in fact a valid use case, for example the sstate.bbclass code
sets up SSTATE_MIRRORS as PREMIRRORS. Its quite common to map those
file:// urls to remote http:// urls and with the above change, this
no longer works.
|
|
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.
This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.
Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.
[YOCTO #8864]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:
SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The PREMIRROR isn't useful for "file://", so avoid using it, this is
good for searching speed and can reduce useless lines in log.do_fetch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For two reasons:
1) The important one: we hit the following bug when doing upstream version checks
on some webpages:
https://bugs.launchpad.net/beautifulsoup/+bug/1471755
2) Also, documentation for beautifulsoup states that memory usage and
speed is improved that way.
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This looks reasonable until you realise self.localpath is a function. Data
expansion of something which isn't a string is the original value so this
code just wastes CPU cycles and makes no sense. Remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Markus Lehtonen
Leonardo Sandoval
Ross Burton
Robert Yang
Olof Johansson
Richard Purdie
Aníbal Limón
Derek Straka
Brendan Le Foll
Peter Kjellerstedt
Paul Eggleton
Alexander Shashkevich
Aníbal Limón
Felipe F. Tonello
Kristian Amlie
Juro Bystricky
Christopher Larson
Alexander Kanavin