aboutsummaryrefslogtreecommitdiffstats
path: root/lib/bb/fetch/__init__.py
AgeCommit message (Collapse)Author
2012-03-03bitbake: Drop fetch v1, v2 provides a much better codebase to build fromRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-03bitbake: Convert getVar/getVarFlag(xxx, 1) -> (xxx, True)Richard Purdie
Using "1" with getVar is bad coding style and "True" is preferred. This patch is a sed over bitbake directory of the form: sed \ -e 's:\(\.getVar([^,()]*, \)1 *):\1True):g' \ -e 's:\(\.getVarFlag([^,()]*, [^,()]*, \)1 *):\1True):g' \ -i `grep -ril getVar *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27Update users of getVar/setVar to use the data store functions directlyRichard Purdie
(From Poky rev: affd0907253387cee0ba0c7e9f92bcc109895446) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-04-05persist_data: raise KeyError on missing elementsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-02-11persist_data: drop SQLData indirectionChris Larson
The common case (if not the only case) is to only use a single domain. The only reason SQLData exists is to make it easier to delete a domain. Yet, there's no need for us to delete a domain if SQLTable knows how to clear itself out. So, add clear() to the table and pass the domain to persist(). Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-02-10bb.fetch: add fetch version to distinguish bb.fetch and bb.fetch2Yu Ke
there is case that we need to distingush bb.fetch and bb.fetch2, and use different API for bb.fetch and bb.fetch2. so it is necessary to add version info for distinguish purpose (From Poky rev: f12e71484593039cd58b6e7fadd038b28b05d849) Signed-off-by: Yu Ke <ke.yu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-10Fetcher: break the "SRCREVINACTION" deadlockYu Ke
Current fetcher has annoying "SRCREVINACTION" deadlock, which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev(): get_srcrev()->setup_localpath()->srcrev_internal_helper() ->evaluate SRCREV->get_srcrev() current fetcher resolve the deadlock by introducing a "SRCREVINACTION" condition check. Althoguh it works, it is indeed not clean. This patch use antoehr idea to break the deadlock: break the dependency among SRCREV and get_srcrev(), i.e. assign a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet this keyword, it will check and set the latest revision to urldata.revision. get_srcrev later can use the urldata.revision for value evaluation(SRCPV etc). In this case, SRCREV no longer depends on get_srcrev, and there is not deadlock anymore. (From Poky rev: 1589a1172f9432aed1cc9ce006f68cddf3073774) Signed-off-by: Yu Ke <ke.yu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-09Export KRB5CCNAME variableJavier Martin
This allows fetching git repositories using Kerberos authentication. Signed-off-by: Javier Martin <javier.martin@vista-silicon.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-02-09Fix comparison with SRCREVINACTION constantJavier Martin
Use '==' instead of 'is', otherwise it will always return true since 'rev' and "SRCREVINACTION" are not the same object. Signed-off-by: Javier Martin <javier.martin@vista-silicon.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-01-11*: use utils.remove() some moreBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2011-01-10fetch/__init__.py: Add missing debug level argumentRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2010-12-29fetch: add SRC_URI checksumYu Ke
This patch adds per-recipe SRC_URI checksum verification. - SRC_URI format The format of SRC_URI checksum follow OE definition: 1. SRC_URI has single src SRC_URI = "http://some.domain/file.tar.gz" SRC_URI[md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" 2. SRC_URI has multiple src, every src need specify name SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \ http://some.domain/file2.tar.gz;name=name2 " SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" - SRC_URI checking invocation: the checksum checking is invoked in do_fetch phase, so it can be invoked manually by if recipes has no SRC_URI checksum item, bitbake will show warning: " WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072" SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768" " thus recipe author can add it to recpie file after SRC_URI - control variable BB_STRICT_CHECKSUM when SRC_URI checksum is missing, this variable decide pass or not if BB_STRICT_CHECKSUM = "1", missing checksum is fatal Signed-off-by: Yu Ke <ke.yu@intel.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-12-29fetch: Ensure SRCREV is still set correctly if the pn-X override has been ↵Richard Purdie
expanded Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-29fetch: Fix named SRCREVS variable name orderingRichard Purdie
Retains backward compatibility. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-17Update persist_data usage to the new APIChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-12-10Rename command events, adjust compareRevisionsChris Larson
- Moved the logic for comparing revisions from cooker into command - Removed 'Cooker' from the event names - Renamed the 'ExitCode' event into CommandExit, and changed CommandFailed to be a subclass of CommandExit Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-12-06Merged the following poky fetcher fixes:Chris Larson
- bitbake/fetch: Allow checking of a single url at a time (massive sstate speed improvement) - bitbake/fetch: Fix handling of mirrors when checking for url validity - bitbake/fetch: When fetchers return errors, ensure any partial download is cleared - bitbake/fetch: Make URL checking slightly less verbose (distracting with the sstate code) - bitbake/fetch/git: Ensure fullclone repositories are fully fetched - bitbake/fetcher: Deal with a ton of different bugs - bitbake/fetch/git.py: Fix git fetcher to correctly use mirror tarballs - bitbake/fetch/__init__.py: Abstract mirror variable handling and enhance to accept \n delimitation in variables - bitbake/fetch: fix logic to prevent fetches when the file already exists - bitbake/fetch: ensure the go() method completes when not using premirrors - Export ALL_PROXY for bitbake commands - bitbake: Add proxy variables to standard export list - bitbake/fetch/git: fix try_premirror() definition .Method takes 4 parameters. - bitbake/fetch: fix by moving try_premirror() to the right place - bitbake/fetch: add try_premirror method and special case git fetcher - fetch: fix setting of localpath in SRC_URI parameters - bitbake/fetch: ensure the mirrored repository is updated as required - bitbake/git: define a forcefetch method - bitbake/fetch: Respect forcefetch even when pulling from a mirror - bitbake/git.py: Make sure different branches can have different revisions without triggering build count increases - bitbake/git.py: Allow tracking of branches in SRC_URI without cloning for use with fullclone - bitbake/git.py: Make sure a full clone checkout always updates Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-19Revert "persist_data: cache connection and use cursor"Chris Larson
Caching the database connection can cause serious issues if it results in multiple processes (e.g. multiple tasks) simultaneously using the same connection. This reverts commit 8a6876752b90efd81d92f0947bfc9527d8260969. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-11-19fetch: be more pythonicBernhard Reutner-Fischer
no functional changes Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-11-19fetch: use os.path.joinBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-11-19fetch: add common helper _strip_leading_slashes()Bernhard Reutner-Fischer
Several fetcher need a way to strip leading slashes off a local path. This helper-function consolidates all such occurances. Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-11-19fetch: rename suppports_srcrev to supports_srcrevBernhard Reutner-Fischer
osc had it already spelled correctly?! Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-10-06fetch: if mirror fetching fails, ensure exception is raisedJoshua Lock
We catch any exception raised by the fetchers go() method and attempt to work around it by trying any (post) mirrors which are configured. However, should the mirrors fail the exception is lost and the fetch is assumed to have completed successfully. Instead, save the exception and if the local file does not exist after trying the mirrors re-raise the exception. Fixes poky [BUGID #362] Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-03Switch bitbake internals to use logging directly rather than bb.msgChris Larson
We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-08-03fetch: Only mark srcrev recipes as uncached when outside of SRCREV caching modeRichard Purdie
(From Poky rev: 36cfff8ed5b8a777a09cc8abcd8b01a1f297e404) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-26*: cosmetic whitespace removalBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-07-01bitbake decodeurl: fix the file:// url handlingNitin A Kamble
Without this patch decoding a url of this kind file://dir/filename gives path=/filename host=dir. With the patch it decodes as path=/dir/filename host="" Probably nobody stumbled on this issue yet because nobody used file:// urls with directory names in the path. (From Poky rev: c3682bf89bdf236a4b4d834069d947b6a1456791) Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-06-213.0 prepChris Larson
2010-06-21Apply some 2to3 refactoringsChris Larson
2010-06-10Start ditching fatal() calls in favor of raising appropriate exceptionsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-09Change wget fetcher to use the runfetchcmdRyan Phillips
* Fixes proxy support to honor standard proxy environment variables. * Quote environment variables Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04persist_data: cache connection and use cursorBernhard Reutner-Fischer
Store database connection to persistent database in fetcher. Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04fetch: commentary typo fix in fetcher_init()Bernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-06-04fetch: documentation typo fixBernhard Reutner-Fischer
s/initilize/initialize/ Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
2010-05-22fetch: Check for existence of tar file along with md5 signatureKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-12Kill unnecessary usages of the types moduleChris Larson
types.IntType -> int types.StringType -> basestring ... Also moves our ImmutableTypes tuple into our own namespace. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-10Add a missing import of print_functionChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Apply the 2to3 print function transformChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Formatting cleanupsChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Stop using functions via 'bb' that were moved from there to other modulesChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Drop doctest, as we aren't utilizing itChris Larson
We should add back in some proper unit testing. Doctest is rather limited, and we haven't actually made use of it since the original implementation of the datastore when the project started, as far as I'm aware. Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-03-24Fix 7 references to undefined variables, as spotted by pyflakesChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-03-23Expand PREMIRRORSEnrico Scholz
When not expanding PREMIRRORS, the functions fails/does not work correctly when PREMIRRORS is not a plain string (e.g. contains ${...} or a ${@...} statements). Signed-off-by: Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de> Signed-off-by: Chris Larson <clarson@kergoth.com>
2010-02-25Avoid unnecessary calls to keys() when iterating over dictionaries.Chris Larson
dict objects provide an __iter__ method for the iteration which gives you the keys, so calling keys directly is unnecessary, and isn't really a best practice. The only time you really need to call the keys is if there's a danger of the dict changing out from underneith you, either due to external forces or due to modification of the iterable in the loop. Iterations over os.environ are apparently subject to such changes, so they must continue to use keys(). As an aside, also switches a couple spots to using sorted() rather than creating a temporary list with keys() and sorting that. Signed-off-by: Chris Larson <clarson@mvista.com>
2010-02-24Add initial 'repo' fetcherTom Rini
Signed-off-by: Tom Rini <tom_rini@mentor.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-02-19Move encodeurl, decodeurl into bb.fetch.Chris Larson
Signed-off-by: Chris Larson <clarson@kergoth.com>
2010-02-19Move MalformedUrl, VarExpandError into appropriate modules.Chris Larson
Signed-off-by: Chris Larson <clarson@kergoth.com>
2010-02-15fix invalid method and logicJoshua Lock
The previous bitbake change included an invalid method call and inverted logic. (From Poky rev: 52cbb074235fc7b1bcd94bc1147c4607612fe701) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-02-15Test premirrors and mirrors in checkstatus()Joshua Lock
checkstatus() is used to ensure we can fetch a copy of each file, so it makes sense to also test PREMIRRORS and MIRRORS in the method. This patch adds calls to try_mirrors() to the Fetch.checkstatus() method and changes the try_mirrors() method to take a check argument, which is False by default. When check is True try_mirrors() will call a fetchers checkstatus() with the replaced uri. (From Poky rev: ec92133d6fc049b4b666d670d3ac263de5a1d103) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-02-15Fix try_mirrors() logic, was only trying the first mirror.Joshua Lock
The logic in try_mirrors() was buggy such that only the first entry in the mirrors list was tried. This patch fixes this. Thanks to Richard for the review. (From Poky rev: c9fd126f64ec57cc0799ad03f32f509b5266e0ff) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-26 14:29:47 +0000