| Age | Commit message (Collapse) | Author |
|---|
|
Now that all of the functions in cve-check open the database read-only,
we can remove this lockfile.
This means cve-check can run in parallal again, improving runtimes
massively.
This reverts commit d55fbf4779483d2cfd71df78d0f733b599fef739.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
All of the function in cve-check should open the database read-only, as
the only writer is the fetch task in cve-update-db. However,
get_cve_info() was failing to do this, which might be causing locking
issues with sqlite.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Create directory of the CVE_CHECK_MANIFEST variable before copy to it,
so that the variable can use an arbitrary directory name.
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
this should prevent running into the very rare error
sqlite3.OperationalError: attempt to write a readonly database
As highlighted by https://www.sqlite.org/faq.html#q5
it is likely that the adapter won't allow use multiple exec calls
at the same time.
So it's best to prevent multiple accesses at a time, by reusing
the already in place CVE_CHECK_DB_FILE_LOCK
YOCTO #14110
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Moving the function will allow other classes to capture which CVEs have
been patched, in particular SBoM generation.
Also add a function to capture the CPE ID from the CVE Product and
Version
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This variable has been deprecated since Yocto Project version 3.0.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The old URL schema
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-ID
now redirects to
https://nvd.nist.gov/vuln/detail/CVE-ID
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This implements various fixes in comments in cve-check.bbclass
In particular, the "whitlisted" typo is important as the "whitelisted"
word is going to be replaced in a near future.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are times when exluding or including a layer
may be desired. This provide the framwork for that via
two variables. The default is all layers in bblayers.
CVE_CHECK_LAYER_INCLUDELIST
CVE_CHECK_LAYER_EXCLUDELIST
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Lets include whcih layer a package belongs to and
add it to the cve logs
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
add CVE_VERSION_SUFFIX to indicate the version suffix type, currently
works in two value, "alphabetical" if the version string uses single
alphabetical character suffix as incremental release, blank to not
consider the unidentified suffixes. This can be expand when more suffix
pattern identified.
refactor cve_check.Version class to use functools and add parameter to
handle suffix condition.
Also update testcases to cover new changes.
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The way distutils.version.LooseVersion compare version are tricky, it treat
all these ( "1.0-beta2", "1.0-rc1", "1.0A", "1.0p2" and "1.0pre1") as greater
version than "1.0". This might be right for "1.0A" and "1.0p1" but not for
the rest, also these version could be confusing, the "p" in "1.0p1" can be
"pre" or "patched" version or even other meaning.
Replace Looseversion with custom class, it uses regex to capture common
version format like "1.1.1" or tag format using date like "2020-12-12" as
release section, check for following known string/tags ( beta, rc, pre, dev,
alpha, preview) as pre-release section, any other trailing characters
are difficult to understand/define so ignore them. Compare release
section and pre-release section saperately.
included selftest for the version class.
[YOCTO#14127]
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The output currently shows the remapped product and version fields,
which may not be the actual recipe name/version. As this report is about
recipes, use the real values.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
patched CVEs
Default behavior is not changed. To suppress patched CVEs, set:
CVE_CHECK_REPORT_PATCHED = ""
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
per-recipe check file
The addition of this variable also makes it possible to change the
output suffix of the check files, e.g. in local.conf:
CVE_CHECK_MANIFEST_append = ".txt"
CVE_CHECK_RECIPE_FILE_append = ".txt"
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of inventing a new task to fetch the CVE data, use the existing
fetch task.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For example, if you just run 'bitbake cve-update-db-native' in a clean
build system, |cve_tmp_file| won't exist yet.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
multiconfigs which led to a race condition wherein multiple
cve-update-db-native:do_populate_cve_db tasks could attempt to write to
the same sqlite database. This led to the following task failure:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_populate_cve_db(d)
0003:
File: '/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 103, function: do_populate_cve_db
0099: if year == date.today().year:
0100: cve_f.write('CVE database update : %s\n\n' % date.today())
0101:
0102: cve_f.close()
*** 0103: conn.commit()
0104: conn.close()
0105:}
0106:
0107:def initialize_db(c):
Exception: sqlite3.OperationalError: disk I/O error
Use a lockfile to ensure multiple tasks don't step over each other.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The cve-check file should be saved always, it has good info.
Put a copy in the log dir as cve-summary with symlinks to latest run.
[Yocto #13974]
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the generated cve.log files, include the epoch in the product
version. This better matches how versions are displayed elsewhere,
in particular the bb.warn("Found unpatched CVE...") that appears
on the terminal when CVEs are found.
Signed-off-by: Ralph Siemsen <ralph.siemsen@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Certain recipes e.g. bash readline ( from meta-gplv2 ) download patches instead of having them in
metadata, this could fail cve_check
ERROR: readline-5.2-r9 do_cve_check: File Not found: qemuarm/build/../downloads/readline52-001
This patch ensures that download is done before running CVE scan, even
though these will be external patches and may not contain CVE tags as it
expects, but it will fix the run failures as seen above
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
CPE version could be '-' to mean no version info.
Current cve_check treat it as not valid and does not report these
CVE but some of these could be a valid vulnerabilities.
Since non-valid CVE can be whitelisted, so treat '-' as all version
and report all these CVE to capture possible vulnerabilities.
Non-valid CVE to be whitelisted separately.
[YOCTO #13617]
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
fix below error for whitelisted recipe and recipe skip cve check.
Error:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_cve_check(d)
0003:
File: '/poky-master/meta/classes/cve-check.bbclass', lineno: 59, function: do_cve_check
0055: try:
0056: patched_cves = get_patches_cves(d)
0057: except FileNotFoundError:
0058: bb.fatal("Failure in searching patches")
*** 0059: whitelisted, patched, unpatched = check_cves(d, patched_cves)
0060: if patched or unpatched:
0061: cve_data = get_cve_info(d, patched + unpatched)
0062: cve_write_data(d, patched, unpatched, whitelisted, cve_data)
0063: else:
Exception: ValueError: not enough values to unpack (expected 3, got 2)
Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
change whitelisted CVE status from "Patched" to "Whitelisted".
[Yocto #13687]
Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With out these changes, a traceback displayed when a file
is listed in the SRC_URI but the file does not exist.
raise FileNotFoundError and print the patch then mark the task as failed.
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Switch to recently released version 1.1 of NVD CVE JSON feed, as in
https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release
it is mentioned that
Due to changes required to support CVSS v3.1 scoring, the JSON
vulnerability feeds must be modified. This will require the consumers
of this data to update their internal processes. We will be providing
the JSON 1.1 schema on the data feeds page and the information below
to prepare for this transition.
...
The JSON 1.1 data feeds will be available on September 9th, 2019. At
that time the current JSON 1.0 data feeds will no longer available.
This change was tested briefly by issuing 'bitbake core-image-minimal'
with 'cve-check.bbclass' inherited via local.conf, and then comparing
the content between the resulting two
'DEPLOY_DIR_IMAGE/core-image-minimal-qemux86.cve' files, which did not
seem to contain any other change, except total of 167 entries like
CVSS v3 BASE SCORE: 0.0
were replaced with similar 'CVSS v3 BASE SCORE:' entries which had
scores that were greater than '0.0' (up to '9.8').
Signed-off-by: Niko Mauno <niko.mauno@iki.fi>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This code used to construct a single SQL statement that fetched the NVD data for
every CVE requested. For recipes such as the kernel where there are over 2000
CVEs to report this can hit the variable count limit and the query fails with
"sqlite3.OperationalError: too many SQL variables". The default limit is 999
variables, but some distributions such as Debian set the default to 250000.
As the NVD table has an index on the ID column, whilst requesting the data
CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time
different is insignificant: 0.05s verses 0.01s on my machine.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Remove obsolete Python 2 code, and use convenience methods for neatness.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A previous optimisation was premature and resulted in false-negatives in the report.
Rewrite the checking algorithm to first get the list of potential CVEs by
vendor:product, then iterate through every matching CPE for that CVE to
determine if the bounds match or not. By doing this in two stages we can know
if we've checked every CPE, instead of accidentally breaking out of the scan too
early.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The patch scanner works with patch files in the layer, not in the workdir, so it
doesn't need to unpack.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
CVEs that are whitelisted or were not vulnerable when there are version
comparisons were not included in the report, so alter the logic to ensure that
all relevant CVEs are in the report for completeness.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
djb2 hash algorithm was found to do collisions, so the database was
sometime missing data. Remove this hash mechanism, clear and populate
elements from scratch in PRODUCTS table if the current year needs an
update.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
CVE_CHECK_WHITELIST does not contain version anymore, as it was not
used. This variable should be set per recipe.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some product names are too vague to be searched without also matching the
vendor, for example Flex could be the parser compiler we ship, or Adobe Flex, or
Apache Flex, or IBM Flex.
If entries in CVE_PRODUCT contain a colon then split it as vendor:product to improve the search.
Also don't use .format() to construct SQL as that can lead to security
issues. Instead, use ? placeholders and lets sqlite3 handle the escaping.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes build failure with core-image-minimal:
Exception: UnboundLocalError: local variable 'to_append' referenced before assignment
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As glibc will be scanned for CVEs, we don't need to scan glibc-locale,
glibc-mtrace, and glibc-scripts which are all separate recipes for technical
reasons.
Exclude the recipes by setting CVE_PRODUCT in the recipe, instead of using the
global whitelist.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
CVE-2014-2524 is a readline CVE that was fixed in 6.3patch3 onwards, but the
tooling wasn't able to detect this version. As we now ship readline 8 we don't
need to manually whitelist it, and if we did then the whitelisting should be in
the readline recipe.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that cve-update-db added CPE information to NVD database. We can
check for unpatched versions with operators '<', '<=', '>', and '>='.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
do_populate_cve_db is a native task.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the NVD url is not accessible, print a warning on top of the CVE
report, and continue. The database will not be fully updated, but
cve_check can still run on the previous database.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of generating a series of indexes via range(len(list)), just iterate the
list.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In the NVD json CVE feed, affected versions can be strictly matched to a
version, but they can also be matched with the operator '<='.
Add a new condition in the sqlite query to match affected versions that
are defined with the operator '<='. Then use LooseVersion to discard all
versions that are not relevant.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some rare cases (eg. curl recipe) the CVE_PRODUCT contains more than
one name.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the new update-cve-db recipe to update database.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are both "curl" and "libcurl" CPEs in NVD.
All "curl" CVEs are currently missing in the reports.
Hence, switch "CVE_PRODUCT" to a space separated list.
It is useful for recipes generating several packages,
that have different product names in NVD.
Signed-off-by: Grygorii Tertychnyi <gtertych@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Disable downloading of the vulnerability DB in do_check_cves() task.
When invoked in this task, cve-check-tool attempts re-download of the CVE DB
if the latter is older than certain threshold. While reasonable for a
stand-alone CVE checker, this behavior can cause errors in parallel builds
if the build time is longer than this threshold:
* Other tasks might be using the DB.
* Several packages can start the download of the same file at the same time.
This check is not really needed, as the DB has been downloaded by
cve_check_tool:do_populate_cve_db() which is a prerequisite of any do_build().
The DB will be at most (threshold + build_time) old.
Signed-off-by: Konstantin Shemyak <konstantin.shemyak@ge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Ross Burton
Richard Purdie
Stefan Herbrechtsmeier
Konrad Weihmann
Joshua Watt
Michael Opdenacker
akuster
Lee Chee Yang
Chris Laplante
Ralph Siemsen
Khem Raj
Niko Mauno
Ross Burton
Pierre Le Magourou
Mikko Rapeli
Grygorii Tertychnyi
Konstantin Shemyak