| Age | Commit message (Collapse) | Author |
|---|
|
Measure and display the time it takes to copy the rootfs to its new
location (usually in tmpfs) to try to understand whether it sometimes takes
a lot of time on the autobuilders.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since Linux 5.4 a new read-only filesystem is available, erofs.
Compared to squashfs it offers much better read performance with and
without compression enabled.
It suppports two optional compressors, lz4 and lz4hc.
>From the mkfs.erofs man page:
EROFS is a new enhanced lightweight linux read-only filesystem with
modern designs (eg. no buffer head, reduced metadata, inline
xattrs/data, etc.) for scenarios which need high-performance read-only
requirements, e.g. Android OS for smartphones and LIVECDs.
It also provides fixed-sized output compression support, which improves
storage density, keeps relatively higher compression ratios, which is
more useful to achieve high performance for embedded devices with
limited memory since it has unnoticable memory overhead and page cache
thrashing.
This commit adds support for three new filesystem targets:
erofs: erofs without compression
erofs-lz4: erofs with lz4 compresssion enabled
erofs-lz4hc: erofs with lz4hc compression enabled
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Each of the cores is mapped to a thread on the host, this
should speed up things inside qemu which can take advantage of that.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Dynamic layers may have a conf directory, but don't need (or
want) a conf/layer.conf
This isn't an error, so we can just log it and indicate the
layer is being skipped.
A full layer without a conf file isn't all that useable, so we
aren't letting anything subtle slip through by just logging
it.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.
[YOCTO #14301]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are many Python scripts in oe-core that want to use Tinfoil, and
right now they have to know where they are to work out where BitBake is
likely to be.
This is suboptimal as BitBake could be somewhere else, so this
approach doesn't scale to other layers at all.
Solve this by adding BITBAKEDIR/lib to PYTHONPATH in oe-buildenv-internal,
so that Python has BitBake on its search path once the build system is
configured.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When disk stats don't run frequenctly enough, we see divide by zero
errors. The code already has a fallback path so ensure we use it
for this case too.
[YOCTO #14360]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Collect the last 30 lines from the cooker.log
whenever the timeout is exceeded.
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If iterating a layer with multiple components and auto-adding dependencies
the tests can break since layers are never removed and order isn't guaranteed
to account for that.
Fix this by resetting the layer list back to the original list each time
before auto-adding the dependencies in each case.
This fixes scanning of meta-openembedded in particular where the sublayers
may not be added in order of minimal dependency.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to cleanup snapshot files if we make a copy of them to ensure
the tmpfs doesn't run out of space. There is already NFS code needing
this so make it a generic code path.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Particularly:
- nesting subprocess.run() inside subprocess.check_output() does not work at all.
How was this tested?
- -R and -U options can be combined; no need to separate the invocations based
on packaging format
- both exception handlers are unnecessary; we can simply print the hint if
invocation did not succeed
- to run debuginfod from its own sysroot, '-c addto_recipe_sysroot' for elfutils-native
must be executed
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the previous timeout of 5 seconds, there would be
builds such as:
https://autobuilder.yocto.io/pub/non-release/20210417-13/
which produced 17 files with top output with top running 454 times
and that's a bit too much data to analyze for each run. By
increasing the timeout, we'll find the worse problems
first, fix them and then we can decrease the timeout if needed.
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
image-manifest: script to generate product/image specific BOM
The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.
The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.
image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.
This script extends the oe-image-manifest script [0] done by Paul Eggleton
[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest
------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...
Image manifest utility
options:
-h, --help show this help message and exit
-d, --debug Enable debug output
-q, --quiet Print only errors
subcommands:
recipe-info Get recipe info
list-depends List dependencies
list-recipes List recipes producing packages within an image
list-packages List packages within an image
list-layers List included layers
dump-config Dump default config
manifest-info Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command
Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Improvements:
- increase width to 512
- pass -c option to show full command-line
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have a working theory that IO queues on the autobuilder are impacting
runtime testing under qemu, particularly async writes which inice does not
influence. We already pass the snapshot option to qemu which copies the
image and runs out of the copy. Add in the ability to copy the image to
a specificed location which can be a tmpfs. This means that writes to the
image would no longer be blocked by other writes to disk in the system.
Preliminary tests show that this does improve the qemu errors at the expense
of sometimes showing qemu startup timeouts as on a loaded system with a large
test image, it can take longer than 120s to copy the image to tmpfs. Having
a most consistent failure mode for loaded tests is probably desireable though.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some options such as egl-headless are fully compatible with it, so
there is no need to quit.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
oe-time-dd-test records how much time it takes to
write <count> number of kilobytes to the filesystem.
It also records the number of processes that are in
running (R), uninterruptible sleep (D) and interruptible
sleep (S) state from the output of "top" command.
The purporse of this script is to find which part of
the build system puts stress on the filesystem io and
log all the processes.
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
anonscm.debian.org is not used any more, update the URL.
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update after tinfoil API changes in bitake for REQUIRED_VERSION.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Pretty sure that trailing quote should be a period; it
appears to work properly.
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The do_kernel_configme task is no longer part of SRCTREECOVEREDTASKS,
its been removed from the kernel-yocto.bbclass since b72dbb2e4, but
there wasnt a matching patch for devtool for those changes.
This patch enables us to invoke the do_kernel_configme task when
using a devtool workspace, it also prepends a check for an existing
.config file in the source directory and moves it if thats the case,
since when using devtool modify a .config is created and
do_kernel_configme complains about it, this is not the case when
using bitbake since the .config file would be on B instead.
Alowing do_kernel_configme to run also fixes the flow where testing a
new config fragment from devtool workspace isnt added properly
(config queue shows it as ///frg.cfg) and as a side effect it never
gets merged into the final config.
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Passing 'novga' only adds '-vga none' to the qemu commandline and does
not prevent other vga devices from being added, contrary to the
documentation/help. Clean-up the vga logic and add the ability to
prevent other vga devices from being added.
Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I7ff8f9f3e419ce8bae3f3847d75f9275ca30406a
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some platforms do not use ttyS* for their serial consoles (e.g., qemuarm
and qemuarm64). The hardcoding of this can cause issues. Modify
runqemu to use the serial consoles defined in SERIAL_CONSOLES instead of
hardcoding.
Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I746d56de5669c955c5e29d3ded70c0a4d3171f17
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a new runqemu field for VGA devices. Currently, these are being set
in QB_OPT_APPEND, which can make them difficult to override if importing
the config file into another one.
Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I8cb9527954c5b06c083c42fe2466cb3338584b7d
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The sstate-diff-machines.sh script's --analyze option was only supporting
ipk packages, this adds rpm, deb and tar packages, checking all the used
package types simultaneously.
Without this changing sstate between machines would not report an error
if the difference occured in packages and ipk packaging was not enabled.
Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bin is not an acceptable format for qemu and will throw an error is set.
The correct format in this case is raw.
Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I9d78e356d39ecb15bef886c19870b029d9badd65
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Added a script that fetches the package manager directory and runs the elfutils-native debuginfod on DEPLOY_DIR
Added a check to ensure that PACKAGECONFIG options is set in local.conf
Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Modern dosfstools automatically determines the appropriate size
and will error out if something that doesn't make sense is
supplied on the command line.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We are getting closer and closer to the year 2038 where the 32 bit
time_t overflow will happen. While products (= embedded systems) with an
expected life time of 15 years are still save the situation may change
if your system has to survive the next 20 years.
ext2 and ext3 filesystems are always affected by the time overflow, so
let's warn the user if these filesystems are still being used.
If ext4 is affected depends on the inode size chosen during filesystem
creation. At least 256 bytes are necessary to be safe. As ext4 is
used very often (and partitions may be created small first and extended
later) this might be an issue for many users.
Some filesystems created during CI runs were already affected by the Y2038
problem. By using `--mkfs-extraopts "-T default"` we tell mke2fs not to
auto-detect the usage type based on the filesystem size. mke2fs will use
the default values for tuning parameters instead. The inode size is one
of these parameters.
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using nographic, explicitly disabling vga is needed some cases
since some qemu firmware (OpenFirmware ) defaults to std vga, and
when vga is enabled then it disables output to serial and redirects
that to vga which is by design, hwoever we expect the console output
to go to serial when using nographic, therefore its important to
disable vga with nographic especially on qemu based ppc platforms
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tinfoil doesn't behave well if environment is not initialized, this check ensures a proper error log if environment is not initialized.
[YOCTO #12096]
Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
files in wic tmp directory can be usefull for debugging, so do not remove
tmp directory when wic create run with debugging mode (-D or --debug).
also update wic.Wic.test_debug_short and wic.Wic.test_debug_long to
check for tmp directory.
[YOCTO#14216]
Signed-off-by: Lee Chee Yang <Chee.Yang.Lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add relative path if file is under directory.
[YOCTO #13738]
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.
[YOCTO #13036]
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.
We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.
This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.
An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.
Fixes [Yocto #13994]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.
To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Sinan Kaya <okaya@kernel.org>
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Before adding a npmsw fetcher to a recipe we
should first check if the generated shrinkwrap file
contains dependencies.
Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes a crash with "find" when running a native tool and *-native
directories do not exist under the binary directory in the sysroot.
This happened because the directory wildcard was passed as part of the
root directory.
The directory wildcard is now passed by "-name", which returns an empty
result if no matching directory.
Signed-off-by: Adrian Herrera <adrian.herrera@arm.com>
Change-Id: Iba7acd8bbd7e0beb4d25c984f6af7a4fd21486e6
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
vfat/msdos filesystem should have fsuuid in format 0xYYYYYYYY where "0x"
in front follow with 8 hexadecimal number in uppercase. In wic, when using
custom fsuuid for vfat/msdos partition in wks, it is able to set the value
in any length, with or without leading "0x". This can cause fsuuid
missaligned when fstab updates, fstab expect exactly 10 character
fsuuid for vfat/msdos partition and all in uppercase.
if custom fsuuid for vfat/msdos is set, check the length and format,
error if it exceed the format size. Amend it so it is align with format
0xYYYYYYYY. This is done before image create and fstab update to ensure the
fsuuid are same in all followup process. if custom fsuuid length less than
expected, fill in "0".
[YOCTO #14161]
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The empty wic plugin is used to create unformatted empty partitions for wic
images.
To use it you must pass "empty" as argument for the "--source" parameter in
the wks file. For example:
part foo --source empty --ondisk sda --size="1024" --align 1024
Also adds a selftest for this plugin where the 'Fstype' column from 'wic
ls' should be empty for the second partition as listed in
test_empty_plugin.wks.
Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add "payload_dir" parameter so user can add other files/dirs
to the ISO.
Signed-off-by: Chaitanya Vadrevu <chaitanya.vadrevu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Alexandre Belloni
Richard Weinberger
Alexander Kanavin
Bruce Ashfield
Devendra Tewari
Ross Burton
Richard Purdie
Sakib Sajal
Randy MacLeod
Anders Wallin
Khem Raj
Petr Vorel
Robert P. J. Day
Alejandro Hernandez Samaniego
Jon Mason
Diego Santa Cruz
Jon Mason
Dorinda
Florian Bezdeka
Lee Chee Yang
Tomasz Dziendzielski
Paul Barker
Sinan Kaya
Kamel Bouhara
Adrian Herrera
Peter Kjellerstedt
Diego Sueiro
Chaitanya Vadrevu