The OS-autoinst project aims at providing a means to run fully automated tests. Especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily and safely be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS.
os-autoinst can be executed alone, but is currently designed to be executed together with openQA, the web user interface that allows to run more than one os-autoinst instance at the same time.
More information on os-autoinst and openQA can be found on http://os-autoinst.github.io/openQA/
Under openSUSE the os-autoinst package can be installed from the official
repository or from our
devel repository.
For further details, have a look at the
openQA documentation.
For building os-autoinst manually checkout the build instructions below.
The main executable isotovideo can read test parameters from the command
line or read test parameters from a file named vars.json. This file stores
the values of the different variables that will configure the behavior of the
test execution.
A container is provided and can be pulled and the main execution can be called
in one step, for example using the podman container engine for tests defined
in the current directory on x86_64 if your environment supports KVM
virtualization acceleration:
podman run --rm -it -v .:/tests registry.opensuse.org/devel/openqa/containers/isotovideo:qemu-kvm casedir=/testsUse the image variant ending with qemu-x86 on x86_64 if no KVM support is
available.
Take a look on https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA for all available container images.
Additional test variables can be supplied on the command line. There are some variables used by os-autoinst itself and other that are used by the tests. A minimal command line can look like this:
isotovideo distri=opensuse casedir=/full/path/for/tests iso=/full/path/for/isoAs alternative or completementary a corresponding vars.json with additional
parameters could be:
{
"DISTRI" : "opensuse",
"CASEDIR" : "/full/path/for/tests",
"NAME" : "test-name",
"ISO" : "/full/path/for/iso",
"VNC" : "91",
"BACKEND" : "qemu",
"DESKTOP" : "kde"
}Be advised that the file vars.json is also modified by os-autoinst so make
sure to backup handcrafted versions of this file.
For more concrete instructions read on in the "How to run test cases" section below. Find sections about "How to contribute" or "Build instructions" further below.
This following instructions shows how to run test cases. First one needs to clone the test distribution. Checkout os-autoinst-distri-example for an example of a minimal test distribution.
Example for openSUSE’s tests:
mkdir distri && cd distri git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git opensuse cd opensuse/products/opensuse git clone git@github.com:os-autoinst/os-autoinst-needles-opensuse.git needles
Example for openQA’s self-tests ("openQA-in-openQA" test):
mkdir distri && cd distri git clone git@github.com:os-autoinst/os-autoinst-distri-openQA.git openqa cd openqa git clone git@github.com:os-autoinst/os-autoinst-needles-openQA.git needles
Then create a working directory for the test execution, e.g.:
mkdir /tmp/os-autoinst-run && cd /tmp/os-autoinst-run
Create a minimal vars.json config file within that directory, e.g.:
{
"ARCH" : "x86_64",
"BACKEND" : "qemu",
"CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
"DESKTOP" : "gnome",
"DISTRI" : "opensuse",
"ISO" : "/path/to/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso",
"PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/opensuse",
"VNC" : 90,
}You will need to correct the file paths to point to real locations. Some of the variables you can use are listed here. Test case specific variables are listed in the distri directories e.g. os-autoinst-distri-opensuse/variables.
Then you can run the isotovideo script within the created working directory.
When doing a manual build, that script can be found at the top-level of the
os-autoinst Git checkout.
All of these examples were using the QEMU backend which is usually the easiest backend to handle and therefore recommended. If you need to develop and test other backends, have a look at the backend-specific documentation.
When using the QEMU backend it is possible to access the system under test via VNC:
vncviewer localhost:91 -ViewOnly -Shared
Run isotovideo with the environment variable RUN_VNCVIEWER set to autostart
a VNC viewer on the right port.
Run isotovideo with the environment variable RUN_DEBUGVIEWER to start the
internal debug screenshot viewer updated with an always recent screenshot of the
test run.
Individual test modules are written with one test module per file using the test API in Perl code. Experimental support for test modules in the Python programming language is provided.
Find more details about how to write tests on http://open.qa/docs/#_how_to_write_tests
To check if your hardware is able to successfully execute os-autoinst based tests one can execute openQA tests, all the development tests or simply call something like
podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ test-perl-testsuite TESTS=t/99-full-stack.t'
which only requires the container runtime environment "podman" and will run a container based os-autoinst full-stack test, here without KVM hardware accelerated virtualization support.
If you want to contribute to this project, please clone and send pull requests via https://github.com/os-autoinst/os-autoinst.
More information on the contribution can be found on http://os-autoinst.github.io/openQA/contact/, too.
Issues are tracked on https://progress.opensuse.org/projects/openqav3/.
For an overview of the architecture, see doc/architecture.md.
-
Every commit is checked by our CI system as soon as you create a pull request but you should run the os-autoinst tests locally. Checkout the build instructions for further details.
-
For git commit messages use the rules stated on How to Write a Git Commit Message as a reference
-
Every pull request is reviewed in a peer review to give feedback on possible implications and how we can help each other to improve
If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal.
On openSUSE one can install the package os-autoinst-devel which provides all
the dependencies to build and run os-autoinst for the corresponding version of
the sources. To build a current version of os-autoinst it is recommended to
install os-autoinst-devel from
devel:openQA as the
distribution-provided packages might be too old or miss dependencies. This is
particularly true for openSUSE Leap. Also see
the openQA docs.
The required dependencies are also declared in dependencies.yaml. (The names
listed within that file are specific to openSUSE but can be easily transferred
to other distributions.)
Simply call
make
in the top folder which automatically creates a build directory and builds the complete project.
Call
make help
to list all available targets.
The above commands use a convenience Makefile calling cmake. For packaging,
when using an IDE or to conduct the steps manually it is suggested to use CMake
directly and do the following: Create a build directory outside of the source
directory. The following commands need to be invoked within that directory.
Configure build:
cmake $path_to_os_autoinst_checkout
You can specify any of the standard CMake variables, e.g.
-DCMAKE_BUILD_TYPE=Debug and -DCMAKE_INSTALL_PREFIX=/custom/install/prefix.
The following examples assume that GNU Make is used. It is possible to generate
for a different build tool by adding e.g. -G Ninja to the CMake arguments.
Build executables and libraries:
make symlinks
This target also creates symlinks of the built executables and libraries within
the source directory so isotovideo can find them.
Run all tests:
make check
By default CTest is invoked in verbose mode because prove already provides
condensed output. Add -DVERBOSE_CTEST=OFF to the CMake arguments to avoid
that.
Run all Perl tests (*.t files found within the t and xt directories):
make test-perl-testsuite
Run individual tests by specifying them explicitly:
make test-perl-testsuite TESTS="t/15-logging.t t/28-signalblocker.t"
Run perl author tests:
make test-local-author-perl
Run all author tests:
make test-local
Notice that the user needs to include the test directory for each test (either t for normal or xt for developer-centric tests) when specifying individual tests.
Add additional arguments to the prove invocation, e.g. enable verbose output:
make test-perl-testsuite PROVE_ARGS=-v
Gather coverage data while running tests:
make test-perl-testsuite WITH_COVER_OPTIONS=1
Generate a coverage report from the gathered coverage data:
make coverage
If no coverage data has been gathered so far the coverage target will invoke
the testsuite automatically.
Reset gathered coverage data:
make coverage-reset
Install files for packaging:
make install DESTDIR=…
Automatically tidy all perl files:
tools/tidyall
Tidy all changed perl files:
tools/tidyall --git
Further notes:
-
When using the
test-perl-testsuitetarget,ctestis not used (and thereforectestspecific tweaks have no effect). -
One can always run Perl tests manually via
proveafter the build has been conducted withmake symlinks. Note that some tests need to be invoked within thetdirectory. An invocation likeprove -vI.. -I../external/os-autoinst-common/lib 28-signalblocker.tis supposed to work. -
It is also possible to run
ctestwithin the build directory directly instead of using the mentioned targets. -
All mentioned variables to influence the test execution (
TESTS,WITH_COVER_OPTIONS, …) can be combined and can also be used with thecoveragetarget.
We provide a container to run isotovideo which can be used to run QEMU-based
tests directly in a CI runner. Checkout this
example workflow
for how it can be used. The
README of the example test distribution
also contains further details.
The script imgsearch in the repository’s script folder allows to use the
fuzzy image comparison independently of the normal test execution. Invoke the
script with no parameters to show its usage. There is also an
example file
showing what output you can expect. There is one key for each file to be
searched. The best matching image to be found will show up as match and the
other images under candidates. If no image matches well enough, match will
be null.
To use the script the previously shown build instructions need to be executed
(including the invocation of the symlinks target).
At a time Bernhard M. Wiedemann who later joined was on the openSUSE testing team and was assigned the task of testing the installer. Which meant tedious and dull work of waiting for 4GB ISO files to download when it’s not even clear if those things even boot. And as the Perl founder Larry Wall states, important traits of programmers are laziness, impatience and hybris. Which quickly led to developing os-autoinst to automate installations ;) See https://lizards.opensuse.org/2010/04/29/making-of-the-opensuse-install-video/ and https://lizards.opensuse.org/2010/05/25/automated-opensuse-testing/ for Bernhard’s blog posts.
When using the QEMU backend, also ensure your user running os-autoinst has
access to /dev/kvm.
modprobe kvm-intel || modprobe kvm-amd chgrp kvm /dev/kvm ; chmod g+rw /dev/kvm # maybe redundant # optionally use a new user; just to keep things separate useradd -m USERNAME -G kvm passwd USERNAME # and/or add ~USERNAME/.ssh/authorized_keys