Description
Package standard-test-roles
provides shared Ansible roles and inventory scripts implementing the Standard Test Interface. It has support for multiple testing frameworks (such as BeakerLib or Avocado) and in this way allows to easily enable existing tests in Fedora CI.
Setup
Packages
STR is available for Centos/RHEL from EPEL repository. As the first step install all necessary packages:
sudo dnf install fedpkg libselinux-python standard-test-roles
You can also install the latest version from the copr repo:
dnf copr -y enable @osci/standard-test-roles dnf update standard-test-roles
Artifacts
Output of the test (such as the stdout/stderr output, log files or screenshots) is by default saved in the artifacts
directory. Use TEST_ARTIFACTS
environment variable to choose a different location if desired:
export TEST_ARTIFACTS=/tmp/artifacts
Inventory
A test subject is what we call the thing to be tested. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Use the following command to enable it:
export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
As you can see from the way how the inventory is set, tests may contain their own inventory, which defines their own instructions for turning a test subject into one or more testable systems.
Testing
Classic
You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image test subject.
There may be more than one test present in a package git repository, but the file tests.yml
is the main entry point. To run it use the following command:
# ansible-playbook tests.yml
You can find output artifacts of the tests in an artifacts/
or specify a specific directory like this:
# ansible-playbook -e artifacts=/tmp/output tests.yml
You can filter which kinds of tests are run by providing a --tags
argument. To only run tests that are suited for classic systems installed by yum
or dnf
you can use a command like:
# ansible-playbook --tags=classic tests.yml
When run by a CI System the tests are invoked according to the Standard Test Interface. Look here for more details and standard invocation variables.
Package
When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.
A test subject is what we call the thing to be tested. RPMs are a particular kind of test subject. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Let's invoke the tests with an inventory and a specific version of gzip:
curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm export TEST_SUBJECTS=$PWD/gzip.rpm ansible-playbook tests.yml
You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles
inventory in /usr/share/ansible/inventory
to do this.
Container
Another example is to use a test subject of a container image. This is also a fully formed and integrated deliverable. The test subject again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an Ansible dynamic inventory.
export TEST_SUBJECTS=docker:docker.io/library/fedora:27 ansible-playbook --tags=container tests.yml
If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.
The --tags
argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.
See the Debug section for instructions how to log into a running container and diagnose why the tests failed.
Additional arguments for Docker
Tests for containers are run with a help of Docker. Containers are run within default security context. For more info see Seccomp security profiles for Docker. It is possible that some tests require additional privileges. In this case specify necessary arguments for Docker using an environment variable TEST_DOCKER_EXTRA_ARGS. For this create a file inventory file in tests directory with the following content:
#!/bin/bash export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined" exec merge-standard-inventory "$@"
or
#!/bin/bash export TEST_DOCKER_EXTRA_ARGS="--privileged" exec merge-standard-inventory "$@"
See merge-standard-inventory documentation for details.
Atomic
The former example may seem a bit contrived, but the concept of a test subject starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The test subject again represents the thing to be tested. The test subject in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use Ansible dynamic inventory.
curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest export TEST_SUBJECTS=/tmp/atomic.qcow2 ansible-playbook --tags=atomic tests.yml
If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host test subject. But most of the tests in core packages should work here.
Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles
inventory to do this.
The --tags
argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.
See the Debug section to learn how to diagnose why the tests failed, and log into the running Atomic Host.
Debug
To increase output verbosity use option -v
or -vvv
:
ansible-playbook --tags=container tests.yml -v
or for full verbosity:
ansible-playbook --tags=container tests.yml -vvv
To debug tests in a running container or atomic host use the TEST_DEBUG
environment variable. After the playbook runs, you'll see diagnosis information with a helpful command to log in.
export TEST_DEBUG=1
For container you'll see output like this:
DIAGNOSE: docker exec -it 56de801f0ddde36fc9770666f7be2a68f89d7f18f52b7b6fe7df7a12b193bf08 /bin/bash DIAGNOSE: kill 18261 # when finished
For atomic host the instructions are a bit different:
DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar DIAGNOSE: export ANSIBLE_INVENTORY=/tmp/inventory-cloudxyhF2M/inventory DIAGNOSE: kill 16611 # when finished
Now you can easily connect using these commands. Use suggested kill
command to finish the running instance when done with investigation.
Roles
Here's the list of currently supported roles for test execution:
- standard-test-avocado - role for executing tests written via the Avocado testing framework
- standard-test-basic - a simple role for executing runtest.sh scripts, or other scripts in given directories
- standart-test-beakerlib - role for executing tests written via Beakerlib testing framework, supporting all testing environments
- standard-test-rhts - role for executing tests written via Beakerlib testing framework, supporting only classic environment, works with upstream beakerlib-libraries
- standard-test-scripts - role for executing arbitrary test scripts
Here's list of currently supported helper roles:
- standard-test-repo - a role for installing packages for additional yum repository files
- standard-test-rpm - a role for installing additional rpms
- standard-test-source - a role for extracting upstream source tarball (with tests)
BeakerLib
This is the recommended role for running tests written via the Beakerlib Testing Framework as it supports all currenlty supported testing environments (atomic, classic, container), but it does not support beakerlib-libraries.
To use this role create tests.yml
file with contents similar to the following snippet. The list of tests is provided for the 'tests' parameter should be the list of directories with your beakerlib tests. The 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests.
--- - hosts: localhost tags: - atomic - classic - container roles: - role: standard-test-beakerlib tests: - cmd-line-options required_packages: - which # which package required for cmd-line-options - rpm-build # upstream-testsuite requires rpmbuild command - libtool # upstream-testsuite requires libtool - gettext # upstream-testsuite requires gettext
Note: The 'required_packages' parameter is ignored when running on Atomic Host--since there is no way to install additional packages in that environment.
Basic
Basic role can be used for executing simple scripts. Here's an example tests.yml
file which uses parametrizing a single integration test from a shared repository:
- hosts: localhost roles: - role: standard-test-basic tags: - classic repositories: - repo: "https://src.fedoraproject.org/tests/python.git" dest: "python" tests: - smoke27: dir: python/smoke run: VERSION=2.7 METHOD=virtualenv ./venv.sh - smoke37: dir: python/smoke run: VERSION=3.7 ./venv.sh required_packages: - python27 - python37 - python2-virtualenv - python3-virtualenv - python2-devel - python3-devel
RHTS
This is an alternative role for running tests written via the Beakerlib Testing Framework. This role supports upstream beakerlib-libraries but supports only classic environment. Note that when using the beakerlib libraries you need to explicitly ask for its usage by setting role variable 'use_beakerlib_libraries: yes'.
To use this role create tests.yml
file with contents similar to the following snippet. The list of tests is provided for the 'tests' parameter should be the list of directories with your beakerlib tests. The 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests.
--- - hosts: localhost tags: - classic - docker roles: - role: standard-test-rhts use_beakerlib_libraries: yes tests: - cmd-line-options required_packages: - which # which package required for cmd-line-options - rpm-build # upstream-testsuite requires rpmbuild command - libtool # upstream-testsuite requires libtool - gettext # upstream-testsuite requires gettext
Important Note: Tests using the 'standard-test-rhts' role are not compatible with atomic and docker environments. Please reflect that by omitting atomic
and container
from the <tags> section of the tests.yml playbook.
More
Links
Pagure and Copr repositories:
- https://pagure.io/standard-test-roles/
- https://copr.fedorainfracloud.org/coprs/g/osci/standard-test-roles/builds/
Contact
- Andrei Stepanov (astepano)
- Miroslav Vadkerti (mvadkert)