From Fedora Project Wiki

< CI

(Add initial documentation for writing tests)
(Migrated to the new Fedora docs site)
 
(61 intermediate revisions by 11 users not shown)
Line 1: Line 1:
Tests are stored in [[Package_maintenance_guide|package or module git repositories]] along with the packages and modules that they test. The tests are updated together with the software.
+
Moved to: https://docs.fedoraproject.org/en-US/ci/tests/
 
 
== Setting up ==
 
 
 
You can use the <code>fedpkg</code> tool to checkout a package git repository. If a <code>tests/</code> subdirectory exists, then the repository contains tests. The files ending in <code>.yml</code> in the <code>tests/</code> subdirectory each represent a test or a part of a test.
 
 
 
The tests are wrapped or written as [http://docs.ansible.com/ansible/playbooks.html Ansible playbooks] and
 
[https://fedoraproject.org/wiki/Changes/InvokingTests invoked according to this specification]. To invoke the tests you need the following dependencies (as described by the specification) installed on a modern Fedora system:
 
 
 
<pre>
 
$ sudo dnf install ansible python2-dnf libselinux-python standard-test-roles
 
</pre>
 
 
 
For the following documentation we'll checkout the <code>gzip</code> tests:
 
 
 
<pre>
 
$ fedpkg clone gzip
 
$ cd gzip/tests/
 
$ sudo -s
 
</pre>
 
 
 
{{admon/important|Work in progress|The upstreaming of tests is currently ongoing.
 
Use the following command until the end of Summer 2017.}}
 
 
 
<pre>
 
$ git clone https://upstreamfirst.fedorainfracloud.org/gzip.git
 
$ cd gzip/
 
$ sudo -s
 
</pre>
 
 
 
Although some playbooks may function without sudo, tests are always invoked as root. The test
 
itself may set up users and/or drop permissions if a part of that test. But in general be sure
 
to be root when invoking tests.
 
 
 
== Running tests ==
 
 
 
You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following tests invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated system, such as an Atomic Host or container image ''test subject''.
 
 
 
There may be more than one test present in a package git repository, but the file <code>tests.yml</code> is the main entry point. To run it use the following command:
 
 
 
<pre>
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
You can find output artifacts of the tests in an <code>artifacts/</code> or specify a specific directory like this:
 
 
 
<pre>
 
# ansible-playbook -e artifacts=/tmp/output tests.yml
 
</pre>
 
 
 
When run by a CI System the tests are [[Changes/InvokingTests|invoked according to a specification]]. Look there for more details and standard invocation variables.
 
 
 
=== Testing specific RPMs ===
 
 
 
When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.
 
 
 
A ''test subject'' is what we call the thing to be tested. RPMs are a particular kind of ''test subject''. To turn a test subject into a launched, installed system to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory]. Lets invoke the tests with an inventory and a specific version of gzip:
 
 
 
<pre>
 
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /etc/ansible/inventory)
 
# curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
 
# export TEST_SUBJECTS=gzip.rpm
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a ''test subject'' into one or more testable systems. But in this case we use the default <code>standard-test-roles</code> inventory in <code>/etc/ansible/inventory</code> to do this.
 
 
 
=== Testing an Atomic Host ===
 
 
 
The former example may seem a bit contrived, but the concept of a ''test subject'' starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The ''test subject'' again represents the thing to be tested. The ''test subject'' in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
 
 
 
<pre>
 
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /etc/ansible/inventory)
 
# curl -Lo atomic.qcow2 https://ftp-stud.hs-esslingen.de/pub/Mirrors/alt.fedoraproject.org/atomic/stable/Fedora-Atomic-26-20170707.1/CloudImages/x86_64/images/Fedora-Atomic-26-20170707.1.x86_64.qcow2
 
# export TEST_SUBJECTS=atomic.qcow2
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact for certain cases the software to be tested may not be included in the Atomic Host ''test subject''. But most of the tests in core packages should work here.
 
 
 
Some tests contain their own inventory, that is their own instructions for turning a
 
''test subject'' into one or more testable systems. But in this case we use the default
 
<code>standard-test-roles</code> inventory to do this.
 
 
 
=== Testing a Container Image ===
 
 
 
Another example is to use a ''test subject'' of a container image. This is also a fully formed and integrated deliverable. The ''test subject'' again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
 
 
 
<pre>
 
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /etc/ansible/inventory)
 
# export TEST_SUBJECTS=docker:docker.io/library/fedora:26
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact for certain tests the software to be tested may not eb included in the container. But many of the tests for core packages should work here.
 
 
 
== Adding tests ==
 
 
 
Tests are stored in package or module git repositories along with the packages and modules that they test. The tests are updated together with the software. If you're not a package maintainer be sure to talk with the maintainers or team and come to a such a common understanding.
 
 
 
=== Repository for test ===
 
 
 
{{admon/important|Work in progress|One can contribute to dist-git packages without being the maintainer
 
by using a Pagure interface. It is not yet ready. Place tests in the following location until end of Summer 2017.}}
 
 
 
At the current time you need to find or create a repository for the tests in the [https://upstreamfirst.fedorainfracloud.org/ upstreamfirst.fedorainfracloud.org] repositories. The name should be identical to the dist-git repository of the package or module that you are targetting.
 
 
 
# Use the ''Browse'' button to find existing repos.
 
# If a repository with the name of the target dist-git repository does exist, use the <code>[+]</code> create button to make a new one.
 
# Use the ''Project name'' that is identical to the name of the dist-git repository.
 
# Leave the ''Create README'' option unchecked
 
 
 
These repositories will soon be moved into the real package dist-git repositories. The contents of each repository will go into the <code>tests/</code> folder of the target dist-git repository.
 
 
 
=== Writing a new test ===
 
 
 
Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. If you wish to wrap an existing test, see the sections below.
 
 
 
Create an [http://docs.ansible.com/ansible/latest/playbooks.html Ansible playbook] with a new name. Make sure the extension is <code>.yml</code>. Lets place the following example in <code>test_pid_1.yml</code> file.
 
 
 
<pre>
 
---
 
- hosts: localhost
 
  vars:
 
  - artifacts: ./artifacts
 
  tasks:
 
  - name: Make artifacts directory
 
    file: path={{ artifacts }} state=directory owner=root mode=755 recurse=yes
 
 
 
  - block:
 
    - name: Test that /proc/1 exists
 
      shell: find /proc > {{ artifacts }}/proc && grep -q /proc/1
 
 
 
  - always:
 
    - name: Pull out the artifacts
 
      fetch:
 
        dest: "{{ artifacts }}/"
 
        src: "{{ artifacts }}/"
 
</pre>
 
 
 
All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.
 
 
 
The <code>block</code> is the section that runs the actual test. In this example we use a rather convoluted way of checking that PID 1 exists. However by doing so, we place an extra test artifact in the artifacts directory.
 
 
 
Lastly we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker image, using the examples above. It should pass. Try changing the <code>/proc/1</code> argument to another value, and the test should fail.
 
 
 
You can use most of the Ansible techniques your playbooks. And take a look at the [https://pagure.io/standard-test-roles standard-test-roles] for Ansible roles to make writing your tests easier.
 
 
 
=== Marking the test to be run ===
 
 
 
Just having a <code>.yml</code> file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add  it from a <code>tests.yml</code> playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
 
 
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
 
 
 
<pre>
 
- include: test_pid_1.yml
 
</pre>
 
 
 
You can now run this test with the standard commands above.
 
 
 
{{admon/important|Work in progress|Work is being done to define how to tag Ansible playbooks in such a way as to indicate whether a test is able to be run in an Atomic Host, a Docker image, and so on.}}
 
 
 
=== Wrapping a script test ===
 
 
 
TODO
 
 
 
=== Wrapping an installed test ===
 
 
 
TODO
 
 
 
=== Wrapping a beakerlib test ===
 
 
 
TODO
 
 
 
=== Wrapping a Restraint Test ===
 
 
 
TODO
 

Latest revision as of 15:54, 18 March 2019