From Fedora Project Wiki

< CI

(Minor adjustments)
(Migrated to the new Fedora docs site)
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
Moved to: https://docs.fedoraproject.org/en-US/ci/tests/
= Enabling =
 
Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the [[CI/Standard_Test_Interface|Standard Test Interface]] directly in the package [https://src.fedoraproject.org/projects/rpms/%2A git repository]. To start working on tests you can clone a package repo directly:
 
git clone https://src.fedoraproject.org/rpms/qrencode.git
 
You can also use the <code>fedpkg</code> to clone the repo. See the [[Package_maintenance_guide|Package Maintenance Guide]] for more info about the tool:
 
fedpkg clone -a qrencode
 
Tests are enabled by including the <code>tests.yml</code> file under the <code>tests</code> directory:
 
cd qrencode/tests
cat tests.yml
 
Tests are wrapped or written as [http://docs.ansible.com/ansible/playbooks.html Ansible playbooks]. Here is an example of a simple playbok which enables a single <code>smoke</code> test of the <code>qrencode</code> package:
 
- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    - container
    - atomic
    tests:
    - smoke
    required_packages:
    - qrencode
    - file
 
Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:
 
* '''role''' — this test uses role <code>standard-test-beakerlib</code> from [[CI/Standard_Test_Roles|Standard Test Roles]] to run a BeakerLib test
* '''tags''' — all three test subjects ([[CI/Standard_Test_Roles#Atomic|classic]] rpm, docker [[CI/Standard_Test_Roles#Container|container]] and [[CI/Standard_Test_Roles#Atomic|atomic]] host) are relevant for this test
* '''tests''' — list of tests to be executed (here we have just a single smoke test)
* '''required_packages''' — list of rpm packages required for test execution
 
There may by multiple files ending in <code>.yml</code> in the <code>tests/</code> subdirectory and each of them can represent a test or a part of a test. All of them need to be included in the main <code>tests.yml</code> file. Let's have a look at the <code>gzip</code> example:
 
> fedpkg clone -a gzip
Cloning into 'gzip'...
 
> cd gzip/tests/
> ls
test-simple  test_simple.yml  tests.yml
 
> cat tests.yml
- include: test_simple.yml
 
= Executing =
 
Before running tests make sure you have the following dependencies installed on your system:
 
dnf install ansible python2-dnf libselinux-python standard-test-roles
 
Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.
 
{{admon/important|Tests may modify or destroy your environment|It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.}}
 
Running a test directly on the current system is easy:
 
ansible-playbook tests.yml
 
To only run tests that are suited for classic systems installed by <code>yum</code> or <code>dnf</code> use the <code>--tags</code> argument:
 
ansible-playbook --tags=classic tests.yml
 
See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for detailed instructions how to run tests for a specific [[CI/Standard_Test_Roles#Package|Rpm Package]], [[CI/Standard_Test_Roles#Container|Docker Container]] or [[CI/Standard_Test_Roles#Atomic|Atomic Host]].
 
= Adding =
 
Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure such as the [[CI/Share_Test_Code|Test Namespace]]. The simplest way to add a new test is by using one of the existing [[CI/Standard_Test_Roles|Standard Test Roles]] which take care of many implementatin details. If you want to create a custom test follow instructions below.
 
Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an [http://docs.ansible.com/ansible/latest/playbooks.html Ansible playbook] with a new name. Make sure the extension is <code>.yml</code>. Lets place the following example in <code>test_pid_1.yml</code> file.
 
<pre>
---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  tasks:
  - name: Test block
    block:
      - name: Test that /proc/1 exists
        shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log
 
    always:
      - name: Pull out the artifacts
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes
</pre>
 
All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.
 
By use of <code>tags</code> we note what kind of systems this test is suitable to run on.
 
The <code>block</code> is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.
 
Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass. Try changing the <code>/proc/1</code> argument to another value, and the test should fail.
 
You can use most of the Ansible techniques in your playbooks. Take a look at the [[CI/Standard_Test_Roles|Standard Test Roles]] for Ansible roles to make writing your tests easier.
 
'''Marking the test to be run'''
 
Just having a <code>.yml</code> file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a <code>tests.yml</code> playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
 
- import_playbook: test_pid_1.yml
 
You can now run this test with the standard commands above.
 
= Wrapping =
 
Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called <code>test-simple</code>
 
#!/bin/sh
set -ex
# exercise installed gzip/gunzip programs
echo "Bla" > bla.file
cp bla.file bla.file.orig
gzip bla.file
gunzip bla.file.gz
cmp bla.file bla.file.orig
rm bla.file bla.file.orig
 
We can write an Ansible wrapper for this script like this in <code>test_simple.yml</code>:
 
<pre>
---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  remote_user: root
  tasks:
  - name: Install the test files
    copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
    with_items:
    - {file: test-simple, dest: test-simple }
 
  - name: Test block
    block:
      - name: Execute the tests
        shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple
 
    always:
      - name: Pull out the logs
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes
</pre>
 
All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.
 
The <code>block</code> is the section that runs the actual test.
 
Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
 
- import_playbook: test_simple.yml
 
Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass.
 
See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for instructions how to wrap a [[CI/Standard_Test_Roles#BeakerLib|BeakerLib]] and [[CI/Standard_Test_Roles#RHTS|RHTS]] tests.
 
[[Category:FedoraAtomicCi]]

Latest revision as of 15:54, 18 March 2019