From Fedora Project Wiki

< CI

(Add "Testing Tests" section)
(Migrated to the new Fedora docs site)
 
Line 1: Line 1:
= Motivation =
Moved to: https://docs.fedoraproject.org/en-US/ci/share-test-code/
 
In order to make the CI workflow reliable and efficient it is
crucial to keep the test coverage in a good shape at all times.
Sharing test code between several packages (even within multiple
branches of the same package) may significantly help to:
 
* Prevent test code duplication
* Minimize test maintenance
* Catch incompatibilities early
 
In general, tests define how the software works and the basic
functionality of many packages doesn’t change that often. We try
hard to keep the backward compatibility where possible. Thus it
seems natural that, for such components, tests guarding the spec
could change at a slower pace than the distribution branches.
 
See the whole [https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/ ci-list discussion] for some more context.
 
= Implementation =
 
Store test code in your preferred repository and reference the
tests from the dist-git yaml file. There is also a special
<code>tests</code> namespace dedicated for storing Fedora CI
integration tests:
 
* https://src.fedoraproject.org/projects/tests/*
 
Use <code>fedpkg</code> to quickly clone repositories from the
tests namespace:
 
fedpkg clone tests/shell
 
Some of the [[CI/Standard_Test_Roles|Standard Test Roles]]
(currently basic and beakerlib) support fetching test code from
remote repositories directly in their config in this way:
 
- role: standard-test-beakerlib
  repositories:
  - repo: "https://src.fedoraproject.org/tests/shell.git"
    dest: "shell"
 
It is also possible to specify version (branch, commit hash) which should
be fetched from the remote repository:
 
- role: standard-test-beakerlib
  repositories:
  - repo: "https://src.fedoraproject.org/tests/shell.git"
    dest: "shell"
    version: "devel"
 
= Testing Tests =
 
It is a good idea to ensure that updating tests in the shared
repository does not negatively impact packages which they are
testing. To enable pull request pipeline for tests stored in the
Fedora dist git tests namespace simply include `tests.yml` file in
the root of the test repository.
 
- hosts: localhost
  roles:
  - role: standard-test-basic
    tags:
    - classic
    tests:
    - smoke27:
        dir: smoke
        run: VERSION=2.7 METHOD=virtualenv ./venv.sh
    - smoke36:
        dir: smoke
        run: VERSION=3.6 METHOD=virtualenv ./venv.sh
    - smoke37:
        dir: smoke
        run: VERSION=3.7 METHOD=virtualenv ./venv.sh
    required_packages:
    - python27
    - python36
    - python37
 
The example above is a simplified version of the
[https://src.fedoraproject.org/tests/python/blob/master/f/tests.yml tests.yml]
file from the Python shared test repo and shows how to enable
`smoke` test to be executed against three versions of the Python
interpreter.
 
= Examples =
 
Here are some real-life examples where sharing test code can
increase long-term efficiency.
 
== Shell ==
 
There are several shells which implement the POSIX specification:
bash, ksh, mksh, zsh, dash. All of them share a significant amount
of test coverage and it does not make sense to commit & maintain
identical tests in five different repositories (+ possible
branches).
 
Shell tests repository:
 
* https://src.fedoraproject.org/tests/shell
 
Bash [https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml tests.yml]:
 
- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/shell.git"
      dest: "shell"
    tests:
    - shell/func
    - shell/login
    - shell/smoke
    required_packages:
    - expect            # login requires expect
    - which            # smoke requires which
 
Ksh [https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml tests.yml]:
 
- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/shell.git"
      dest: "shell"
    tests:
    - shell/func
    - shell/login
    - shell/smoke
    environment:
      PACKAGES: ksh
      SH_BIN: ksh
    required_packages:
    - ksh
    - expect            # login requires expect
    - which            # smoke requires which
 
== Ruby ==
 
Another example is Ruby: With about 80 packages related to Ruby on
Rails it would be useful and efficient to have a single place for
integration tests which verify that the framework is correctly
working after updating any of these packages. Conversely,
maintaining those tests in 80 repos would be a tedious task.
 
Currently the shared [https://src.fedoraproject.org/tests/ruby tests/ruby]
repository hosts these three ruby integration tests:
 
Available integration tests:
* systemtap-static-probes-in-ruby - exercising ruby's systemtap api
* bundler-unit-test - run bundler's unit tests
* run-basic-rails-application - run a simple rails application
 
== SELinux ==
 
Several SELinux user space components are sharing test coverage in
a single [https://src.fedoraproject.org/tests/selinux.git selinux] test
repository:
 
* [https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml libsepol]
* [https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml libselinux]
* [https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml libsemanage]
* [https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml policycoreutils]
 
= Start =
 
In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command. For example to create a shared test repository with the name foo, which will be available at [https://src.fedoraproject.org/tests/foo.git]
 
* Setup authentication to pagure according to the help in request-repo command
 
fedpkg request-repo -h
 
* Request a new repository with a sensible decription
 
fedpkg --module-name foo request-tests-repo "Description of the foo repository"

Latest revision as of 15:55, 18 March 2019