From Fedora Project Wiki

< CI

(Remove jython example)
(Add SELinux example)
Line 91: Line 91:


See the [https://fedoraproject.org/wiki/CI/Metadata Metadata] page for the full list of so far drafted attributes.
See the [https://fedoraproject.org/wiki/CI/Metadata Metadata] page for the full list of so far drafted attributes.
= SELinux =
There are several components related to SELinux. They are tightly connected so change in one of them can cause problems in other. That's why their tests are shared and executed together:
* https://src.fedoraproject.org/tests/selinux
Instead of listing relevant tests to be executed manually in each dist git rpms repository [https://fedoraproject.org/wiki/Flexible_Metadata_Format Flexible Metadata Format] is used:
    - hosts: localhost
      roles:
      - role: standard-test-beakerlib
        tags:
        - classic
        repositories:
        - repo: "https://src.fedoraproject.org/tests/selinux.git"
          dest: "selinux"
          fmf_filter: "tier: 1 | component: selinux-policy"
Provided `fmf_filter` selects all tests relevant for the `selinux-policy` component plus all Tier 1 selinux tests:
tier: 1 | component: selinux-policy
The following six components are covered:
* [https://src.fedoraproject.org/rpms/checkpolicy/blob/master/f/tests/tests.yml checkpolicy]
* [https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml libselinux]
* [https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml libsemanage]
* [https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml libsepol]
* [https://src.fedoraproject.org/rpms/policycoreutils/blob/master/f/tests/tests.yml policycoreutils]
* [https://src.fedoraproject.org/rpms/selinux-policy/blob/master/f/tests/tests.yml selinux-policy]
Use the `fmf` command line tool to quickly check which tests will be scheduled:
> dnf install -y fmf
> fedpkg clone -a tests/selinux
> cd selinux
> fmf ls --filter "tier: 1 | component: checkpolicy"
/selinux-policy/policy-rpm-macros
/checkpolicy/sedispol
/checkpolicy/checkmodule
/checkpolicy/sedismod
/checkpolicy/checkpolicy
/checkpolicy/checkpolicy-docs
/libsepol/sepol_check_context
/libsemanage/verify-options-in-semanage-conf
/libselinux/getsebool
/policycoreutils/booleans
See the Flexible Metadata Format documentation for other options how to [https://fmf.readthedocs.io/en/latest/overview.html#install install] fmf.

Revision as of 12:21, 2 October 2018

On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI.

Python

There are multiple versions of Python programming language available in Fedora and a number of related subpackages. As all of them should be tested (including their various combinatios) we share test coverage for them in the tests namespace:

The test repo contains basic smoke test for virtualenv together with example test Metadata stored in the Flexible Metadata Format:

Once the test is avaible in the share test repository it can be easily linked from supported Python versions:

We test additional Python implementations as well:

Plus we ensure that essential tools for venv and virtualnv, such as setuptools, pip or virtualenv itself correctly work with all supported versions:

Note that for the last set of examples we run the same test several times with modified environment. For example:

   - smoke36:
       dir: python/smoke
       run: VERSION=3.6 ./venv.sh
   - smoke37:
       dir: python/smoke
       run: VERSION=3.7 ./venv.sh
   - smoke26:
       dir: python/smoke
       run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh
   - smoke27:
       dir: python/smoke
       run: VERSION=2.7 METHOD=virtualenv ./venv.sh
   - smoke34_virtualenv:
       dir: python/smoke
       run: VERSION=3.4 METHOD=virtualenv ./venv.sh

In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance.

Shell

There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). Thus we store test code in the tests namespace:

These tests are then linked from all relevant tests.yml files:

Flexible Metadata Format filter is used to select appropriate tests instead of listing individual tests manually. Environment variables PACKAGES and SH_BIN are used to specify which shell implementation is being tested:

   - hosts: localhost
     roles:
     - role: standard-test-beakerlib
       tags:
       - classic
       repositories:
       - repo: "https://src.fedoraproject.org/tests/shell.git"
         dest: "shell"
         fmf_filter: "tier: 1, 2 & tags: classic"
       environment:
         PACKAGES: ksh
         SH_BIN: ksh
       required_packages:
       - ksh
       - expect            # login requires expect
       - which             # smoke requires which

Some of the tests might be relevant only for selected components. This can be handled easily by additional component condition:

   repositories:
   - repo: "https://src.fedoraproject.org/tests/shell.git"
     dest: "shell"
     fmf_filter: "tier: 1, 2 & component: dash"

See the Metadata page for the full list of so far drafted attributes.

SELinux

There are several components related to SELinux. They are tightly connected so change in one of them can cause problems in other. That's why their tests are shared and executed together:

Instead of listing relevant tests to be executed manually in each dist git rpms repository Flexible Metadata Format is used:

   - hosts: localhost
     roles:
     - role: standard-test-beakerlib
       tags:
       - classic
       repositories:
       - repo: "https://src.fedoraproject.org/tests/selinux.git"
         dest: "selinux"
         fmf_filter: "tier: 1 | component: selinux-policy"

Provided fmf_filter selects all tests relevant for the selinux-policy component plus all Tier 1 selinux tests:

tier: 1 | component: selinux-policy

The following six components are covered:

Use the fmf command line tool to quickly check which tests will be scheduled:

> dnf install -y fmf
> fedpkg clone -a tests/selinux
> cd selinux
> fmf ls --filter "tier: 1 | component: checkpolicy"
/selinux-policy/policy-rpm-macros
/checkpolicy/sedispol
/checkpolicy/checkmodule
/checkpolicy/sedismod
/checkpolicy/checkpolicy
/checkpolicy/checkpolicy-docs
/libsepol/sepol_check_context
/libsemanage/verify-options-in-semanage-conf
/libselinux/getsebool
/policycoreutils/booleans

See the Flexible Metadata Format documentation for other options how to install fmf.