< Changes
Metamorph: Standard Discovery, Staging, Invocation of Integration Tests
This file is incomplete.
Feedback is more than welcome. There's a discussion tab above.
Summary
Lets define a clear delineation of between a testsuite (including framework) and the CI system that is running the test suite. This is the standard interface.
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.
Owner
- Name: Jenny Severance
- Email: jgalipea@redhat.com
Terminology
- Test Subject: The items that are to be tested.
- Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
- Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a test subject.
- Test Suite: The collection of all tests that apply to a test subject.
- Test Framework: A library or component that the test suite and tests use to accomplish their job.
- Examples: Avocado, GNOME Installed Tests, Modularity Testing Framework, Ansible tests in Atomic Host, Tunir tests, docker test images, ...
- Test Result: A boolean pass/fail output of a test suite.
- Test results are for consumption by automated aspects of a testing systems.
- Test Artifact: Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.
- Test artifacts are for consumption by humans, archival or big data analysis.
- Testing System: A CI or other testing system that would like to discover, stage and invoke tests for a test subject.
Responsibilities
The testing system is responsible to:
- Build or otherwise acquire the test subject, such as package, container image, tree …
- Decide which test suite to run, often by using the standard interface to discover appropriate tests for the dist-git repo that a test subject originated in.
- Schedule, provision or orchestrate a job to run the test suite on appropriate compute, storage, ...
- Stage the test suite as described by the standard interface.
- Invoke the test suite as described by the standard interface.
- Gather the test results and test artifacts as described by the standard interface.
- Announce and relay the test results and test artifacts for gating, archival ...
The standard interface describes how to:
- Discover a test suite for a given dist-git repo.
- Uniquely identify a test suite.
- Stage a test suite and its dependencies such as test frameworks.
- Provide the test subject to the test suite.
- Invoke a test suite in a consistent way.
- Gather test results and test artifacts from the invoked test suite.
The test suite is responsible to:
- Declare its dependencies such as a test framework via the standard interface.
- Execute the test framework as necessary.
- Provision (usually locally) any containers or virtual machines necessary for testing the test subject.
- Provide test results and test subjects back according to the standard
The format of the textual logs and test artifacts that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible test suites.
Requirements
- The test suite and test framework SHOULD NOT leak its implementation details into the testing system, other than via the standard interface.
- The test suite and test framework SHOULD NOT rely on behavior of the testing system other than the standard interface.
- The standard interface MUST enable a dist-git packager to run a test suite locally.
- Test suites or test frameworks MAY call out to the network for certain tasks.
- It MUST be possible to stage an upstream test suite using the standard interface.
- Both in-situ tests, and more rigorous outside-in tests MUST be possible with the standard interface.
- For in-situ tests the test suite is in the same file system tree and process space as the test subject.
- For outside-in tests the test suite is outside of the file system tree and process space of the test subject.
- The test suite and test framework SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the testing system.
- The standard interface SHOULD describe how to uniquely identify a test suite,
Detailed Description
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.
Staging
TODO: Fill in description of how Metamorph will describe or encapsulate test suite and test framework dependencies. Describe how a testing system will stage this using the Metamorph metadata.
Invocation
TODO: Describe the standard way that testing system will invoke the test suite. Describe the layout of the files in the test suite, metadata, test results, and test artifacts to gather after invocation.
Discovery
A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.
TODO: Describe how a test suite can be uniquely identified, and how to identify a test suite given a dist-git repo.
Scope
TODO: Note if this change requires changes to Fedora infrastructure itself. Or only affects contents spec files in dist-git repos and how.
Benefit to Fedora
Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
TODO: note any additional benefits to Fedora.
User Experience
A standard way to package tests benefits Fedora stability, and makes Fedora better for users.
TODO: note any additional benefits to users.
Upgrade/compatibility impact
TODO: note any upgrade or compatibility impact.
Examples
What follows are examples of writing and/or packaging existing tests to this standard.
TODO: Put general example notes here.
Example: Simple in-situ test
Add simple downstream integration test for gzip:
TODO: Build an example that uses the gzip tests. Candidate tests are here: https://patches.ubuntu.com/g/gzip/
Example: GNOME style "Installed Tests"
Add downstream integration test running in gnome installed tests.
TODO: Build an example that wraps the glib2-tests installed tests already in the dist-git repo. Reference: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
Example: Tests run in Docker Container
Add integration test running tests in a docker container.
TODO: Build an example that stages and runs a docker container and runs the tests in or against it.
Example: Modularity testing Framework
TODO: Port an example
Example: Ansible with Atomic Host
TODO: Port an existing test
Example: Beakerlib based test
TODO: Port and shim a beakerlib test
Evaluation and Notes
- ...
- ...