From Fedora Project Wiki

(Tests, Links)
(Use redirect)
 
(7 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Why =
#REDIRECT [[CI]]
 
== Vision ==
 
There are hundreds of packages which make up the Operating System.
Making sure that they all work together as a whole is not an easy
task. This becomes even harder as the number of packages and their
inter-dependencies grows. An extensive testing is required before
a new version of the operating system is released to ensure it is
stable enough. That is the past.
 
Imagine an '''Always Ready Operating System''' which consists of
packages which are constantly kept in a good shape.  Integrated
and stable thanks to an extensive test coverage which is
continuously executed upon changes in individual packages, in this
way allowing to prepare a new release in much shorter time, or
even in no time.
 
Imagine an operating system distribution which you could release
at any moment. This is where we are heading. Here comes the CI,
Continuous Integration, as an invaluable tool to ensure everything
is working together as expected in every point of time.
 
== Manifesto ==
 
Continuous integration aims to ensure broken changes are revealed
as soon as possible and do not affect other developers, packagers,
maintainers or users. The feedback that continuous integration
provides is vital for fast paced agile delivery of software. Late
testing, long after a change occurs, does not scale to the pace of
Fedora. Learn about the steps that are crucial for a working
Continuous Integration in the [[CI/Manifesto|CI Manifesto]].
 
= How =
 
There are three main pieces of the puzzle to get this nicely
working: A process which clearly defines how to discover and
execute tests, a set of tools which help to efficiently implement
the process and the tests themselves.
 
== Process ==
 
In order to clearly distinguish test from the CI system running it
the [[CI/Standard_Test_Interface|Standard Test Interface]] was
introduced. It clearly defines essential terms such as test, test
subject, test suite, test framework, test result, test artifact,
test system and describes what are their responsibilities and
requirements.
 
This general approach gives a nice flexibility as it does not
enforce any specific tools or frameworks to be used. Basically it
only describes how tests are discovered and where the testing
results should be stored to be processed by the automation.
 
== Tools ==
 
[[CI/Standard_Test_Roles|Standard Test Roles]] were implemented
to enable both automation tools and developers in their local
environments to easily execute tests. This set of ansible roles
supports various frameworks and allows to execute tests against
different test subjects (such as classic rpm package, docker
container or Atomic Host).
 
The testing [[CI/Pipeline|Pipeline]] detects tests for enabled
packages, executes the test coverage and gathers the results.
Currently tests are scheduled for new commits, support for pull
request testing is on the way. Results are displayed in
[https://src.fedoraproject.org/ Pagure] web interface.
 
== Tests ==
 
The core of the CI success are reliable tests of a good quality,
well selected, stable, organized and continuously maintained.
 
=== Test Types ===
 
In general it makes sense to store tests as close to the upstream
as possible. So what are the appropriate test types recommended
for testing the Always Ready Operating System?
 
* Basic functionality tests
* Integration tests
 
For unit tests it usually makes more sense to store them directly
within the upstream project repository. However, in some cases it
might be worth to fetch tests for Fedora CI from the upstream
repository as well.
 
=== Test Code ===
 
Tests are enabled by including the <code>tests.yml</code> file in
the package dist-git repository as defined by the Standard Test
Interface. Test code itself can be stored directly in the dist-git
or fetched from another repository. Shared tests namespace can be
used for storing test code relevant for multiple packages.
 
=== Test Execution ===
 
Executing a test written with the use of Standard Test Roles is as
simple as running an ansible playbook:
 
    ansible-playbook tests.yml
 
However, a set of environment variables needs to be set properly
in order to execute the test against the desired test subject. See
[[CI/Tests|Tests]] for more detailed instructions about running
tests and adding new test coverage.
 
= More =
 
== Contact ==
 
If you have questions or would like to get involved:
 
* IRC channel: <code>#fedora-ci</code> on FreeNode
* Mailing list: <code>ci@lists.fedoraproject.org</code>
* Issues: [https://pagure.io/fedora-ci/AtomicCi pagure.io/fedora-ci/AtomicCi]
 
== Links ==
 
Here's a summary of useful links:
 
* [[CI/Standard_Test_Interface|Standard Test Interface]] ... definition of the process
* [[CI/Standard_Test_Roles|Standard Test Roles]] ... set of ansible roles
* [[CI/Pipeline|Pipeline]] ... CI pipeline executing the tests
* [[CI/Tests|Tests]] ... executing and adding tests
* [https://upstreamfirst.fedorainfracloud.org/ Upstream First] ... tests to be ported to dist-git
 
== FAQ ==
 
* Where do I get the latest Atomic Host images for testing?

Latest revision as of 10:38, 8 December 2018

Redirect to: