From Fedora Project Wiki

(Add a question and answer about %check vs the CI pipeline)
(Migrated to the new Fedora docs site)
 
Line 1: Line 1:
= Why =
+
Page moved to: https://docs.fedoraproject.org/en-US/ci/
 
 
== Vision ==
 
 
 
There are hundreds of packages which make up the Operating System.
 
Making sure that they all work together as a whole is not an easy
 
task. This becomes even harder as the number of packages and their
 
inter-dependencies grows. An extensive testing is required before
 
a new version of the operating system is released to ensure it is
 
stable enough. That is the past.
 
 
 
Imagine an '''Always Ready Operating System''' which consists of
 
packages which are constantly kept in a good shape.  Integrated
 
and stable thanks to an extensive test coverage which is
 
continuously executed upon changes in individual packages, in this
 
way allowing to prepare a new release in much shorter time, or
 
even in no time.
 
 
 
Imagine an operating system distribution which you could release
 
at any moment. This is where we are heading. Here comes the CI,
 
Continuous Integration, as an invaluable tool to ensure everything
 
is working together as expected in every point of time.
 
 
 
== Manifesto ==
 
 
 
Continuous integration aims to ensure broken changes are revealed
 
as soon as possible and do not affect other developers, packagers,
 
maintainers or users. The feedback that continuous integration
 
provides is vital for fast paced agile delivery of software. Late
 
testing, long after a change occurs, does not scale to the pace of
 
Fedora. Learn the goals, terminology and rules for a working CI
 
in the manifesto.
 
 
 
* [[CI/Manifesto|Continuous Integration Manifesto]]
 
 
 
= How =
 
 
 
There are three main pieces of the puzzle to get this nicely
 
working: A process which clearly defines how to discover and
 
execute tests, a set of tools which help to efficiently implement
 
the process and the tests themselves.
 
 
 
== Process ==
 
 
 
=== Standard Test Interface ===
 
 
 
In order to clearly distinguish test from the CI system running it
 
the '''Standard Test Interface''' was introduced. It clearly
 
defines essential terms such as test, test subject, test suite,
 
test framework, test result, test artifact, test system and
 
describes what are their responsibilities and requirements.
 
 
 
This general approach gives a nice flexibility as it does not
 
enforce any specific tools or frameworks to be used. Basically it
 
only describes how tests are discovered and where the testing
 
results should be stored to be processed by the automation.
 
 
 
* [[CI/Standard_Test_Interface|Standard Test Interface]]
 
 
 
=== Gating ===
 
 
 
When a test fails, CI can prevent the broken change from affecting
 
other packages. That gating happens in Bodhi. Greewave is used
 
together with ResultsDB and WaiverDB to make the gating decision.
 
 
 
* [[CI/Gating|Gating]] ... how to enable gating, waiving instructions
 
* [https://bodhi.fedoraproject.org/ Bodhi] ... Fedora updates System
 
* [https://pagure.io/greenwave Greewave] ... service to evaluate gating policies based on test results
 
* [https://pagure.io/taskotron/resultsdb ResultsDB] ... results store engine
 
* [https://pagure.io/waiverdb WaiverDB] ... service for recording waivers against test results
 
 
 
=== Notifications ===
 
 
 
[https://apps.fedoraproject.org/notifications Fedora Notifications]
 
have been adjusted to notify by default every packager when any
 
step of the CI pipeline fails on one of the package they maintain.
 
So if you are a kernel maintainer and a commit made to the kernel
 
dist-git repository fails to compose an OSTree, FMN will notify
 
you of it.
 
 
 
Bodhi includes the CI results in its update page, just as it
 
already includes tests results from taskotron.
 
 
 
=== Messages ===
 
 
 
Various tools involved in the CI automation are sending updates
 
about the progress of the testing to the message bus. Consistent
 
format of the CI messages allows to build simpler services which
 
will provide useful features for everyone.
 
 
 
* [https://pagure.io/fedora-ci/messages Fedora CI Messages] ... CI messages specification
 
* [https://apps.fedoraproject.org/datagrepper/ Datagrepper] ... search sent messages (e.g. by [https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.pipeline.allpackages-pr.complete topic])
 
* [https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline#fedmsg-bus fedmsg bus] ... messages sent by the pipeline
 
* [https://fedora-fedmsg.readthedocs.io/en/latest/topics.html#ci topics] ... list of fedmsg topics related to CI
 
 
 
== Tools ==
 
 
 
=== Standard Test Roles ===
 
 
 
'''Standard Test Roles''' were implemented to enable both
 
automation tools and developers in their local environments to
 
easily execute tests. This set of ansible roles supports various
 
frameworks and allows to execute tests against different test
 
subjects (such as classic rpm package, docker container or Atomic
 
Host).
 
 
 
* [[CI/Standard_Test_Roles|Standard Test Roles]]
 
* https://pagure.io/standard-test-roles/
 
 
 
=== Frameworks ===
 
 
 
As Standard Test Interface does not define test framework which
 
should be used for testing it is possible to choose the most
 
suitable framework for your project. Here are some examples:
 
 
 
* [https://github.com/beakerlib/beakerlib/wiki/man BeakerLib] and it's [https://pagure.io/beakerlib-libraries Libraries]
 
* [https://github.com/avocado-framework/avocado Avocado]
 
 
 
=== Metadata ===
 
 
 
Standard Test Interface defines only a very simple metadata for
 
selecting which tests should be run. For more complex scenarios
 
'''Flexible Metadata Format''' can be used:
 
 
 
* Fedora CI [https://pagure.io/fedora-ci/metadata Metadata Specification]
 
* [[Flexible_Metadata_Format|Flexible Metadata Format]]
 
* https://github.com/psss/fmf
 
 
 
=== Pipeline ===
 
 
 
The testing '''Pipeline''' detects tests for enabled
 
packages, executes the test coverage and gathers the results.
 
 
 
* [[CI/Pipeline|Pipeline]]
 
* [https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md#ci-pipeline-architecture-and-design CI Pipeline Architecture and Design]
 
 
 
=== Pagure ===
 
 
 
Test results from the CI pipeline are displayed in '''Pagure'''
 
web interface. See the commits page and pull request pages of
 
respective package to see the results. Pagure integration with
 
COPR provides auto-rebuilding and pull request/commit flagging on
 
new changes.
 
 
 
* [https://src.fedoraproject.org/ Package Sources]
 
* [https://pagure.io/ Pagure]
 
* [https://docs.pagure.org/copr.copr/user_documentation.html#pagure-integration COPR Integration]
 
 
 
== Tests ==
 
 
 
The core of the CI success are reliable tests of a good quality,
 
well selected, stable, organized and continuously maintained.
 
 
 
=== Test Types ===
 
 
 
In general it makes sense to store tests as close to the upstream
 
as possible. So what are the appropriate test types recommended
 
for testing the Always Ready Operating System?
 
 
 
* Basic functionality tests
 
* Integration tests
 
 
 
For unit tests it usually makes more sense to store them directly
 
within the upstream project repository. However, in some cases it
 
might be worth to fetch tests for Fedora CI from the upstream
 
repository as well.
 
 
 
=== Test Code ===
 
 
 
Tests may be written all sorts of different ways, but have to be
 
exposed and invoked in a standard way. Tests are enabled by
 
including the <code>tests/tests.yml</code> file in the package
 
dist-git repository as defined by the Standard Test Interface.
 
Test code itself can be stored directly in the dist-git or fetched
 
from another repository. Shared tests namespace can be used for
 
storing test code relevant for multiple packages.
 
 
 
* [[CI/Quick_Start_Guide|Quick Start Guide]] ... quick intro for the impatient
 
* [[CI/Tests|Tests]] ... how to run, write and wrap tests
 
* [[CI/Share_Test_Code|Share Test Code]] ... shared tests namespace
 
* [[CI/Pull_Requests|Pull Requests]] ... how to create pull requests
 
* [[CI/Examples|Examples]] ... real-life success stories of enabled tests
 
 
 
=== Test Execution ===
 
 
 
Executing a test written with the use of Standard Test Roles is as
 
simple as running an ansible playbook <code>ansible-playbook
 
tests.yml</code>. However, a set of environment variables needs to
 
be set properly in order to execute the test against the desired
 
test subject. The '''Tests''' wiki contains detailed instructions
 
about running tests and adding new test coverage.
 
 
 
* [[CI/Tests|Tests]] ... How to run, write and wrap tests
 
 
 
=== Test Coverage ===
 
 
 
There are statistics pages to track how many Fedora dist-git repositories have tests in the Standard Test Format. These pages are updated automatically.
 
 
 
* [[CI/Tests/stat|Basic Operating System Status]]
 
 
 
=== Test Porting ===
 
 
 
There is an active effort to open source existing internal Red Hat
 
tests to Fedora called Upstream First. Tests being migrated can be
 
found in the upstream first repository:
 
 
 
* [https://upstreamfirst.fedorainfracloud.org/ Upstream First]
 
 
 
=== Shared Responsibility ===
 
 
 
The ownership and maintenance of tests should be shared between QE & Devel. For tests that reside in the rpm namespace, QE can use pull requests to create/update tests. Likewise in the tests namespace, both QE and Devel will have commit rights, both QE and Devel should review and sign-off with each commit.
 
 
 
= More =
 
 
 
== Contact ==
 
 
 
If you have questions or would like to get involved:
 
 
 
* IRC channel: <code>#fedora-ci</code> on FreeNode
 
* Mailing list: [mailto:ci@lists.fedoraproject.org ci@lists.fedoraproject.org] ([https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/ archive])
 
* Issues: [https://pagure.io/fedora-ci/general General], [https://pagure.io/fedora-ci/AtomicCi Atomic CI]
 
 
 
== Links ==
 
 
 
Here's a summary of useful links:
 
 
 
* [[CI/Standard_Test_Interface|Standard Test Interface]] ... definition of the process
 
* [[CI/Standard_Test_Roles|Standard Test Roles]] ... set of ansible roles
 
* [https://pagure.io/fedora-ci/workshop Workshop] ... workshop materials
 
* [[CI/Tests|Tests]] ... executing and adding tests
 
* [https://upstreamfirst.fedorainfracloud.org/ Upstream First] ... tests to be ported to dist-git
 
 
 
== FAQ ==
 
 
 
* Where do I get the latest Atomic Host images for testing?
 
 
 
* Why not rely on %check?
 
 
 
%check and the tests run in the CI pipeline are complementary. %check allows to perform a number of checks at build time but will not detect missing requirement, missing configuration file and this kind of situation which the CI pipeline will be able to test and find.
 
We could see this as %check being unit-test (where the unit is the package) versus integration tests where the tests are run in the entire distribution just as we distribute it to our users. So the Ci pipeline will be able to find integration bugs that the %check section will never be able to.
 

Latest revision as of 15:46, 18 March 2019