From Fedora Project Wiki
m (fixing wiki link syntax)
mNo edit summary
(8 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{autolang|base=yes}}
== Objective: Deliver Fedora Atomic Host via CI/CD ==
== Objective: Deliver Fedora Atomic Host via CI/CD ==


Line 41: Line 43:
* Tests check that Fedora Atomic Host works with key containers produced by Modularity effort.
* Tests check that Fedora Atomic Host works with key containers produced by Modularity effort.


== Working Group and Special Interest Group ==
== Plans and Work Areas ==


We will engage with the [https://fedoraproject.org/wiki/Atomic/PRD Fedora Atomic Working Group] to align on goals, content and deliverables of Fedora Atomic Host.  
* Tests: Tests are to be included in dist-git in a standard format.
** Landing page: https://fedoraproject.org/wiki/CI
** Test Specification: https://fedoraproject.org/wiki/Changes/InvokingTests
** Tutorial: https://fedoraproject.org/wiki/CI/Tests
* CI/CD Integration into Fedora: Work will continue to integrate CI and CD into Fedora infrastructure.
** [[FedoraAtomicCI/KojiBuilds|Integration with Koji]]
** Gating of packages on test results
** Displaying test results from Pipeline
** Enabling multiple contributors to a dist-git repository
* CI Pipeline: For running tests
** Based on Openshift and Jenkins
** Source Code: https://github.com/CentOS-PaaS-SIG/ci-pipeline
** Documentation: https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md
 
Changes to how Atomic Host is built are needed. We have identifed [https://pagure.io/atomic-wg/issue/312 several requirements] that are needed for the CI/CD pipeline:
 
Requirements:
* The definition of what is composed into Atomic Host artifacts should include specific versions of packages, and all dependencies included.
* The definition of what is composed in an Atomic Host should be stored in a git repository so that changes can be detected easily. The CI/CD pipeline can be triggered off of changes to this repository.
* A mechanism to make a future composed Atomic Host artifact, contain an earlier (in RPM NVR parlance) version of a package.
* The high level functional Atomic Host tests should live in the same git repository with the rigorous definition of what goes into an Atomic Host.
* A mechanism to tell the CI Pipeline that multiple dist-git repository changes (i.e. multiple changing RPMs) should be built and tested together.
 
It is possible the build and MBS parts of [[Modularity|Modularity]] could be used to accomplish these requirements. Experimentation is needed in order to determine if there are any blockers for Atomic Host being built from a module.
 
== CI Working Group ==


Much of the pipeline and testing effort on this Objective will be undertaken by a new "'''CI Special Interest Group'''". This group will invite people from the Fedora Engineering team, Fedora Release-Engineering team, Fedora QA, Security Team and more. Work will be coordinated when possible with the Factory 2.0 effort. Contact info:
Much of the pipeline and testing effort on this Objective will be undertaken by a new "'''CI Special Interest Group'''". This group will invite people from the Fedora Engineering team, Fedora Release-Engineering team, Fedora QA, Security Team and more. Work will be coordinated when possible with the Factory 2.0 effort. Contact info:
Line 49: Line 76:
* #fedora-ci on FreeNode
* #fedora-ci on FreeNode
* Mailing list: [https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/ ci@lists.fedoraproject.org]
* Mailing list: [https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/ ci@lists.fedoraproject.org]
* If you don't know where to go: [[user:Stefw|Stef Walter]] [[user:pingou|Pierre-Yves Chibon]]
* For CI topics if you don't know where to go: [[user:Stefw|Stef Walter]] [[user:pingou|Pierre-Yves Chibon]]
* For Atomic Host Topics
 
== Atomic Host Working Group ==
 
This effort is co-owned by the [https://fedoraproject.org/wiki/Atomic/PRD Fedora Atomic Working Group]. The content and deliverables of Atomic Host affect the manner in which it is tested and delivered.


== Objective Lead ==
== Objective Lead ==


[[user:Stefw|Stef Walter]]
* [[user:dperpeet|Dominik Perpeet]] - Facilitating test generation, pipeline creation and wiring into pipeline
 
Dominik will work closely with [[user:dustymabe|Dusty Mabe]] on Atomic Host and Atomic Working Group communications, build tooling and wiring


== Timeframe ==
== Timeframe ==
Line 60: Line 94:


By Flock in September 2017, we will discuss the concrete results of this objective so far, and together imagine next steps for CI/CD objectives in Fedora.
By Flock in September 2017, we will discuss the concrete results of this objective so far, and together imagine next steps for CI/CD objectives in Fedora.
[[Category:FedoraAtomicCi]]

Revision as of 00:44, 12 October 2019

Objective: Deliver Fedora Atomic Host via CI/CD

Goal

We aim to deliver Atomic Host in Fedora via Continuous Integration and Continuous Delivery.

A continuous integration pipeline builds and composes Fedora Atomic Host artifacts exactly as they will be delivered and used (eg: in QCow2 and OSTree formats), and runs integration tests on it in that form.

Continuous integration pipeline will gate changes to packages. It will prevent a broken package change, as defined by integration tests, from affecting other developer's and/or testers of Fedora. The package maintainer will be able to change the tests on which their package is gated. The pipeline also gates broken combinations of package changes.

Continuous delivery will deliver the artifacts composed during integration step above to users of Fedora Atomic Host. It allows, but does not mandate constant or frequent delivery of changes to the Fedora mirrors. During delivery we can prevent unacceptable changes, due to policy or schedule, from reaching users.

More Details

Deliverables and Scope

During this objective, the focus and scope of CI will be on packages included in Atomic Host. These represent many of the core packages in Fedora. Testing and gating of build requirements is out of initial scope, and should be covered by future objectives.

In addition the delivery will be limited to composed artifacts of Atomic Host:

  • QCow2 images
  • Vagrant images
  • OSTree trees
  • Amazon Public Cloud images

The composes delivered by the CI/CD pipeline will not include full RPM based composes of Fedora at this point. Although the integration tests will help improve the quality of Fedora overall.

Key Results

Look for the following key results during progress on this objective:

  • 100 packages in Fedora Atomic host have at least one test stored in dist-git according to the spec.
  • Work with package maintainers to start to add additional tests or update tests when needed.
  • 100% of tests in Fedora Atomic Host packages pass on OSTree composes before a composed Fedora Atomic Host image is delivered.
  • Fedora Atomic Host artifact composes have no human interaction after the gating points.
  • Tests ported upstream from downstream sources have been reviewed by QE.
  • Tests check that Fedora Atomic Host works with key containers produced by Modularity effort.

Plans and Work Areas

Changes to how Atomic Host is built are needed. We have identifed several requirements that are needed for the CI/CD pipeline:

Requirements:

  • The definition of what is composed into Atomic Host artifacts should include specific versions of packages, and all dependencies included.
  • The definition of what is composed in an Atomic Host should be stored in a git repository so that changes can be detected easily. The CI/CD pipeline can be triggered off of changes to this repository.
  • A mechanism to make a future composed Atomic Host artifact, contain an earlier (in RPM NVR parlance) version of a package.
  • The high level functional Atomic Host tests should live in the same git repository with the rigorous definition of what goes into an Atomic Host.
  • A mechanism to tell the CI Pipeline that multiple dist-git repository changes (i.e. multiple changing RPMs) should be built and tested together.

It is possible the build and MBS parts of Modularity could be used to accomplish these requirements. Experimentation is needed in order to determine if there are any blockers for Atomic Host being built from a module.

CI Working Group

Much of the pipeline and testing effort on this Objective will be undertaken by a new "CI Special Interest Group". This group will invite people from the Fedora Engineering team, Fedora Release-Engineering team, Fedora QA, Security Team and more. Work will be coordinated when possible with the Factory 2.0 effort. Contact info:

Atomic Host Working Group

This effort is co-owned by the Fedora Atomic Working Group. The content and deliverables of Atomic Host affect the manner in which it is tested and delivered.

Objective Lead

  • Dominik Perpeet - Facilitating test generation, pipeline creation and wiring into pipeline

Dominik will work closely with Dusty Mabe on Atomic Host and Atomic Working Group communications, build tooling and wiring

Timeframe

This Objective aims to deliver Fedora Atomic Host in parallel with the current delivery methods. At a point at or around the Fedora release, we would aim for a cut over to using the CI/CD pipeline for delivery.

By Flock in September 2017, we will discuss the concrete results of this objective so far, and together imagine next steps for CI/CD objectives in Fedora.