From Fedora Project Wiki
(Initial template of the page)
 
m (Jflory7 moved page Objectives/Continuous Integration and Delivery of Fedora Atomic Host to Initiatives/Continuous Integration and Delivery of Fedora Atomic Host: Change namespace: Objectives are retired and are now known as Initiatives. https://docs.fedoraproject.org/en-US/project/initiatives/)
 
(18 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{autolang|base=yes}}
== Objective: Deliver Fedora Atomic Host via CI/CD ==
== Objective: Deliver Fedora Atomic Host via CI/CD ==


=== Goal ===
=== Goal ===


We aim to deliver Atomic Host in Fedora via Continuous Integration and Continuous Delivery.
We aim to deliver Atomic Host in Fedora via '''Continuous Integration''' and '''Continuous Delivery'''.
 
A '''continuous integration''' pipeline builds and composes Fedora Atomic Host artifacts exactly as they will be delivered and used (eg: in QCow2 and OSTree formats), and runs integration tests on it in that form.  


A contintuous integration pipeline builds and composes Fedora Atomic Host artifacts exactly as they will be delievered and used (eg: in QCow2 and OSTree formats), and runs integration tests on it in that form.  
'''Continuous integration''' pipeline will gate changes to packages. It will prevent a broken package change, as defined by integration tests, from affecting other developer's and/or testers of Fedora. The package maintainer will be able to change the tests on which their package is gated. The pipeline also gates broken combinations of package changes.


Continuous delivery is about choosing specific integrated composes above and delivered exactly those artifacts to users of Fedora Atomic Host.
'''Continuous delivery''' will deliver the artifacts composed during integration step above to users of Fedora Atomic Host. It allows, but does not mandate constant or frequent delivery of changes to the Fedora mirrors. During delivery we can prevent unacceptable changes, due to policy or schedule, from reaching users.


=== More Details ===
=== More Details ===


TODO: Link to previous discussions.
* [[Fedora_requirements_for_CI_and_CD|Fedora requirements for CI/CD Pipelines]]
* [[CI|CI Landing Page]]
* Tests will be [[Changes/InvokingTests|stored in dist-git]]
 
=== Deliverables and Scope ===
 
During this objective, the focus and scope of CI will be on packages included in Atomic Host. These represent many of the core packages in Fedora. Testing and gating of build requirements is out of initial scope, and should be covered by future objectives.
 
In addition the delivery will be limited to composed artifacts of Atomic Host:
 
* QCow2 images
* Vagrant images
* OSTree trees
* Amazon Public Cloud images
 
The composes delivered by the CI/CD pipeline will not include full RPM based composes of Fedora at this point. Although the integration tests will help improve the quality of Fedora overall.


TODO: Link to test definitions.
== Key Results ==


=== Deliverables ===
Look for the following key results during progress on this objective:


TODO: Fill this section in.
* 100 packages in Fedora Atomic host have at least one test stored in dist-git according to the spec.
* Work with package maintainers to start to add additional tests or update tests when needed.
* 100% of tests in Fedora Atomic Host packages pass on OSTree composes before a composed Fedora Atomic Host image is delivered.
* Fedora Atomic Host artifact composes have no human interaction after the gating points.
* Tests ported upstream from downstream sources have been reviewed by QE.
* Tests check that Fedora Atomic Host works with key containers produced by Modularity effort.


The contents of Fedora Atomic Host should be consistent with previous
== Plans and Work Areas ==
releases.


== Special Interest Group ==
* Tests: Tests are to be included in dist-git in a standard format.
** Landing page: https://fedoraproject.org/wiki/CI
** Test Specification: https://fedoraproject.org/wiki/Changes/InvokingTests
** Tutorial: https://fedoraproject.org/wiki/CI/Tests
* CI/CD Integration into Fedora: Work will continue to integrate CI and CD into Fedora infrastructure.
** [[FedoraAtomicCI/KojiBuilds|Integration with Koji]]
** Gating of packages on test results
** Displaying test results from Pipeline
** Enabling multiple contributors to a dist-git repository
* CI Pipeline: For running tests
** Based on Openshift and Jenkins
** Source Code: https://github.com/CentOS-PaaS-SIG/ci-pipeline
** Documentation: https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md


TODO: Describe SIG
Changes to how Atomic Host is built are needed. We have identifed [https://pagure.io/atomic-wg/issue/312 several requirements] that are needed for the CI/CD pipeline:
 
Requirements:
* The definition of what is composed into Atomic Host artifacts should include specific versions of packages, and all dependencies included.
* The definition of what is composed in an Atomic Host should be stored in a git repository so that changes can be detected easily. The CI/CD pipeline can be triggered off of changes to this repository.
* A mechanism to make a future composed Atomic Host artifact, contain an earlier (in RPM NVR parlance) version of a package.
* The high level functional Atomic Host tests should live in the same git repository with the rigorous definition of what goes into an Atomic Host.
* A mechanism to tell the CI Pipeline that multiple dist-git repository changes (i.e. multiple changing RPMs) should be built and tested together.
 
It is possible the build and MBS parts of [[Modularity|Modularity]] could be used to accomplish these requirements. Experimentation is needed in order to determine if there are any blockers for Atomic Host being built from a module.
 
== CI Working Group ==
 
Much of the pipeline and testing effort on this Objective will be undertaken by a new "'''CI Special Interest Group'''". This group will invite people from the Fedora Engineering team, Fedora Release-Engineering team, Fedora QA, Security Team and more. Work will be coordinated when possible with the Factory 2.0 effort. Contact info:
 
* #fedora-ci on FreeNode
* Mailing list: [https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/ ci@lists.fedoraproject.org]
* For CI topics if you don't know where to go: [[user:Stefw|Stef Walter]] [[user:pingou|Pierre-Yves Chibon]]
* For Atomic Host Topics
 
== Atomic Host Working Group ==
 
This effort is co-owned by the [https://fedoraproject.org/wiki/Atomic/PRD Fedora Atomic Working Group]. The content and deliverables of Atomic Host affect the manner in which it is tested and delivered.


== Objective Lead ==
== Objective Lead ==


* [[user:dperpeet|Dominik Perpeet]] - Facilitating test generation, pipeline creation and wiring into pipeline


[[user:Stefw|Stef Walter]]
Dominik will work closely with [[user:dustymabe|Dusty Mabe]] on Atomic Host and Atomic Working Group communications, build tooling and wiring


== Timeframe ==
== Timeframe ==


TODO: More details
This Objective aims to deliver Fedora Atomic Host in parallel with the current delivery methods. At a point at or around the Fedora release, we would aim for a cut over to using the CI/CD pipeline for delivery.
 
Fedora Atomic 27


Describe Plan B around Atomic Host.
By Flock in September 2017, we will discuss the concrete results of this objective so far, and together imagine next steps for CI/CD objectives in Fedora.


By Flock in September (also of this year), ...
[[Category:FedoraAtomicCi]]

Latest revision as of 14:06, 7 February 2024

Objective: Deliver Fedora Atomic Host via CI/CD

Goal

We aim to deliver Atomic Host in Fedora via Continuous Integration and Continuous Delivery.

A continuous integration pipeline builds and composes Fedora Atomic Host artifacts exactly as they will be delivered and used (eg: in QCow2 and OSTree formats), and runs integration tests on it in that form.

Continuous integration pipeline will gate changes to packages. It will prevent a broken package change, as defined by integration tests, from affecting other developer's and/or testers of Fedora. The package maintainer will be able to change the tests on which their package is gated. The pipeline also gates broken combinations of package changes.

Continuous delivery will deliver the artifacts composed during integration step above to users of Fedora Atomic Host. It allows, but does not mandate constant or frequent delivery of changes to the Fedora mirrors. During delivery we can prevent unacceptable changes, due to policy or schedule, from reaching users.

More Details

Deliverables and Scope

During this objective, the focus and scope of CI will be on packages included in Atomic Host. These represent many of the core packages in Fedora. Testing and gating of build requirements is out of initial scope, and should be covered by future objectives.

In addition the delivery will be limited to composed artifacts of Atomic Host:

  • QCow2 images
  • Vagrant images
  • OSTree trees
  • Amazon Public Cloud images

The composes delivered by the CI/CD pipeline will not include full RPM based composes of Fedora at this point. Although the integration tests will help improve the quality of Fedora overall.

Key Results

Look for the following key results during progress on this objective:

  • 100 packages in Fedora Atomic host have at least one test stored in dist-git according to the spec.
  • Work with package maintainers to start to add additional tests or update tests when needed.
  • 100% of tests in Fedora Atomic Host packages pass on OSTree composes before a composed Fedora Atomic Host image is delivered.
  • Fedora Atomic Host artifact composes have no human interaction after the gating points.
  • Tests ported upstream from downstream sources have been reviewed by QE.
  • Tests check that Fedora Atomic Host works with key containers produced by Modularity effort.

Plans and Work Areas

Changes to how Atomic Host is built are needed. We have identifed several requirements that are needed for the CI/CD pipeline:

Requirements:

  • The definition of what is composed into Atomic Host artifacts should include specific versions of packages, and all dependencies included.
  • The definition of what is composed in an Atomic Host should be stored in a git repository so that changes can be detected easily. The CI/CD pipeline can be triggered off of changes to this repository.
  • A mechanism to make a future composed Atomic Host artifact, contain an earlier (in RPM NVR parlance) version of a package.
  • The high level functional Atomic Host tests should live in the same git repository with the rigorous definition of what goes into an Atomic Host.
  • A mechanism to tell the CI Pipeline that multiple dist-git repository changes (i.e. multiple changing RPMs) should be built and tested together.

It is possible the build and MBS parts of Modularity could be used to accomplish these requirements. Experimentation is needed in order to determine if there are any blockers for Atomic Host being built from a module.

CI Working Group

Much of the pipeline and testing effort on this Objective will be undertaken by a new "CI Special Interest Group". This group will invite people from the Fedora Engineering team, Fedora Release-Engineering team, Fedora QA, Security Team and more. Work will be coordinated when possible with the Factory 2.0 effort. Contact info:

Atomic Host Working Group

This effort is co-owned by the Fedora Atomic Working Group. The content and deliverables of Atomic Host affect the manner in which it is tested and delivered.

Objective Lead

  • Dominik Perpeet - Facilitating test generation, pipeline creation and wiring into pipeline

Dominik will work closely with Dusty Mabe on Atomic Host and Atomic Working Group communications, build tooling and wiring

Timeframe

This Objective aims to deliver Fedora Atomic Host in parallel with the current delivery methods. At a point at or around the Fedora release, we would aim for a cut over to using the CI/CD pipeline for delivery.

By Flock in September 2017, we will discuss the concrete results of this objective so far, and together imagine next steps for CI/CD objectives in Fedora.