FedoraAtomicCI/gating

Why Gating?

Continuous Integration is a developer process. The developers in this context are mainly packagers. To have an effect, continuous integration must affect the packager workflow.

In large part, gating enables Continuous Integration to have an effect. It prevents the integration of broken changes that would affect other Fedora packagers.

Requirements

• It SHOULD be possible for a package in Fedora Atomic Host to opt into gating by CI.
• When a package is gated by CI, if a change to that package fails CI testing, then that change MUST NOT affect other Fedora packagers or later package changes.
• When a package is gated by CI, if a change to that package passes CI testing, then that changed package MUST be available during CI testing of later package changes.

Testing and gating released packages for users MAY be handled separately and be out of scope here. Fedora users will naturally benefit from gating in the developer workflow.

Where do we perform Gating?

In the Fedora packager workflow there are a few places where we could imagine adding gating.

dist-git

Our dist-git setup is composed of basic, plain git repos. There are a few limitations to it but very basic:

• users are not allowed to create random f* branches
• users are not allowed to create random el* and epel* branches (to prevent conflicts with the official ones)
• private branches can be created but not deleted (to ensure no history is ever lost)

CCL: Since dist-git is composed of git, there are no mechanism in place to do any gating there.

pagure

pagure is a git-centric forge that is being adjusted to be used as a front-end for dist-git. While not replacing the existing system, it would complement it by allowing contributors to use the now well-known pull-request contribution model. Users would fork their repo of interest, make changes and over them for inclusion via a pull-request that will be reviewed, commented and potentially merged by the maintainer of the package.

CCL: This would be a good place to implement gating with two remarks

• there are currently no timeline for putting pagure as a front-end for dist-git (though this is actively being worked on)
• the gating there would be only pre-merge and thus still a little far from what is being pushed to the users.

Koji

koji is the build system used in Fedora. For a specified git hash in a specified git repo it builds the desired artifact (rpm, container, module...).

This could be a place where gating could be done. A few ways this could be achieve:

• Only end the build once the test results are known, if the tests pass, the build succeed, if the tests fail, the build fails
  • requires writing a task to be done after the build and before that build is tagged, task that will listen to fedmsg for the test results
  • CONS: if the testing system is overloaded, this could take a really long time
  • CONS: it introduces a new gating in the workflow in a tool that is currently not a real gate
  • CONS: there is no place in koji for feedback, you would see that the build failed because one task failed but it will not be straight forward to access to the place giving you info about the test results.
• Allow the build to finish but post-pone tagging the build until after the test results are known (and only tag if they passed)
  • CONS: feedback wise this would be even worst as the build would pass but the tagging which would be an async process would not happen and the packager would have no idea why (tests still running? tests failed? tests still in their queue?)
  • CONS: The packager would not know when they can create the corresponding Bodhi update since they would not know when the tagging has occured (Note: even if the system notifies the packager who built the package, there are no guarantees that whoever builds the packager is the same person as the one creating the update in Bodhi)

CCL: Koji is not a natural place for gating and lack an easy way to integrate feedback to make it obvious why a build failed or wasn't tagged.

Bodhi

Bodhi is the current system gating the updates to allow pushing them to the mirror all at once and only when desired. It is also the place where testers and early adopters can give feedback on an update via a karma system. It is also the system allowing to bundle together related/depending builds.

This would be the most natural place to gate packages based on test results since it is already a gating system that Fedora packagers are used to.

Gating in Bodhi could be achieve using the following workflow:

• Packagers create the updates as they do today
• The testing system listens for the corresponding fedmsg messages, find out if any of the packages in the update is of interest and if so queue the builds for testing.
• The testing system sends a fedmsg message when a build is queued for testing, that message is caught by a fedmsg listener in Bodhi that flag the update as: TEST_QUEUED
• The testing system sends a fedmsg message when the test are started, that message allows Bodhi's fedmsg listener to flag the update as TEST_RUNNING
• The testing system sends a fedmsg message when the test are finished, that message allows Bodhi's fedmsg listener to flag the update as TEST_PASSED or TEST_FAILED
• The UI will reflect the current status the update is in.
• All updates who have a flag and that flag is not TEST_PASSED will not be excluded from any push to the repositories, effectively gating the update on test results

pros/cons:

• PROS: Bodhi is already a gating point in the Fedora packager workflow
• PROS: It is most likely, work-wise, the easiest solution to implement
• PROS: Gating is done close to delivery
• CONS: Gating is done further away from the change

CCL: Bodhi is already a gating place, we would introduce an extra step delaying or blocking the push to the repository