User:Lmacken/AtomicWorkflowDraft

Blockers / dependencies for redoing the Atomic workflow (try to prioritize these projects):

• PDC implementation
• https://fedoraproject.org/wiki/Changes/ProductDefinitionCenter
• Implement a pdc-updater service that listens to fedmsg and updates PDC accordingly
• Map packages to each compose artifact
• Ability to automate signing
• https://fedorahosted.org/sigul/
• Sigul needs development resources in order to fix existing bugs and add new features needed for automation.
  • Server and bridge regularly crash when releng is signing rpms in batched mode https://bugzilla.redhat.com/show_bug.cgi?id=1272535
• Additional Koji artifacts
• Mashed repositories https://fedoraproject.org/wiki/Changes/KojiSignedRepos
• OSTrees
• Taskotron & consolidating Tunir & Autocloud
• Implement tests for each compose artifact
• Listen to fedmsg and automatically trigger the appropriate tests
• Write tests for the installer, AMIs, etc.
• Integrate Atomic CI jobs http://git.app.eng.bos.redhat.com/git/atomic-ci-jobs.git

Things that are not blocked (add to the short term epic):

• better test case coverage
• better ostree management
• Single Atomic Host OSTree repository. https://fedorahosted.org/rel-eng/ticket/6125
• creating new ostrees starts with an empty tree and merges this into master if there are changes
• tagging, garbage collection, etc
• automated build, compose, test, release process for all artifacts: pxe2live, AMIs, Installer ISO, GCE (?)

Functional Requirements

• Automated build/compose pipeline
• Atomic Host build artifacts should only be refreshed/rebuilt when something in their package manifest changed/updated in a new compose
• Automated test pipeline
• All Atomic Host build artifacts should be automatically tested with an acceptable level of test coverage that we trust the resulting "successful" marked artifacts are ready for release
• Automated Release pipeline
• Once all of the artifacts pass the automated tests, they are then automatically signed and released.
• Websites
• The Cloud download website should be automatically updated with the new content as it's released. https://stg.getfedora.org/en/cloud/download/atomic.html
• Updates
• Updates for the components of Atomic Host's OSTrees should have automated tests in taskotron
• Updates for the OSTree of Atomic Host should be refreshed as soon as one of the components of it's manifest hit's the Fedora or Fedora Updates stable repository.

"Releng-o-tron" (for lack of a better name)

A fedmsg consumer that listens for changes, queries PDC for products that are affected, and triggers composes & automatic tests accordingly.

Updates/updates-testing Workflow

• Package maintainer commits package changes to git, builds it in koji, and submits an update to bodhi
• Taskotron automatically kicks off the RPM-based tests
• Updates are signed (currently by hand, which should eventually be automated)
• Bodhi "pushes" the update in a batch with others, which currently involves mashing a repository and composing the Atomic OSTrees.
• Ideally, we want to create koji tasks for both mash and ostree composes, and have bodhi simply send a single fedmsg with the batch of updates listed, and wait for releng-o-tron to do the coordination.
• Releng-o-tron picks up the fedmsg with all of the updates for the push
• Triggers mashing updates/updates-testing tasks in koji
• For each update in the batch, determine which artifacts are affected by querying PDC
• Upon completion of the repositories, trigger artifact composes against them
• Send a fedmsg upon completion. Really, it will store the results in a releng-o-tron "resultsdb" and the resultsdb is already instrumented to send a fedmsg.
• Task-o-tron picks up the fedmsg and tests the artifacts, and kicks off tests accordingly
• Tests OSTree, ISO, AMI, etc
• After Taskotron gives everything the "thumbs-up" (via fedmsg), releng-o-tron symlinks/rsyncs everything live, and then sends a fedmsg
• Bodhi picks up the fedmsg and "finishes" the push (sends emails, updates/closes bugs, comments on updates, etc)
  • A question: do we then only "finish" a push after compose artifacts are completed? That's new territory if so. We currently only finish a push when both the mash and rpm-ostree are completed anyway. So it's already an "all or nothing" model.

Rawhide Gating

• Packagers build packages in a 'rawhide-pending' tag, or something like that. We disallow building in the traditional rawhide tag directly to enforce this.
  • We could potentially streamline the workflow of getting new upstream releases into rawhide and tested with anitya->new-hotness
• koji publishes a message when a new rawhide-pending build is built.
• qa taskotron notices this and runs some series of integrity checks on rawhide-pending. It publishes a message about this.
• If good, releng-o-tron notices qa-taskotron's result and it moves all the rawhide-pending builds into rawhide-proper (or whatever its called).
• If bad, it moves suspect builds from rawhide-pending into rawhide-badnewsbears.
• qa-taskotron can be set up to notice this as well, and start the integrity-check again.
• We can configure FMN to send emails to packagers when their packages go to rawhide-badnewsbears.

Requirements

• releng-o-tron should know how to compose everything, ideally with a simple plugin framework that can allows us to easily add new artifacts.
• releng-o-tron should be the final gate-keeper of all of the bits
• It should perform/trigger the syncing of bits to the master mirror, instead of relying on a cron job to do so.
• We should ensure no fedmsgs ever get dropped (via gilmsg)
• Along with knowing how to compose OSTrees, it should also be able to handle merging, tagging, and archival/garbage collection of branches.
• Streamline the End Of Life process. With a single command/click of a button it should update the pkgdb, update bugs, carve out OSTrees, and archive content.

Questions

• Do we want to use the Bodhi web interface to list all of the "testing" compose artifacts and facilitate community testing/feedback/karma?

Concerns

• We may need to rethink how we push bits to the users. With our current mirroring setup, there are some mirrors that only pull twice a day. If we're hoping to asynchronously push bits out all day, we might want to investigate a proper CDN setup.