The build, release, distribution, and update changes associated with and required for the [Changes/Modular_Server] and [Changes/Host_and_Platform] Changes.
- Name: Ralph Bean
- Email: firstname.lastname@example.org
- Release notes owner:
- Edition: Fedora Server
- Responsible WG: Modularity WG
- Targeted release: Fedora 27
- Last updated: 2017-06-20
- Tracker bug: <will be assigned by the Wrangler>
This change is intended to cover the workflow and technical tooling aspects of a “modular” release for F27.
There are other Changes that are not part of the scope of this Change, but which are related:
- Changes/Modular_Server is a proposal to replace the Fedora Server edition with a modular release for F27.
- Changes/Host_and_Platform deals with the content of what goes into the core of the modular release for Fedora Server (the Modular Server) in F27.
This Change is about how we’re going to get those two other changes through the infra/build/release tooling.
Client tooling is being worked on (I have seen some cool demos from the May-June timeframe. Ask about them!), but client tooling is not part of the scope of this Change proposal.
Note that there currently are no proposals to replace either Fedora Workstation or Fedora Atomic with modular revisions. This means that here we cannot move to replace existing workflows wholesale, but have to look at maintaining the two flows (traditional and modular) in tandem, for at very least the F27 release cycle.
(The Changes/ArbitraryBranching change is also related to this proposal, but not covered in any further detail here.)
Module builds for the F27 cycle will look much like they did for the F26 cycle. No major changes here.
See the F26 Change document for details here, Changes/Module_Build_Service
As soon as the FPC approves the Module:Guidelines, we can open up the MBS for use by the general packager group.
- We need the Modularity team to take the module guidelines to the FPC, work through them, and get an agreed-upon version approved.
- We are indirectly dependent on release engineering to create some initial tags for bootstrapping the host and platform content. This should be referenced in that Change proposal.
Ok, I lied. One thing about builds will change: we’re going to automate rebuilds with Infrastructure/Factory2/Focus/Freshmaker.
For F26, if a packager wanted to update an rpm in a module, they would:
- Patch the spec file of that rpm, commit, and push.
- Switch to a checkout of the module that included that rpm, and run
mbs-build submitwhich would kick off the appropriate builds.
- If another module included that same rpm stream, and the packager forgot about it, then they’re just out of luck.
For F27, we’re working on a system to automate this. The workflow will be:
- Patch the spec file of an rpm, commit, and push.
- Watch the fireworks.
The freshmaker daemon will notice the commit, then look up all modules that depend on that rpm stream. It will submit requests to the module-build-service to build those modules.
We won’t have a nice UI for this for F27, but we will have a JSON API provided by freshmaker to query and find the list of module builds that were triggered as a result of your commit (or anyone else’s commits to any other packages).
There are some exceptions here. We will have a site-wide policy configured for freshmaker to not do automated rebuilds for a blacklisted set of modules. This blacklist set must include the bootstrap module. It includes many thousands of rpm streams and an update to any one of them would cause a mass-rebuild of (nearly) every other module. This is too much, so we’ll instead rely on the bootstrap maintainers and release engineering to only request these rebuilds via MBS by hand.
- We’ll need Fedora Infrastructure to kindly support us through the request-for-resources process. In particular, freshmaker will need a kerberos keytab to submit builds to the MBS.
We will continue to do composes with the same tool - pungi.
- For rawhide, the compose configuration should list the master streams of all modules to be included in the rawhide compose. The latest builds in those streams will be included in each nightly compose: https://pagure.io/pungi-fedora/blob/master/f/variants-modular.xml
- For branched, the compose configuration should list f27 streams of all modules to be included in the F27 server compose. The latest builds in those streams will be included in each nightly compose.
- For the alpha and beta candidates, release engineering will need to pin the versions of the module streams in the variants-module.xml file in the f27 branch of the pungi-fedora repo. This will allow them to stabilize the release and allow for systematic QA and go/no-go decisions.
- The image building work from F26. For F26 "Boltron", we wanted to produce base images but ran into more issues than expected. Most all of those are resolved now. Full completion of that work (being able to produce base images in koji via pungi from modular content) is a hard requirement for F27, where it was a soft requirement for F26.
- The variants files for F27. The Factory2 team will produce these and submit them to release-engineering's pungi-fedora repo. Releng will need to review and eventually merge.
- As the beta cycle unfolds, Release Engineering will need to freeze the module versions in variants-modular.xml, adding in new versions only to address blocker bugs (like how we do today with the f26-compose tag).
Updates to the (modular) Fedora Server release will be managed by Bodhi.
- The Bodhi database model now has multi-type (modular) support.
- The Bodhi API now has multi-type (modular) support.
- The Bodhi bindings and CLI and web UI now have multi-type (modular) support.
The only thing left to do is (take a deep breath) the masher.
See Infrastructure/Factory2/Focus/Bodhi for how to plan to handle the code changes to Bodhi.
As for configuration, we’re going to need to be able to mash updates for traditional style Fedora Workstation and Fedora Atomic while also mashing updates for (Modular) Fedora Server.
We will need:
- f27-updates, f27-updates-testing and all the normal associated -pending and -candidate tags for Workstation and Atomic. The F27 “release” in Bodhi’s database will point at this tags, as per usual.
- f27-modular-updates, f27-modular-updates-testing, and similar kinds of -pending and -candidate tags for Server. We will then need to additionally create a separate Modular F27 “release” in Bodhi’s database that points to those tags.
The masher will produce separate modular-updates and modular-updates-testing repos alongside the traditional updates and updates-testing repos.
- We need the content generator flags enabled in koji for the MBS (see https://pagure.io/releng/issue/6799 ).
- We need the tags hierarchy created for modular updates.
- We need the release created in the bodhi DB (for the modular repos) at the same time that the traditional release is created.
We don’t expect that mirroring this content will require any changes to mirrormanager or mirrorlist.
We will be creating an extra compose and an extra set of updates repos, but there are no fundamental changes to the mirroring process. We will have an additive increase in load on mirrors, but no an exponential one.
There has been lots of worry here. Let me explain. Once upon a time, we thought that (with Modularity) we would distribute every version of every module ever built. Users could switch between any of them, at any time, since they would all be available on the mirrors. This would explode our storage ask of the mirror volunteers.
But, this is not how our modularized bodhi masher will work. We will distribute multiple streams of the same rpms, but we will not retain or mirror multiple versions of the same stream. Only the latest builds of a module stream will be included in the updates repo, not every version we ever built.
Benefit to Fedora
Pull together all of this modularity stuff.
- Proposal owners: We have to finish and deploy freshmaker, our bodhi changes, including changes to pungi. We'll be actively working with the infrastructure and releng teams on deployment and configuration.
- Other developers: Is there any work required of other developers? No. They will still be able to use the "traditional" workflow to build and ship content for Fedora Workstation and Fedora Atomic. To participate in the development of Fedora Server, they'll need to learn how to create new modules (the module guidelines should cover this).
- Release engineering: ticket #6852
I don't think anything on this list changes. See Changes/Modular_Server.
- Policies and guidelines: See Changes/Modular_Server.
- Trademark approval: N/A (not needed for this Change)
How To Test
If the user can install the Fedora 27 Server and we can deliver updates, then it is a success.
See the dependencies bullets in the "Detailed Description" section above.
- Contingency mechanism: Since we're going to keep the traditional flow for Workstation and Atomic for F27, we should also have the chance to ditch all this stuff and produce Server in the same way at any point (provided we have enough time to stabilize the compose).
- Contingency deadline: 2017-08-29 (the Bodhi activation point).
- Blocks release? Yes
- Blocks product? Fedora Server