Changes/OnDemandSideTags

On-demand Side Tags

Summary

Allow on-demand side tags, and allow packagers to a) tag whatever rpms as build dependencies into those side tags, b) build packages they want in those side tags, and then c) tag all or some of those new packages back into appropriate “main” tag (e.g. fNN-updates-candidate).

This new process will allow packagers to build rpms using older or newer versions of build dependencies than the buildroot in a given Fedora release, providing a subset of Modularity features in an additive way (no influence on the way dependent packages are built or installed). It may also coexist with Modularity in its current form.

As a policy change, default modules are disallowed and modules are only opt-in.

Owners

• Name: Zbigniew Jędrzejewski-Szmek
• Email: zbyszek at in.waw.pl

• Name: Miro Hrončok
• Email:

Current status

• Targeted release: Fedora 32
• Last updated: 2019-10-16
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Motivation

Modularity in its design and current implementation provides a number of features that can be divided into two: for people building rpms in modules, and for people installing rpms from modules. There have been a number of issues, RFEs, and discussions about modularity, but they are mostly about the second, user-facing part. Rpms installed from modules interact with non-modular rpms, and of course every aspect of this interaction is visible to users. To make Modularity a success, we would not only need to make this interaction well-behaved, we would also need to explain it to users in a satisfactory way.

The first part, module build internals, is not visible to users. Some packagers don’t like the syntax, which is opaque and best-suited for programmatic generation and consumption. Constant rebuilding of modules also creates noticeable load on the infrastructure, making everything else slower. If modularity became much more popular, and a significant number of packagers was using it, we are convinced many discussions would start about this first part too.

Despite some problems, there is clear demand for the features that Modularity brings. In particular, we need some solution for the "too fast too slow" problem, where some packages cannot update fast enough to maintain compatibility with the rest of the distribution.

New approach: on-demand side tags

The new process proposed here provides an alternative to both parts of Modularity.

The second part of Modularity is simply removed: users only get normal rpms. After rpms have been built, some or all of them are merged into the normal tags and delivered to users as usual. This is pretty much the same as the usual rebuild of specific language or graphic stacks in side tags that we do already. Dependencies between rpms must be expressed using Requires/Recommends/Provides/Suggests/Conflicts.

The first part is replaced by a simpler scheme: we don’t define any specific syntax or repositories to build packages. The packager has to issue a sequence of 'fedpkg tag-build', 'fedpkg build', and 'koji wait-repo' commands as appropriate (or 'fedpkg chain-build' or …). The packager may link an older rpm into the side tag, allowing building against old versions of packages. This means that if the packager is not a committer to the package, they may only use a version which was previously built in koji. The packager selects which rpms should be merged back into the main tag. Old packages obviously cannot be merged back.

To minimize disruption for users and current packagers, modules are required to be opt-in only. There are no default modules and non-modular rpms cannot depend for build or installation on modular rpms. This way there is less burden to other packagers and Modularity can evolve more easily.

Implementation

Ability to create side tags “on demand”, i.e. with an automated API call must be implemented. Koji and other infra tools would need to be able to deal with a large number of tags. Modularity already creates a large number of tags, so this should not be a significant change.

Packager tools like fedpkg must grow new commands to create side tags.

Both the workflow in this proposal, and modules as they currently are would be allowed to coexist. After we have some experience with both approaches, one or both could be retired.

Since this proposal removes the possibility to have default modules, modules need to be reset. The details are not specified yet. One approach would be to do the equivalent of dnf module reset all during upgrade to F32, similarly to the proposed solution for https://bugzilla.redhat.com/show_bug.cgi?id=1747408.

This proposal assumes that fedpkg will be enhanced to do extended chain builds smartly. This is not a precondition: even without that packagers may script something that is appropriate for them. This is one of the main advantages of the proposal: we put the “smarts” in local tooling which can be enhanced and modified without breaking what other packagers do. E.g. a solution to rebuild all 500 rust packages with new versions may be significantly different than building a compat version of postgresql, and that is OK.

Buildroot only packages

They are possible, because not all packages are merged back. The same is currently possible with side tags, and with non-rawhide builds for which packagers may never create a bodhi update, so there is no fundamental change. Whether or not we need to disallow packages which are purposefully buildroot-only by a policy is left out of this proposal on purpose. Technically, it is possible.

Benefit to Fedora

Costs, for packages converted from modules to this scheme

Transitional costs: according to this proposal, modules may still be used. Nevertheless, if the contents of a module are to be used to build normal rpms, that module will need to switch to the new process or simply demodularized and built in the traditional way. In the case where this happens, packaging would need to be “translated” from module .yaml to normal spec files. This should mostly be about adjusting metadata, i.e. package versions and dependencies.

We would lose one of the main advertised features of Modularity: the ability to build the same module in different versions. This might not be a huge loss, for two reasons. First, great majority of modules are single-version. Second, if desired it is possible to provide compat packages with different names. This is a bit of annoying work, but we have the process very well understood. Compat packages have the additional advantage that they may be made co-installable (vide python26, python2, python35, python36, python3, python38 packages currently in F31). Compat packages are exempt from package review. We may consider additional simplifications to the process to make compat packages easier.

Benefits

Vast simplification for other packagers, i.e. those who would like to depend on rpms which are currently modularized, and for users, who do not have separate modular repos, just a single repo like before.

Reduced load on the build infrastructure.

Easier to find rpm sources because there again is one canonical repo.

Third parties building on top of Fedora can use our modules to build their own rpms without trouble.

dnf does not need to handle complicated logic. It should be faster if less repos are used.

Rpms are built against an up-to-date buildroot, with only the minimal compat package inserted, instead of some older-than-all-current-releases Fedora release.

Normal package-level package replacement and obsoletion mechanisms are used.

Scope

• Proposal owners: detailed documentation how to do things in the new scheme, client-side tooling improvements.

• Other developers: demodularize specific packages if wanted or necessary.

• Release engineering: implement the server side of side tags on demand [RELENG TICKET WILL BE FILED LATER]

• Policies and guidelines: the docs would need to be added to packager documentation. Pull requests will be submitted by proposal owners.

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

This change should be fully additive to normal rpm packaging, so there should be no upgrade/compatibility issues.

Some modules will need to be demodularized. This will require packager work. Once this happens, rpms built this way will be fully available to build normal rpms and modules, so the upgrade impact should be limited to the modules being rewritten.

How To Test a.k.a. examples of the new workflow

Version bump in a language stack

It is determined that a new set of rust rpm versions should be built. A member of the rust-sig bumps the versions in dist-git for all relevant packages, creates a side tag, and does the huge chain-build. Once finished, this side tag is merged into rawhide-build. If desired, it may also be merged into f31-updates-candidate and/or f30-updates-candidate and bodhi updates submitted.

rust-sig already has tooling to generate the appropriate build-order for modularity’s .yaml file. The same code can be reused to generate the order for the chain build.

Three streams of a database package

A security patch needs to be applied to three versions of postgresql. There are postgresql, postgresql82, postgresql81 dist-git repos. The patch is added in dist-git for each of the packages, and they are rebuilt as appropriate. Since we want to build each package with different compiler versions, the packager uses a bash loop to schedule separate builds of each of the three packages in each of the supported Fedora versions.

(Once we have automatic creation of bodhi updates from koji builds, the packager will not have to do anything more. Right now they still need to create one big update that will then be split into separate updates for each Fedora release.)

A small advantage compared to current modules is that packages are not built against an already obsolete Fedora base, so users get the benefit of toolchain improvements.

Package needs some old dep to build

Poor packager needs gcc-9.0, at the time where gcc-9.1 is the default in the build root, to build koo. They bump the version of koo in dist-git, create a side tag, tag old gcc-9.0-12.fc31 build into the tag, build koo, merge the built package back into f32-build and f31-updates-candidate.

Package needs some old dep to install

koo-gui is only compatible with python3-qt5-5.12 when python3-qt5-5.13 has been released, and cannot be built with the new version. koo maintainers may either a) arrange for python-qt5_5_12 compat package to be created and then use it in the build root and as installation dep, b) install old python3-qt5-5.12 package in build root, and bundle some contents of that package into its output rpms, and add Provides:bundled(python3-qt5)=5.12.

FAQ for “other developers”

I maintain several modular streams of my package additionally to the default version in Fedora. Can I keep doing this?

Yes, you can keep any alternate version in modular branches. When appropriate, consider compat packages that can often be co-installable, unlike modular streams.

I maintain my packages as modular only. Can I keep doing this?

Yes, as long as those modules are “leaves”, in the sense that no non-modular rpms depend on them.

I would like to use an rpm which is currently only available as module as a build dependency

The owner of the module should be asked to demodularize the module (and either convert it to fully traditional rpms, or build some rpms using the new side-tag process). Those rpms can then be used as build- and installation-time dependencies. If the owner of the module is not interested in providing non-modular rpms, other packages may add such rpms to the distribution (and again build them in one of the two ways).

Which rpms can be used as build dependencies in the side tag?

In principle, any rpm which is available in koji (including the recursive dependencies, so it can actually be installed). In addition, the packager may build rpms and use them in the side tag build root. In practice, we expect that packagers will use one of the Fedora releases (with updates), as the base.

User Experience

Users have a clearer picture where their rpms are coming from. If they opt-in into a module, they get some additional content. Without that, they only get rpms from the normal Fedora repository. They are able to rebuild rpms locally if there are no version conflicts. In case of version conflicts, they might need to install older versions of rpms from koji to satisfy the build (which might sound bad, but really isn’t much different from current state with normal rpms.)

Dependencies

On-demand side tags. Without this, this would be completely infeasible. For limited and initial testing, side tags could be created manually.

Contingency Plan

In the implementation phase, this should not impact other rpms in any way (since it is change really happens on the build farm).

Once deployed, and rpms are built using this approach, they would still be installable, even if the new process is abandonded. So there is no flag day for revert, but some packages would effectively become FTBFS. Any rpms which can only be built in this new scheme (e.g. because they require older packages that the current ones in the default build root), would have to be either fixed to build in the normal build root, or would have to be modularized.

If this approach is abandoned, other approaches including the proposal in https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot may be implemented, but that is only possible in the longer term.

• Contingency mechanism:

1. disable side-tag creation in koji (releng). Hopefully this would be just a single switch somewhere. Any code that is added to koji can be removed later (koji developers).

2. undo any documentation changes

3. adjust packages as described above (change owners, package owners). This is very hard to say how big the scope of this would be, because it is unclear how many packages would be involed. I think we can assume that similarly as with Modularity, it would be <10 in the first Fedora release where this is available.

• Contingency deadline: TBD
• Blocks release? Yes
• Blocks product? N/A

Documentation

Will be submitted by Change owners.