Sgallagh/Changes/ELN Buildroot and Compose

ELN Buildroot and Compose

Summary

ELN is a new buildroot and compose process for Fedora that will take Fedora Rawhide dist-git sources and emulate a Red Hat Enterprise Linux compose. Feedback from this build, compose and integration testing will be provided to Fedora packagers so that they can see the potential impact of their changes on RHEL development.

ELN will allow us to explore new ideas like a higher baseline for CPU architectures in a way that will not disrupt the rest of Fedora. It will also aid us in future-proofing some of the spec file conditionals today that assume RHEL <= 8.

Owner

• Name: Aleksandra Fedorova
• Email: alpha@bookwar.info
• Name: Stephen Gallagher
• Email: sgallagh@redhat.com

Current status

• Targeted release: Fedora 33
• Last updated: 2020-04-06
• FESCo issue: #2365
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

This Change supersedes the previously-approved Change to enable an additional buildroot. During development, its scope has expanded to include the entire process of how the Fedora sources are built and composed into shippable artifacts.

ELN is an evolution of the request for an alternate buildroot for newer x86_64 processors. The reasoning behind that new buildroot was that we expected that the next major release of RHEL would likely drop support for older hardware and therefore could take advantage of enhancements and processor extensions available for newer hardware. As plans for this proceeded, they expanded into a desire to do more than just test out the processor architecture. Instead, we want to have an alternative compose of a portion of Fedora Rawhide that will resemble the way that Red Hat and CentOS builds their packages. The idea is that Fedora developers and third-party vendors who rely on Red Hat Enterprise Linux will have a place where they can directly contribute to what will eventually become the next RHEL.

ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a build environment, where the RPM conditionals (%{fedora} and %{rhel}) behave like the next RHEL. That way we can make existing spec conditionals future proof and pioneer various RHEL features in Fedora, while preserving RHEL-specific behavior where needed. (For example, Fedora often ships with experimental or less-commonly-used features enabled for packages that Red Hat would not want to support for Enterprise Linux customers).

The main goal of ELN is to rebuild Fedora Rawhide as if it were RHEL. See what works, what breaks, what changed. We will be doing this by creating a new set of Koji tags and targets to represent this new buildroot, adjusting some variables in RPM that can be used in conditionals and creating a new compose configuration to produce testable trees. We anticipate only building a relatively small subset of the Fedora Package Collection in ELN: mostly those that will be useful to CentOS and RHEL downstream. The new compose will be made available for testing in Fedora with a new fedora-repos-eln package. This package will provide a repo configuration file that sets the ELN repo to a lower priority value than the standard repos which means that its contents will be preferred over the standard Fedora repos when the same package exists in both. See the "How to Test" section for additional details.

ELN builds are not RHEL builds, to be clear. They are builds done in Fedora with a RHEL-like build configuration. Successful builds of ELN will be synced to the RHEL internal build system. At this sync point, we can apply additional changes that may have been unacceptable for Fedora (such as trademarks). We would like to minimize the divergence between ELN and RHEL, but we understand that this may not always be possible.

As a sister effort to this, the OSCI team will also be implementing automated tests that will run against ELN composes and provide non-blocking feedback to the package maintainers about potential issues in the Enterprise Linux configuration.

The RPM variables:

• %{dist} will return .eln (no numerical suffix)
• %{fedora} will be unset in ELN. (And thus return nothing)
• %{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).
• %{eln} will be set and will expand to %{rhel}

The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. There will be some unknown number of packages whose existing conditionals will need to be updated to account for ELN. At present, we are estimating that there will be around 200 packages that need such attention. The ELN SIG will be providing guidance and pull-requests to assist with this.

The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare.

Currently, we expect the Koji tagging structure to look like:

eln-candidate: empty tag          # packages get tagged 'eln-candidate' when built
eln: empty tag                    # packages are moved to the 'eln' tag from the gate tag

eln-build:                        # buildroot tag which inherits from Fedora Rawhide
-> eln
-> f33-build

and the target:
Target: eln
Build: eln-build
Dest: eln-candidate

Please note that these tags/targets may change at the recommendation of Release Engineering.

How does ELN relate to Rawhide?

ELN will inherit from the Rawhide branch in both the compose and buildroot. All Rawhide packages will be available for ELN to build against. If a package has not been built successfully for ELN, the latest Rawhide version of that package will be included in the ELN compose and buildroot.

One important requirement is that ELN builds must remain Rawhide-compatible, for any hardware that is compatible with ELN. It must always be possible to install packages from ELN atop a Rawhide installation.

What if packages don't build on ELN?

If the package has never been built successfully for ELN, then the Rawhide version of the package will remain in the ELN compose and buildroot until and unless an ELN build succeeds.

If the package has built on ELN previously, the most recent successful build will remain in the ELN compose. In some cases, it may be necessary to replace the older ELN version with the newer Rawhide version. In this situation, the Rawhide package can be manually tagged into ELN.

The vast majority of failed ELN builds will be because of incorrect or outdated %if directives, specifically those that are %if's on %{fedora} and/or %{rhel}. There might be a handful that occur for other reasons, but until we start doing ELN builds, we don't really know what they are. We will be incorporating feedback messaging into the ELN builds so that the maintainers and the ELN SIG will see what was attempted and what failed.

Will you be adding new %{rhel} or %{eln} conditionals to packages?

Regarding %{eln}, we intend not to use this variable if at all possible (an exception might be the kernel).

For packages that build successfully in ELN as-is or where using the rawhide build is sufficient, we won't be proposing new conditionals initially (with limited exceptions for packages such as kernel after discussion with the maintainers).

If a package fails to build on ELN, what will happen?

The failure will be looked at and we will evaluate how it can be fixed with the least change and/or impact.

Often this will be by changing an existing conditional such as

 %if 0%{?fedora}

to

 %if 0%{?fedora} || 0%{?rhel} >= 9

Or something that is less intuitive

 %if 0%{?fedora} < 32

to

 %if 0%{?fedora} < 32 && 0%{?rhel} <= 8

Such fixes will be done via a pull request that states a time limit. We want the regular maintainers to see / comment / commit, but we also don't want things to stall for months and get forgotten about. Please note that the time limit is to determine the active status of the maintainer. Once the maintainer is involved, there is no time limit. The maintainer always reserves the right to reject a suggestion.

For less trivial fixes, we may provide a more complex pull-request or use other methods to communicate with the maintainers to determine a path forwards that is acceptable to them. If the build failure is caused by the different environment, we will always evaluate how that difference can be properly fixed and we will fall back to adding conditionals only as a last resort (and only with approval). The goal is not to make ELN and Fedora different, but to make them more similar.

What if there is a new upstream feature RHEL wants in a package, how will that go in?

That is not in the scope of ELN. As always, this will be the responsibility of the maintainers to coordinate with the package upstream. While ELN may tweak which package options are enabled, it will not be a place to add net-new functionality.

What if I do not want to have %if's in my spec files?

As long as your package builds in ELN then just maintain your package like normal. If there is a build failure, the ELN SIG may provide a PR as described above or will discuss alternative approaches on an individual basis with you. If the package will never build in ELN, then the Rawhide version of it will simply be present.

The ELN SIG will not dictate how you must proceed, but will try to find a solution that you are comfortable with.

Can we do this in a branch instead of in master?

This adds no value to the current approach where Red Hat maintainers would manually merge their changes into the internal build infrastructure. There's currently no way to automate the sync from the master branch to the eln branch that wouldn't break and require maintainer involvement. Attempting to branch only individual packages would introduce significant complexity in the build process as well, leading to far more opportunity for bugs. Lastly, even the most diligent of maintainers can forget to sync every change to a new branch, thus leaving us in a situation where the eln branch has fallen behind and is no longer providing an accurate view of whether the package is still building or functioning in that environment.

What if I do not want to have %if's in my spec files, but I want to see how changes show up in RHEL?

If your package builds fine in ELN without conditionals, then you don't need to do anything special.

If it would require conditionals to build in ELN and you do not wish to put them there, but you want to see how changes appear in RHEL, there will eventually be a way to maintain a fork of your package that will go into RHEL, but the details on this are still being worked out.

How will ELN handle side-tags?

ELN will take no action on the creation or lifetime of a side-tag. When the side-tag is merged back to Rawhide, at this time ELN builds will be scheduled for each included package.

We will first untag any of the affected packages from the ELN buildroot so that the Rawhide versions will be present in it. We'll then rebuild all of those packages for ELN. There will be no need to build them in a specific order, since the Rawhide versions will be available. Any such builds that succeed will be appropriately tagged and any that fail will leave their Rawhide versions in the ELN compose until and unless they can be fixed.

Benefit to Fedora

Classically, Fedora sees a flurry of activity from Red Hat developers in the run-up to a new Red Hat Enterprise Linux release. These Red Hat teams frantically attempt to push a few years' downstream effort up into Fedora before it is branched for the next RHEL development. It is not uncommon for these teams to disappear downstream again once this has happened. This harms both Fedora and Red Hat with the lack of upstream involvement. With ELN, we hope to make Fedora Rawhide a more appealing (and canonical) place for those developers to work. This should increase the amount of ongoing developer involvement in Fedora, and also make Fedora a more valuable resource for Red Hat, which can help to ensure funding and support.

This new buildroot will allow us to experiment with new features without having a disruptive impact on the existing Fedora user-base. One example of such would be exploring setting a new processor baseline for performance improvements in server use-cases. By doing this in ELN, we gain a place to test whether the proposed changes to using AVX2 as a minimum has real performance impact which we could then look at bringing back to standard Fedora (as a new variant, perhaps). Or, if it proves that the performance gains really aren't worthwhile (or are nonexistent), we have learned that without interfering with other Fedora users.

Another similar example could be the usage of link-time-optimization.

Scope

• Proposal owners:
  • Setup build environment for a new disttag (eln).
  • Define and publish the Content Set for ELN, which will be a subset of Fedora Rawhide content that will be useful for the upcoming RHEL major release (presently 9)
  • Setup automation to trigger new ELN build every time there is a new update submitted to Fedora Rawhide.
  • Post build result to Fedora Messaging, so that it appears in Bodhi and can be used as a gating test.
  • Setup automation to run periodic partial composes (via ODCS) without installation media to generate repositories with these packages.
  • Create documentation for ELN compose and ELN project
  • Update packaging documentation to mention new disttag and how it can be used.
  • Setup ELN SIG. The SIG will
    • own the scope of the ELN component set;
    • own the scope of the eln disttag;
    • own the configuration of the compose;
    • maintain infrastructure required to build packages and composes;
    • track the status of package builds and composes;
    • work with other package maintainers to resolve the issues;
    • Keep track of and minimize divergence between ELN and Fedora.

• Other developers:

  Package maintainers who wish their package to diverge in ELN should coordinate their changes with the ELN SIG ahead of time.

• Release engineering: #9154

• Policies and guidelines: #955

• Trademark approval: not needed. There is no new shippable artifact.

Upgrade/compatibility impact

N/A

Though it is a System Wide change it doesn’t impact Fedora upgrades, as it is a development environment tracking Fedora Rawhide.

How To Test

There is going to be a development snapshot of the state of the ELN repositories.

It is under discussion whether this snapshot will have its own installation media. It depends on availability of resources for ODCS infrastructure, but also on ability of the ELN SIG to verify the installation media.

For now the preferred way to test ELN composes would be to use standard Fedora Rawhide images and then include ELN as an additional repository.

To enable ELN (once the repository is composed):

$ dnf install fedora-repos-eln
$ dnf distro-sync

To disable ELN and return to only the standard repos:

$ dnf remove fedora-repos-eln
$ dnf distro-sync

User Experience

ELN artifacts will be made available for testing and development purposes, but we will not be shipping any content produced from ELN directly to the general public (such as on the standard mirror network or via getfedora.org). This does not necessarily mean that they will be shipped directly from Koji, merely that they won't come from the standard locations.

Documentation on how to access ELN project repositories, on how to contribute to ELN configurations and on how to join ELN SIG will be available.

Dependencies

There is no dependency on other changes, though some of the infrastructure services like ODCS composes and Fedora CI will be involved.

Contingency Plan

• Contingency mechanism: This effort is not tightly coupled to the Fedora release cycle, but our intent is to land it during the Fedora 33 development period. At the F33/F34 Branch point, we will evaluate whether to continue with ELN efforts or to discontinue the project.
• Contingency deadline: Fedora 33 Branch (2020-08-11)
• Blocks release? No
• Blocks product? No

Additional risk: Infrastructure resources taken by ELN builds can create load on Fedora build system

Mitigation: In case of such load we cancel or reschedule ELN tasks (package and compose builds and tests) on request from the Fedora Infrastructure team. At least initially, we will also set ELN tasks to low-priority, similar to how COPR does it.

Documentation

https://fedoraproject.org/wiki/ELN

See also previous discussions:

• "Alternative buildroot as a development tool" @ devel

Release Notes

N/A

Though it is a System-Wide Change it has no impact on Fedora 33.