QA:SOP compose request

This SOP explains the procedure for requesting composes from the ReleaseEngineering team. This task is usually performed by QA.

What composes should be requested when

There are three milestones in the Fedora release cycle: Alpha, Beta, and Final. For each milestone, a minimum of one release candidate (RC) must be produced. At least one test compose (TC) should also always be produced under normal circumstances.

The Fedora schedule specifies dates for the first test compose and for the first release candidate, for each milestone: the compose dates are listed on the release engineering schedule and usually also on the QA schedule. The requests should be filed on or (ideally) shortly prior to these dates. However, a release candidate can only be composed when all blocker bugs are addressed, and this requirement supersedes the schedule: see below for more on this.

The schedule for the next Fedora release, with the designated compose dates, can be found here.

Test composes

A test compose is defined as a set of Fedora images built, from the current Branched stable tree (not including updates-testing), shortly prior to the Change Deadline (freeze) for one of the three Fedora milestones (Alpha, Beta and Final), for the purposes of performing release validation testing. It differs from a release candidate in that it is built before, not after, the Change Deadline and hence there is no possibility of its being declared gold and released as the Alpha, Beta or Final release.

Test composes beyond TC1 are done at the discretion of the QA team: there is no strict requirement for a certain number of test composes, or a strict trigger for building a new test compose. It usually makes sense to request a new test compose when significant changes to key components such as the installer, the kernel, X.org, the default bootloader and so on would be included: simply put, request a new test compose when it becomes clear that it may provide significantly different results to the current test compose.

There is no requirement for blocker bugs to be addressed in order for a test compose to be produced. It is a good idea, however, to make a best effort before filing a TC compose request to ensure that the compose will succeed and result in images that do not fail immediately - if you know there is a package conflict in the repositories that will result in live image composes failing, or a showstopper bug which makes the installer crash at the first page, there is very little point in requesting a TC compose. It is better for TC1 to be a day or two late than for it to be on time but untestable.

Release candidates

A release candidate is defined as a set of Fedora images built, from the current Branched tree (not including updates-testing), after the Change Deadline (freeze) for one of the three Fedora milestones (Alpha, Beta and Final) and using a package set which is not known to contain any blocker bugs, for the purposes of performing release validation testing. It differs from a test compose in that it is built after the Change Deadline and may be declared gold and released as the Alpha, Beta or Final release if it passes all validation tests.

A release candidate - including release candidates beyond RC1 - can be requested only after the Change Deadline (and only when the freeze has actually been put into place) and when all accepted blocker bugs are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the classic example is a bug in preupgrade, because in this case, a bug in Fedora N can block the release of Fedora N+1. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the update testing process and been marked as stable (and hence before the bug is actually closed).

The procedure for release candidates is to request the RC1 compose as close as possible to the target date, but only after all accepted blockers are addressed. RC1 can be requested early if all accepted blockers are addressed, but it must be requested after the freeze date ('Change Deadline' on the schedule pages).

If testing of RC1 results in further blocker bugs being proposed and accepted, an RC2 should be requested as soon as all the blockers accepted after RC1 testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.

A new release candidate should never be requested to address freeze exception bugs only: a request should only be made if blocker issues remain. See below and the freeze exception SOP for details on freeze exception bugs.

The Current_Release_Blockers page can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only hourly. When the blocker situation is changing rapidly, it is best to use direct Bugzilla searches to ensure your list is entirely accurate.

How to request a compose

Compose requests are filed as tickets in the release engineering team's trac instance. Only one ticket should be filed for each milestone (Alpha, Beta, Final). So at each milestone, file a new ticket to request the first test compose, and then add comments to that ticket for subsequent test composes and the release candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.

• Example compose request ticket (Fedora 19 Beta)

What to include in a compose request

There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a test compose for Fedora 18 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.

Understanding composes

It will obviously help you to write a good compose request if you understand more or less what happens in a compose. If you were to simply request a compose with no special instructions, ReleaseEngineering would produce:

• A full set of release media
• Install trees for all installable products

using:

• The packages present in the current 'stable' tree for the release (that is, in the repositories at development/(releasever) on the mirrors) - not including packages in updates-testing
• The packages present in the bleed repository, which is populated with packages explicitly requested to be included in composes (see below)
• The versions of the compose tools currently in that same 'stable' tree - that is, livecd-tools from that repository, pungi from that repository, and so on
• The current HEAD of the relevant branch in spin-kickstarts git - not the master branch, but the f(releasever) branch, e.g. here for Fedora 42
• The comps information included in the current 'stable' tree, which is generated daily from the comps-f(releasever).xml file in comps git, e.g. this file for Fedora 42

and place the results in the staging tree.

If the compose should differ in any respect, you should explain in the compose request. This section explains some common cases where you might add particular requests.

Explicitly listing required builds

Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.

When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.

What builds to ensure are included

For test composes, this is essentially a judgement call. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable). It is usually a good idea to explicitly include any pending updates that you know will have a significant impact on testing - so if a big anaconda update is pending, you should probably request that it be pulled in.

For release candidates, this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.

You may also request that pending fixes for freeze exception bugs are included in release candidate composes. Freeze exception bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes may be allowed through the freeze: see the QA:SOP_freeze_exception_bug_process for further details. During release freezes, release engineering may push freeze exception fixes to the stable repository, but if a fix for a freeze exception issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending freeze exception fixes in release candidate composes.

Try and bear in mind that the ideal process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release: this is a hack, so avoid it when possible, but do it when necessary.

It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or freeze exception bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or freeze exception issue, do not just list it anyway: propose the bug as a release blocker or freeze exception issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.

Special instructions

There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.