This SOP explains the procedure for requesting 'candidate' composes from the ReleaseEngineering team. This task is usually performed by QA.
What composes should be requested when
There are three milestones in the Fedora release cycle: Alpha, Beta, and Final. For each milestone, a minimum of one 'candidate' (known in release engineering terms as a 'production compose') must be produced. Usually, there will be more than one.
In the normal course of events, 'nightly' composes are regularly produced by release engineering and validation testing is performed on these. There are two reasons to request a 'candidate' compose. The first is to allow testing of an update before it is pushed stable if it cannot be tested fully by updating an installed system (e.g. installer updates, or GNOME/Plasma updates to test the live images). The second is to produce a compose that can be released: 'nightly' composes have some properties that mean we cannot ship them as Alpha, Beta or Final releases, so a 'candidate' / 'production' compose must be built for release. You could refer to a 'candidate' of this type as a 'release candidate'.
Only a 'candidate' composed after the milestone freeze and with all blocker bugs addressed can be released, and this requirement supersedes the schedule: see below for more on this.
The schedule for the next Fedora release, with the milestone freeze dates, can be found here.
Candidates for updates testing
Prior to the Fedora 24 cycle, there were 'test compose' and 'release candidate' composes. This is no longer the case. In the usual course of events, validation testing before the freeze date for each milestone is simply performed on nightly composes. 'Candidate' composes can be requested prior to the freeze date if they are required to test one or more updates before they can be pushed stable, but ideally this should be avoided to save unnecessary work on the release engineering team's part.
Release candidates
A release candidate is defined as a set of Fedora images built, from the current Branched tree (not including updates-testing), after the milestone freeze for one of the three Fedora milestones (Alpha, Beta and Final) and using a package set which is not known to contain any blocker bugs, for the purposes of performing release validation testing. It may be declared gold and released as the Alpha, Beta or Final release if it passes all validation tests.
A 'candidate' compose can be considered a 'release candidate' only after the milestone freeze (and only when the freeze has actually been put into place) and when all accepted blocker bugs are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the blocker bug process designates two special blocker types, Accepted0Day and AcceptedPreviousRelease, to handle such cases. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the update testing process and been marked as stable (and hence before the bug is actually closed).
The procedure for release candidates is to request the first compose as close as soon as possible after the milestone freeze date, but only after all accepted blockers are addressed.
If testing of the first 'release candidate' results in further blocker bugs being proposed and accepted, another should be requested as soon as all the blockers accepted after testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.
A new release candidate should never be requested to address freeze exception bugs only: a request should only be made if blocker issues remain. See below and the freeze exception SOP for details on freeze exception bugs.
The blocker bugs web application can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only half-hourly. When the blocker situation is changing rapidly, it may be best to use direct Bugzilla searches to ensure your list is entirely accurate.
How to request a compose
Compose requests are filed as tickets in the release engineering team's Pagure project. Only one ticket should be filed for each milestone (Alpha, Beta, Final). So at each milestone, file a new ticket to request the first candidate, and then add comments to that ticket for subsequent candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.
What to include in a compose request
There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a candidate compose for Fedora 24 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.
Understanding composes
It will obviously help you to write a good compose request if you understand more or less what happens in a compose. If you were to simply request a compose with no special instructions, ReleaseEngineering would produce:
- A full set of release media
- Install trees for all installable products
using:
- The packages present in the current stable tree for the release (that is, in the repositories at development/(releasever) on the mirrors) - not including packages in updates-testing
- Any packages previously explicitly requested for inclusion in earlier composes (see below)
- The versions of the compose tools currently in that same 'stable' tree - that is,
livecd-tools - The current HEAD of the relevant branch in fedora-kickstarts git - not the master branch, but the f(releasever) branch, e.g. here for Fedora 40
- The comps information included in the current 'stable' tree, which is generated daily from the comps-f(releasever).xml file in fedora-comps git, e.g. this file for Fedora 40
and place the results in the staging tree.
If the compose should differ in any respect, you should explain in the compose request. This section explains some common cases where you might add particular requests.
Explicitly listing required builds
Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. This is the most common form of request. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.
When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.
What builds to ensure are included
When building a candidate to test update fixes, of course, list all updates that need a candidate compose in order to be fully tested. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable).
When requesting a candidate compose after the freeze, you must only list packages that fix accepted release blocking or freeze exception bugs.
When requesting a candidate intended as a potential 'release candidate' (which by definition always happens after the freeze), this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.
You may also request that pending fixes for freeze exception bugs are included in release candidate composes. Freeze exception bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes may be allowed through the freeze: see the QA:SOP_freeze_exception_bug_process for further details. During release freezes, release engineering may push freeze exception fixes to the stable repository, but if a fix for a freeze exception issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending freeze exception fixes in release candidate composes.
Try and bear in mind that the ideal process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release (and because sometimes updates cannot be tested without a compose): this is a hack, so avoid it when possible, but do it when necessary.
It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or freeze exception bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or freeze exception issue, do not just list it anyway: propose the bug as a release blocker or freeze exception issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.
Special instructions
There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.
The candpush tool
There is a tool called candpush which can help you generate compose requests. You can check it out and run it from the qa-misc repository:
git clone https://pagure.io/fedora-qa/qa-misc.git cd qa-misc dnf install python3-fedora python3-bugzilla python3-fedfind ./candpush
It will print proposed comment text for a candidate compose (and also for a stable push request - you can ignore that part). It will also print warnings for updates with <0 karma and updates that fix bugs which are not accepted blockers or freeze exceptions, for you to inspect manually.
Please only use this text as a starting point, and review it according to the above instructions. You may need to add or remove requests, or tweak the text. The tool is only a helper. It will list all updates which are marked in Bodhi as being for the correct release and associated with at least one accepted blocker or freeze exception bug.
What comes next
Once a candidate is built, it must be tested! This is done according to the release validation test plan. Normally, the necessary result pages are created and the validation test event announced automatically by relvalconsumer (the production relvalconsumer instance is maintained by Adam Williamson). The release validation test event SOP covers the process of doing this manually if that is for any reason necessary.
Ensuring requested builds go stable
It is important that any builds pulled into a candidate via this process are then properly pushed to the stable repository very soon afterwards. This is particularly crucial for the Final milestone: we strongly desire that the final frozen release repository and the release media are derived from the same package set.
QA and release engineering share responsibility for ensuring the list of explicitly-included builds is tracked, and all such builds are pushed to stable (or, if they are broken, superseded or withdrawn from subsequent composes). The QA process for this is not yet documented, but usually every few days someone will check the list of builds included in the current candidate but not yet pushed stable, file a ticket in release engineering Pagure requesting stable push of all those with sufficient karma, and send out a mail requesting high-priority testing of any which do not have sufficient karma. An example 'stable push request' ticket can be found here.