Changes/ChangeComposeSettings

Upgrade systems to createrepo_c 1.0 and change repositories metadata settings

Summary

This is a proposal for upgrading systems which produce composes to createrepo_c > 1.0 and changing some options used to create Fedora repositories metadata. Note that some of these changes are inevitable due to createrepo_c >= 1.0 behavioral change. We aim to change both Rawhide/F41, then move all following releases to the new settings, while preserving most of the current settings for releases <= 40.

Owner

• Name: Mattia Verga, Kevin Fenzi
• Email: mattia@fedorapeople.org, kevin@fedorapeople.org

Current status

• Targeted release: Fedora Linux 41
• Last updated: 2024-04-22
• Announced
• Discussion Thread
• FESCo issue: #3189
• Tracker bug: #2276422
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

With createrepo_c < 1.0 we're currently using different settings for Rawhide repository metadata and stable releases repository metadata.

• Rawhide
  • gzip compression for all metadata
  • no updateinfo.xml file is generated (no updates repository exists)
  • sqlite database of repodata is generated and compressed with gzip as well
  • comps repodata is made available both as uncompressed xml and gzipped
  • zchunk is active
  • DRPMs disabled

• F40
  • gzip compression for primary metadata
  • updateinfo.xml (for updates repository) is compressed with XZ
  • sqlite database of repodata is generated and compressed with BZ2
  • comps repodata is made available both as uncompressed xml and gzipped
  • zchunk is active
  • DRPMs disabled (approved change for F40)

• F<40
  • gzip compression for primary metadata
  • updateinfo.xml (for updates repository) is compressed with XZ
  • sqlite database of repodata is generated and compressed with BZ2
  • comps repodata is made available both as uncompressed xml and gzipped
  • zchunk is active
  • DRPMs enabled

With createrepo_c > 1.0 moving to zstd as the default compression type, we want to have consistent settings for all new releases and to specify those settings manually, so that possible future changes of defaults don't cause unexpected breakages. So we propose the following settings:

• Rawhide/F>=41
  • use --general-compress-type set to zstd to compress all metadata to zstd
  • updateinfo.xml (for updates repository) compressed with zstd as well
  • disable generating the additional sqlite database, as it was only useful for yum
  • comps repodata will be available only zstd compressed
  • zchunk is active
  • DRPMs disabled

• F<=40
  • nothing should change by using the --compatibility flag of createrepo_c >= 1.0

Note that updating bodhi-composer to use createrepo_c >= 1.0 will also introduce an unavoidable change into EPEL8 updates repositories: comps repodata will be made available only in compressed format (which is set to XZ in EPEL8). Other EPEL releases (EPEL7 and EPEL9) can use the --compatibility flag to maintain actual settings.

Feedback

The zstd compression type was chosen to match createrepo_c settings. As an alternative, we might want to choose xz, especially after zlib-ng has been made the default in Fedora and brought performance improvements.

The sqlite database distributed alongside the repodata is not useful for dnf, but it might be used by some external consumer we're not aware. Please do let us know.

Benefit to Fedora

We're aiming at having consistent defaults for Rawhide and stable releases and avoid future changes to createrepo_c defaults to cause unexpected changes to repodata. Also, by using better compression methods and avoid generating the sqlite database we're reducing repodata disk usage.

Scope

• Proposal owners:
  • change createrepo_c settings for Rawhide in compose-rawhide01 (which is already upgraded to f39)
  • upgrade bodhi-composer to f39 and have it using createrepo_c >= 1.0.0
  • upgrade (if a new release is released in time) or backport patch into bodhi to support all createrepo_c settings
  • change bodhi's createrepo_c.ini in ansible to use the new settings for F>=41

• Other developers:

• Release engineering: #Releng issue number

• Policies and guidelines: N/A (not needed for this Change)

• Trademark approval: N/A (not needed for this Change)

• Alignment with Community Initiatives:

Upgrade/compatibility impact

No change should be noticed while upgrading from previous releases.

How To Test

DNF normal day usage should not be affected: upgrading/installing packages should work as before, maybe a little faster in downloading repodata.

User Experience

User experience should not be affected.

Possible external custom repodata consumers might stop working and will need to adjust to the new compression method.

Contingency Plan

• Contingency mechanism: revert back to old createrepo_c < 1.0 and previous settings and wait for the next day compose.
• Contingency deadline: F41-beta freeze
• Blocks release? Yes

Documentation

https://docs.pagure.org/pungi/configuration.html#createrepo-settings