From Fedora Project Wiki
(→‎Owner: ppisar is not a releng with permissions to implement this change)
(adding tracker bug)
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.'''}}


{{admon/tip | Guidance | For details on how to fill out this form, see the [https://docs.fedoraproject.org/en-US/program_management/changes_guide/ documentation].}}
= Upgrade systems to createrepo_c 1.0 and change repositories metadata settings <!-- The name of your change proposal --> =
 
{{admon/tip | Report issues | To report an issue with this template, file an issue in the [https://pagure.io/fedora-pgm/pgm_docs pgm_docs repo].}}
 
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 
= Change Compose Repositories Defaults <!-- The name of your change proposal --> =
 
{{Change_Proposal_Banner}}


== Summary ==
== Summary ==
This is a proposal for changing some of the `createrepo_c` options used to create Fedora repositories repodata. We aim to change both Rawhide and F41, then move all following releases to the new settings, while preserving the current settings for releases <= 40.
This is a proposal for upgrading systems which produce composes to createrepo_c > 1.0 and changing some options used to create Fedora repositories metadata. Note that some of these changes are inevitable due to createrepo_c >= 1.0 behavioral change. We aim to change both Rawhide/F41, then move all following releases to the new settings, while preserving most of the current settings for releases <= 40.


== Owner ==
== Owner ==
Line 27: Line 18:


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF41]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 45: Line 36:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* [<will be assigned by the Wrangler> devel thread]
* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/T4QZAC6FXLJFIG7RARAMXQNOPOCF6C2T/ Announced]
* FESCo issue: <will be assigned by the Wrangler>
* [https://discussion.fedoraproject.org/t/f41-change-propopsal-change-compose-settings-system-wide/108893 Discussion Thread]
* Tracker bug: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/3189 #3189]
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2276422 #2276422]
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>


== Detailed Description ==
== Detailed Description ==
Were currently using different settings for Rawhide repositories and stable releases repodata. Rawhide is using gzip compression for all metadata, while stable releases are using a combination of bz2 for some things and gzip for the others.
With createrepo_c < 1.0 we're currently using different settings for Rawhide repository metadata and stable releases repository metadata.
With createrepo_c 1.0.0 moving to zstd as the default compression type, we want to have consistent settings for all releases and to specify those settings manually, so that possible future changes of defaults don't cause unexpected breakages.
 
* Rawhide
** gzip compression for all metadata
** no updateinfo.xml file is generated (no updates repository exists)
** sqlite database of repodata is generated and compressed with gzip as well
** comps repodata is made available both as uncompressed xml and gzipped
** zchunk is active
** DRPMs disabled
 
* F40
** gzip compression for primary metadata
** updateinfo.xml (for updates repository) is compressed with XZ
** sqlite database of repodata is generated and compressed with BZ2
** comps repodata is made available both as uncompressed xml and gzipped
** zchunk is active
** DRPMs disabled (approved change for F40)
 
* F<40
** gzip compression for primary metadata
** updateinfo.xml (for updates repository) is compressed with XZ
** sqlite database of repodata is generated and compressed with BZ2
** comps repodata is made available both as uncompressed xml and gzipped
** zchunk is active
** DRPMs enabled
 
With createrepo_c > 1.0 moving to zstd as the default compression type, we want to have consistent settings for all new releases and to specify those settings manually, so that possible future changes of defaults don't cause unexpected breakages. So we propose the following settings:


After F40 branching, we want to move Rawhide/F41 to:
* Rawhide/F>=41
** use `--general-compress-type` set to zstd to compress all metadata to zstd
** updateinfo.xml (for updates repository) compressed with zstd as well
** disable generating the additional sqlite database, as it was only useful for yum
** comps repodata will be available only zstd compressed
** zchunk is active
** DRPMs disabled


* use zstd as compression type for all repodata
* F<=40
* disable generating the additional sqlite database, as it was only useful for yum
** nothing should change by using the `--compatibility` flag of createrepo_c >= 1.0


Then all releases starting from F41 onward will se the new settings, while those from F40 and older will maintain the old settings until their EOL.
Note that updating bodhi-composer to use createrepo_c >= 1.0 will also introduce an unavoidable change into EPEL8 updates repositories: comps repodata will be made available only in compressed format (which is set to XZ in EPEL8). Other EPEL releases (EPEL7 and EPEL9) can use the `--compatibility` flag to maintain actual settings.


== Feedback ==
== Feedback ==
Line 72: Line 95:
* Proposal owners:
* Proposal owners:
** change createrepo_c settings for Rawhide in compose-rawhide01 (which is already upgraded to f39)
** change createrepo_c settings for Rawhide in compose-rawhide01 (which is already upgraded to f39)
** upgrade bodhi-composer to f39 (after EPEL7 goes EOL) to have it using createrepo_c >= 1.0.0
** upgrade bodhi-composer to f39 and have it using createrepo_c >= 1.0.0
** upgrade bodhi to v8.0 at least and base image to f39 (after EPEL7 goes EOL) for Bodhi to support per release createrepo_c settings
** upgrade (if a new release is released in time) or backport patch into bodhi to support all createrepo_c settings
** change bodhi config in ansible so that only F41 will use the new settings after branching
** change bodhi's `createrepo_c.ini` in ansible to use the new settings for F>=41


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Line 116: Line 139:


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: revert back to previous settings and wait for the next day compose.
* Contingency mechanism: revert back to old createrepo_c < 1.0 and previous settings and wait for the next day compose.
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: F41-beta freeze
* Contingency deadline: F41-beta freeze

Latest revision as of 14:30, 22 April 2024

Upgrade systems to createrepo_c 1.0 and change repositories metadata settings

Summary

This is a proposal for upgrading systems which produce composes to createrepo_c > 1.0 and changing some options used to create Fedora repositories metadata. Note that some of these changes are inevitable due to createrepo_c >= 1.0 behavioral change. We aim to change both Rawhide/F41, then move all following releases to the new settings, while preserving most of the current settings for releases <= 40.

Owner

Current status

Detailed Description

With createrepo_c < 1.0 we're currently using different settings for Rawhide repository metadata and stable releases repository metadata.

  • Rawhide
    • gzip compression for all metadata
    • no updateinfo.xml file is generated (no updates repository exists)
    • sqlite database of repodata is generated and compressed with gzip as well
    • comps repodata is made available both as uncompressed xml and gzipped
    • zchunk is active
    • DRPMs disabled
  • F40
    • gzip compression for primary metadata
    • updateinfo.xml (for updates repository) is compressed with XZ
    • sqlite database of repodata is generated and compressed with BZ2
    • comps repodata is made available both as uncompressed xml and gzipped
    • zchunk is active
    • DRPMs disabled (approved change for F40)
  • F<40
    • gzip compression for primary metadata
    • updateinfo.xml (for updates repository) is compressed with XZ
    • sqlite database of repodata is generated and compressed with BZ2
    • comps repodata is made available both as uncompressed xml and gzipped
    • zchunk is active
    • DRPMs enabled

With createrepo_c > 1.0 moving to zstd as the default compression type, we want to have consistent settings for all new releases and to specify those settings manually, so that possible future changes of defaults don't cause unexpected breakages. So we propose the following settings:

  • Rawhide/F>=41
    • use --general-compress-type set to zstd to compress all metadata to zstd
    • updateinfo.xml (for updates repository) compressed with zstd as well
    • disable generating the additional sqlite database, as it was only useful for yum
    • comps repodata will be available only zstd compressed
    • zchunk is active
    • DRPMs disabled
  • F<=40
    • nothing should change by using the --compatibility flag of createrepo_c >= 1.0

Note that updating bodhi-composer to use createrepo_c >= 1.0 will also introduce an unavoidable change into EPEL8 updates repositories: comps repodata will be made available only in compressed format (which is set to XZ in EPEL8). Other EPEL releases (EPEL7 and EPEL9) can use the --compatibility flag to maintain actual settings.

Feedback

The zstd compression type was chosen to match createrepo_c settings. As an alternative, we might want to choose xz, especially after zlib-ng has been made the default in Fedora and brought performance improvements.

The sqlite database distributed alongside the repodata is not useful for dnf, but it might be used by some external consumer we're not aware. Please do let us know.

Benefit to Fedora

We're aiming at having consistent defaults for Rawhide and stable releases and avoid future changes to createrepo_c defaults to cause unexpected changes to repodata. Also, by using better compression methods and avoid generating the sqlite database we're reducing repodata disk usage.

Scope

  • Proposal owners:
    • change createrepo_c settings for Rawhide in compose-rawhide01 (which is already upgraded to f39)
    • upgrade bodhi-composer to f39 and have it using createrepo_c >= 1.0.0
    • upgrade (if a new release is released in time) or backport patch into bodhi to support all createrepo_c settings
    • change bodhi's createrepo_c.ini in ansible to use the new settings for F>=41
  • Other developers:
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives:

Upgrade/compatibility impact

No change should be noticed while upgrading from previous releases.


How To Test

DNF normal day usage should not be affected: upgrading/installing packages should work as before, maybe a little faster in downloading repodata.


User Experience

User experience should not be affected.

Possible external custom repodata consumers might stop working and will need to adjust to the new compression method.

Dependencies

Contingency Plan

  • Contingency mechanism: revert back to old createrepo_c < 1.0 and previous settings and wait for the next day compose.
  • Contingency deadline: F41-beta freeze
  • Blocks release? Yes


Documentation

https://docs.pagure.org/pungi/configuration.html#createrepo-settings

Release Notes

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/ChangeComposeSettings&oldid=707590"