From Fedora Project Wiki
< Changes
| (32 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name. This keeps all change proposals in the same namespace --> | <!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name. This keeps all change proposals in the same namespace --> | ||
= createrepo_c 1.0.0 <!-- The name of your change proposal --> = | = Changes of defaults in createrepo_c-1.0.0 <!-- The name of your change proposal --> = | ||
== Summary == | == Summary == | ||
| Line 23: | Line 21: | ||
== Current status == | == Current status == | ||
[[Category: | [[Category:ChangeAcceptedF39]] | ||
<!-- When your change proposal page is completed and ready for review and announcement --> | <!-- When your change proposal page is completed and ready for review and announcement --> | ||
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | ||
| Line 41: | Line 39: | ||
ON_QA -> change is fully code complete | ON_QA -> change is fully code complete | ||
--> | --> | ||
* FESCo issue: | * [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/25J55FCQKTMCEOBS7EZCTP34RKLUOUQ3/ devel thread] | ||
* Tracker bug: | * FESCo issue: [https://pagure.io/fesco/issue/2976 #2976] | ||
* Release notes tracker: | * Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2185617 #2185617] | ||
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/977 #977] | |||
== Detailed Description == | == Detailed Description == | ||
<!-- Expand on the summary, if appropriate. A couple sentences suffices to explain the goal, but the more details you can provide the better. --> | <!-- Expand on the summary, if appropriate. A couple sentences suffices to explain the goal, but the more details you can provide the better. --> | ||
* Switch default compression from gz to zstd. DNF is able to decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression.<BR><BR>Further information: https://github.com/rpm-software-management/createrepo_c/issues/82, https://bugzilla.redhat.com/show_bug.cgi?id=1914876 | |||
Further information: https://github.com/rpm-software-management/createrepo_c/issues/82. | |||
* Stop generating metadata in sqlite database format by default. It will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).<BR><BR>Further information: https://github.com/rpm-software-management/createrepo_c/issues/338 and https://pagure.io/releng/issue/10745 | |||
Further information: https://github.com/rpm-software-management/createrepo_c/issues/338 and https://pagure.io/releng/issue/10745 | |||
* When adding groups.xml to repodata createrepo_c currently adds two variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.<BR><BR>Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318 | |||
Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318 | |||
* To reflect API and overall stability increase major version above 0. | |||
== Feedback == | == Feedback == | ||
| Line 93: | Line 89: | ||
* Using zstd compression provides smaller metadata that are faster to decompress. | * Using zstd compression provides smaller metadata that are faster to decompress. | ||
* Generating sqlite metadata is expensive, its slowing createrepo_c runs and it takes up space on mirrors. (For F37 updates repo: primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.) | * Generating sqlite metadata is expensive, its slowing createrepo_c runs and it takes up space on mirrors. (For F37 updates repo: primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.) | ||
* More consistent metadata. | * More consistent metadata, groups use the same logic as other metadata types. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1904360 | ||
== Scope == | == Scope == | ||
| Line 102: | Line 98: | ||
<!-- What work do other developers have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | <!-- What work do other developers have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | ||
* Release engineering: | * Release engineering: https://pagure.io/releng/issue/11357 <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | ||
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | ||
| Line 120: | Line 116: | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
Repodata created with createrepo_c's new defaults will not be compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6). | |||
This is mainly due to zstd compression but old YUM also cannot handle the changed mdtype of groups (it is compatible only with uncompressed - "group" and compressed - "group_XX" not compressed - "group"). | |||
Regarding the sqlite databases if some other users would require them they can always download the xml metadata and use sqliterepo_c to generate them locally. | |||
== How To Test == | == How To Test == | ||
| Line 155: | Line 154: | ||
The only noticeable effect should be smaller metadata thanks to zstd compression resulting in faster downloads (when zchunk is not used). The (de)compression speed up is not very noticeable. | The only noticeable effect should be smaller metadata thanks to zstd compression resulting in faster downloads (when zchunk is not used). The (de)compression speed up is not very noticeable. | ||
For example for | For example for F37 fedora repo: | ||
* primary.xml.gz: | * primary.xml.gz: 18M -> primary.xml.zst: 14M | ||
* filelists.xml.gz: | * filelists.xml.gz: 52M -> filelists.xml.zst: 41M | ||
* other.xml.gz: | * other.xml.gz: 6.9M -> other.xml.zst: 5.4M | ||
== Dependencies == | == Dependencies == | ||
| Line 166: | Line 165: | ||
Updating createrepo_c does not require any other change requests to complete first. | Updating createrepo_c does not require any other change requests to complete first. | ||
Metadata generated by the following tools could be affected by the described changes if they use createrepo_c's default configuration. It is hard to tell what effect will this have on the tools. If they create metadata for dnf it will work fine. In case there are any problems it is possible to manually setup the old behavior (as described above) or revert the changes. | |||
bodhi-server | bodhi-server | ||
copr-backend | copr-backend | ||
libguestfs | libguestfs | ||
modulemd-tools | modulemd-tools | ||
| Line 193: | Line 191: | ||
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | <!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | ||
* Contingency mechanism: | * Contingency mechanism: If we cannot finish the changes in time there would be no action necessary. We can even ship only part of the changes as they don't depend on each other. In case there are additional unexpected issues with the changes we can revert them in createrepo_c. <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | <!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | ||
* Contingency deadline: | * Contingency deadline: 2023-08-01 <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | <!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | ||
* Blocks release? | * Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
== Documentation == | == Documentation == | ||
Latest revision as of 15:16, 10 April 2023
Changes of defaults in createrepo_c-1.0.0
Summary
Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.
Owner
- Name: Aleš Matěj
- Email: amatej@redhat.com
Current status
- Targeted release: Fedora Linux 39
- Last updated: 2023-04-10
- devel thread
- FESCo issue: #2976
- Tracker bug: #2185617
- Release notes tracker: #977
Detailed Description
- Switch default compression from gz to zstd. DNF is able to decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression.
Further information: https://github.com/rpm-software-management/createrepo_c/issues/82, https://bugzilla.redhat.com/show_bug.cgi?id=1914876
- Stop generating metadata in sqlite database format by default. It will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).
Further information: https://github.com/rpm-software-management/createrepo_c/issues/338 and https://pagure.io/releng/issue/10745
- When adding groups.xml to repodata createrepo_c currently adds two variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.
Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318
- To reflect API and overall stability increase major version above 0.
Feedback
Benefit to Fedora
- Using zstd compression provides smaller metadata that are faster to decompress.
- Generating sqlite metadata is expensive, its slowing createrepo_c runs and it takes up space on mirrors. (For F37 updates repo: primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.)
- More consistent metadata, groups use the same logic as other metadata types. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1904360
Scope
- Proposal owners: Implement specified changes, mostly changes of default.
- Other developers: There are no requirements on other developers.
- Release engineering: https://pagure.io/releng/issue/11357
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
Upgrade/compatibility impact
Repodata created with createrepo_c's new defaults will not be compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6). This is mainly due to zstd compression but old YUM also cannot handle the changed mdtype of groups (it is compatible only with uncompressed - "group" and compressed - "group_XX" not compressed - "group").
Regarding the sqlite databases if some other users would require them they can always download the xml metadata and use sqliterepo_c to generate them locally.
How To Test
Repositories generated with updated createrepo_c have to work with dnf. Specifically we should ensure that groups are still loaded.
User Experience
The only noticeable effect should be smaller metadata thanks to zstd compression resulting in faster downloads (when zchunk is not used). The (de)compression speed up is not very noticeable.
For example for F37 fedora repo:
- primary.xml.gz: 18M -> primary.xml.zst: 14M
- filelists.xml.gz: 52M -> filelists.xml.zst: 41M
- other.xml.gz: 6.9M -> other.xml.zst: 5.4M
Dependencies
Updating createrepo_c does not require any other change requests to complete first.
Metadata generated by the following tools could be affected by the described changes if they use createrepo_c's default configuration. It is hard to tell what effect will this have on the tools. If they create metadata for dnf it will work fine. In case there are any problems it is possible to manually setup the old behavior (as described above) or revert the changes.
bodhi-server copr-backend libguestfs modulemd-tools prunerepo pungi retrace-server swid-tools anaconda-install-env-deps cobbler ddiskit koji-builder mock module-build perl-Fedora-Rebuild python3-dlrn python3-dnf-plugin-local python3-dnf-plugin-modulesync python3-rpmfluff retrace-server
Contingency Plan
- Contingency mechanism: If we cannot finish the changes in time there would be no action necessary. We can even ship only part of the changes as they don't depend on each other. In case there are additional unexpected issues with the changes we can revert them in createrepo_c.
- Contingency deadline: 2023-08-01
- Blocks release? No
Documentation
Createrepo_c documentation will be updated accordingly.