From Fedora Project Wiki
(Not just Copr. All builds will be faster.)
(adding release notes tracker)
 
(17 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->


= [DRAFT] Build Fedora with DNF 5 <!-- The name of your change proposal --> =
= Build Fedora with DNF 5 <!-- The name of your change proposal --> =


{{Change_Proposal_Banner}}


== Summary ==
== Summary ==
Line 26: Line 25:


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF40]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 33: Line 32:


<!-- Select proper category, default is Self Contained Change -->
<!-- Select proper category, default is Self Contained Change -->
[[Category:SelfContainedChange]]
<!-- [[Category:SelfContainedChange]] -->
<!-- [[Category:SystemWideChange]] -->
[[Category:SystemWideChange]]


* Targeted release: [https://docs.fedoraproject.org/en-US/releases/f40/ Fedora Linux 40]
* Targeted release: [https://docs.fedoraproject.org/en-US/releases/f40/ Fedora Linux 40]
Line 44: Line 43:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* [<will be assigned by the Wrangler> devel thread]
* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5B3TF7LQIWSOVTMTT2B3ISEIH7RLOBDI/ Announced]
* FESCo issue: <will be assigned by the Wrangler>
* [https://discussion.fedoraproject.org/t/f40-change-proposal-build-with-dnf5-self-contained/94937 Discussion thread]
* Tracker bug: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/3096 #3096]
* Release notes tracker: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2252074 #2252074]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/1077 #1077]


== Detailed Description ==
== Detailed Description ==
Line 92: Line 92:
* Proposal owners:
* Proposal owners:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
The work to support DNF 5 in Mock is done already. This change should be as simple as setting the Mock option `config_opts['package_manager'] = 'dnf5'` in Mock, Koji, and Copr for F40+ builds. The `dnf5` doesn't necessarily have to be installed on building hosts - in such a case, Mock will automatically use `/bin/dnf-3` (DNF4) from the host to install DNF5 into the bootstrap chroot, to further use *that* DNF5 for build chroot installation.
The work to support DNF 5 in Mock is done already. This change should be as simple as setting the Mock option `config_opts['package_manager'] = 'dnf5'` in Mock, Koji, and Copr for F40+ builds (Koji config option exists from the `yum -> dnf4` era). The `dnf5` doesn't necessarily have to be installed on building hosts - in such a case, Mock will automatically use `/bin/dnf-3` (DNF4) from the host to install DNF5 into the bootstrap chroot, to further use *that* DNF5 for build chroot installation.


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: https://pagure.io/releng/issue/11737 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
Line 112: Line 112:
== Upgrade/compatibility impact ==
== Upgrade/compatibility impact ==
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
None expected.


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== How To Test ==


== How To Test ==
There are no special steps needed to test the change after it happens (updated `mock-core-configs` package is installed on your host), just enjoy the installation speedup.
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.  
 
There's a way to test this on Fedora 37+ or EPEL8+ host (builder) in advance.  Considering you want to build SRCRPM like `mock -r fedora-40-x86_64 your.src.rpm`, you can do this instead:
 
1. `mock -r fedora-40-x86_64 --scrub=all` (mandatory step to cleanup DNF4 from all caches)
 
2.  `mock -r fedora-40-x86_64 --config-opts=package_manager=dnf5 your.src.rpm` (DNF5 is installed and cached in bootstrap)
 
3. `mock -r fedora-40-x86_64 --scrub=all` (to invalidate caches again)
 
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.


Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
Line 132: Line 144:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== User Experience ==
== User Experience ==
Line 149: Line 160:
== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
None, both DNF4 and DNF5 are already part of the distribution and nothing is changing for them.


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== Contingency Plan ==
== Contingency Plan ==
Line 157: Line 169:
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: (What to do?  Who will do it?)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Revert the F40 Mock configuration in Koji and Copr back to using `dnf`.
Revert the F40 Mock configuration in Koji and Copr back to using `dnf` (5-minute work).
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
F40 Beta freeze
F40 Beta freeze
We plan to implement the change at least a month before the [https://fedorapeople.org/groups/schedule/f-40/f-40-all-tasks.html F40 Mass Rebuild event starts], ideally as soon as possible. The builds conducted prior to the mass rebuild will offer us added assurance that the mass rebuild won't be affected, and will provide us with time to address any potential issues in advance.
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? Yes <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? Yes <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
[https://rpm-software-management.github.io/mock/Feature-bootstrap.html Mock bootstrap, aka why DNF5 is not necessary on host]
[https://rpm-software-management.github.io/mock/Feature-package-managers Package manager support in Mock]


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== Release Notes ==
== Release Notes ==
Line 177: Line 194:
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
-->
-->
The build system has now transitioned to using the [https://github.com/rpm-software-management/dnf5 DNF5 package manager] for preparing Fedora 40 package building environment, replacing the [https://github.com/rpm-software-management/dnf DNF4] utilized for earlier Fedora releases. The introduction of DNF5 makes the build environment preparation faster. This is also the initial phase of migrating towards DNF5 as the default package manager in [https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5 forthcoming Fedora releases].

Latest revision as of 20:05, 28 February 2024


Build Fedora with DNF 5

Summary

We are proposing to change the Mock configuration in Mock (mock-core-configs), Koji, and Copr to use DNF 5 as Mock's package manager instead of DNF 4. DNF 5 would be used by Mock to install build dependencies into chroots for package builds. This change is related to the build infrastructure and is distinct from changing the default package manager in Fedora.

Owner

Current status

Detailed Description

DNF 5 is a new package manager intended to replace DNF: https://dnf5.readthedocs.io/en/latest/about.html. It offers significant performance improvements over DNF while achieving lower memory usage and disk footprint. The switch to DNF 5 was originally planned for Fedora 39, but it's been postponed to (likely) Fedora 41: https://pagure.io/fesco/issue/3039.

In the meantime, we would like to start building Fedora with DNF 5. The set of package management features that Mock needs for setting up buildroots is small compared to the full capabilities of DNF, so it's a good place to start deploying DNF 5. We will be able to test the stability of DNF 5 at a large scale and gather data about its performance.

The Mock developers have been working alongside the DNF 5 developers for a while to ensure Mock can use DNF 5, per this tracking issue: https://github.com/rpm-software-management/mock/issues/894. The two remaining items on that issue are "optional" items that are not blocking this proposed change.

Feedback

Benefit to Fedora

With the switch to DNF 5 as the default package manager on the horizon, the build infrastructure offers an opportunity to subject a crucial subset of DNF 5's features to heavy testing. This change will let us verify that every build dependency in the distribution is installable by DNF 5.

In addition, we expect a substantial performance improvement for package builds that spend a significant portion of their time installing build dependencies. Larger, compilation-heavy packages likely won't see much improvement; the difference will be most apparent when building many smaller packages. Switching the build system over to DNF 5 will let us measure the performance improvement over DNF across a wide variety of install transactions.

Scope

  • Proposal owners:

The work to support DNF 5 in Mock is done already. This change should be as simple as setting the Mock option config_opts['package_manager'] = 'dnf5' in Mock, Koji, and Copr for F40+ builds (Koji config option exists from the yum -> dnf4 era). The dnf5 doesn't necessarily have to be installed on building hosts - in such a case, Mock will automatically use /bin/dnf-3 (DNF4) from the host to install DNF5 into the bootstrap chroot, to further use *that* DNF5 for build chroot installation.

  • Other developers:
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives:

Upgrade/compatibility impact

None expected.


How To Test

There are no special steps needed to test the change after it happens (updated mock-core-configs package is installed on your host), just enjoy the installation speedup.

There's a way to test this on Fedora 37+ or EPEL8+ host (builder) in advance. Considering you want to build SRCRPM like mock -r fedora-40-x86_64 your.src.rpm, you can do this instead:

1. mock -r fedora-40-x86_64 --scrub=all (mandatory step to cleanup DNF4 from all caches)

2. mock -r fedora-40-x86_64 --config-opts=package_manager=dnf5 your.src.rpm (DNF5 is installed and cached in bootstrap)

3. mock -r fedora-40-x86_64 --scrub=all (to invalidate caches again)


User Experience

This change will mostly be invisible to users. The builds, namely the buildroot preparation, will be much faster.

Dependencies

None, both DNF4 and DNF5 are already part of the distribution and nothing is changing for them.


Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?)

Revert the F40 Mock configuration in Koji and Copr back to using dnf (5-minute work).

  • Contingency deadline:

F40 Beta freeze

We plan to implement the change at least a month before the F40 Mass Rebuild event starts, ideally as soon as possible. The builds conducted prior to the mass rebuild will offer us added assurance that the mass rebuild won't be affected, and will provide us with time to address any potential issues in advance.

  • Blocks release? Yes

Documentation

Mock bootstrap, aka why DNF5 is not necessary on host

Package manager support in Mock


Release Notes

The build system has now transitioned to using the DNF5 package manager for preparing Fedora 40 package building environment, replacing the DNF4 utilized for earlier Fedora releases. The introduction of DNF5 makes the build environment preparation faster. This is also the initial phase of migrating towards DNF5 as the default package manager in forthcoming Fedora releases.