Latest revision as of 12:34, 9 February 2024

DNF: Do not download filelists by default

Summary

Change the DNF behavior to not download filelists by default. These metadata, which describe all the files contained within each package, are unnecessary in the majority of use cases. Additionally, these metadata files can be large in size, leading to a significant slowdown in the user experience.

Owner

Name: Jan Kolarik
Email: jkolarik@redhat.com

Current status

Targeted release: Fedora Linux 40
Last updated: 2024-02-09
Announced
Discussion thread
FESCo issue: #3097
Tracker bug: #2254789
Release notes tracker: #1064

Detailed Description

Until now, filelists were always downloaded together with other metadata. This was hardcoded and unable to change from the outside of DNF.

With these changes, we are proposing to not download the filelists metadata by default. This default behavior can be modified through the new DNF configuration option. Additionally, specific commands can override this behavior and request loading the filelists metadata at runtime using the existing demands object in DNF.

Note that after this change, users can still use DNF without filelists metadata when querying file provides located in /usr/bin, /usr/sbin or /etc directories.

The proposed behavior has already been incorporated into the future successor, DNF5 project, where they were implemented around the beginning of this year (see this PR for more details).

Feedback

Benefit to Fedora

As DNF is integral to various infrastructure tasks like package building and installation, testing environment creation, and server integration tests, this change significantly reduces processing time and resource usage for these processes.

This change reduces the RAM requirements of the DNF process, addressing existing issues when running the Fedora system on low-memory machines such as the Raspberry Pi (see f.e. Bug 1907030).

Also, omitting the filelists metadata download overall decreases the costs of a Fedora mirror server operation.

As the described behavior already exists in its extended form in DNF5 within the current Fedora release, allowing any optional metadata types to be conditionally loaded, and considering that DNF5 is planned to replace DNF as the main package manager for Fedora 41, implementing these changes will facilitate a smoother and more compatible transition process.

Scope

Proposal owners:
- libdnf
  - Modify the Repo object to enable conditional filelists metadata download
  - Introduce a new main configuration option to set the default behavior
- dnf
  - Enable configuration of filelists download from commandline, DNF commands and DNF plugins
  - Implement filename pattern argument detection heuristics

Other developers:
- Dependencies using the existing DNF C interface may need to adapt if they expect the filelists metadata to be available and explicitly request loading filelists using the existing API due to this change:
  - PackageKit
  - microdnf
  - API users

Release engineering: N/A

Policies and guidelines:
- Package maintainers must follow Fedora's packaging guidelines, particularly concerning file dependency specifications (see here)
  - Adopting the MUST NOT rule in these guidelines would help prevent future issues with the installability of such packages.
  - A few packages in the current Fedora developmental release are not following these rules. Pull requests have already been prepared to fix their spec files. Please refer to Bug 2180842 for details.

Trademark approval: N/A

Alignment with Community Initiatives: N/A (no currently active initiatives)

Upgrade/compatibility impact

In general, applying these changes should not affect any existing user workflows and no additional manual changes are required.

However, the absence of filelists would cause issues for packages that do not follow the recommended file dependencies outlined in the packaging guidelines. This change would render such packages uninstal‌lable without the presence of filelists. In the current Fedora release repository, only a few packages are affected, and none of them is critical to the system. Also, trivial pull requests have already been prepared for each, resolving the issue upon merging.

If DNF fails to resolve a transaction due to a missing file dependency, and the filelists metadata are not currently present on the system, users will receive a hint on how to request the download of filelists from the command line. This action may assist in resolving the situation.

For more information, refer to the Bug 2180842 and the discussion thread on this proposal.

How To Test

When using DNF commands without a filename pattern passed as the argument, filelists metadata should not be downloaded from the remote repositories and should not be needed for the command execution. This can be tested with the following steps:

Clean the local metadata cache (dnf clean metadata)
Run a DNF command not involving the filename spec (e.g. dnf repoquery rpm)
Verify that no *-filelists.* metadata files were downloaded inside the cache subdirectories (by default under the /var/cache/dnf for root)
Check the command works as expected

The same should also apply to RPM package arguments (files ending with .rpm extension).

When using DNF commands with a filename pattern passed as the argument, filelists metadata should be downloaded from the remote repositores as before.

User Experience

Large filelists could be over 200MB in size. It could take 1-2 minutes to download which is greatly slowing down the user experience.

For many operations the filelists metadata are not needed, so downloading them is wasting the resources. Without filelists being downloaded, DNF performance will be improved significantly, mainly regarding the network, CPU and disk space resources. Metadata download size will be reduced by about 60%. The improvement includes deployments of customer built RPMS to containers that have no need for filelists level dependencies.

Dependencies

No changes should be required for any package depending on DNF to implement this behavior.

Contingency Plan

Contingency mechanism: Change the configuration option to download the filelists by default
Contingency deadline: Branch Fedora Linux 40 from Rawhide
Blocks release? No

Documentation

New configuration option optional_metadata_types was added to allow requesting filelists metadata on demand, see configuration docs here.

Release Notes

@@ Line 1: / Line 1: @@
-{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.'''}}
-{{admon/tip | Guidance | For details on how to fill out this form, see the [https://docs.fedoraproject.org/en-US/program_management/changes_guide/ documentation].}}
-{{admon/tip | Report issues | To report an issue with this template, file an issue in the [https://pagure.io/fedora-pgm/pgm_docs pgm_docs repo].}}
-<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 = DNF: Do not download filelists by default <!-- The name of your change proposal --> =
-{{Change_Proposal_Banner}}
 == Summary ==
@@ Line 27: / Line 17: @@
 == Current status ==
-[[Category:ChangePageIncomplete]]
+[[Category:ChangeAcceptedF40]]
 <!-- When your change proposal page is completed and ready for review and announcement -->
 <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
@@ Line 34: / Line 24: @@
 <!-- Select proper category, default is Self Contained Change -->
-[[Category:SelfContainedChange]]
+[[Category:SystemWideChange]]
 <!-- [[Category:SystemWideChange]] -->
@@ Line 45: / Line 35: @@
 ON_QA -> change is fully code complete
 -->
-* [<will be assigned by the Wrangler> devel thread]
+* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5UFFIAR5ITBS7YFS4N5HM5GGPXYVPF7E/ Announced]
-* FESCo issue: <will be assigned by the Wrangler>
+* [https://discussion.fedoraproject.org/t/f40-change-proposal-dnfconditionalfilelists-system-wide/94939 Discussion thread]
-* Tracker bug: <will be assigned by the Wrangler>
+* FESCo issue: [https://pagure.io/fesco/issue/3097 #3097]
-* Release notes tracker: <will be assigned by the Wrangler>
+* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2254789 #2254789]
+* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/1064 #1064]
 == Detailed Description ==
 Until now, filelists were always downloaded together with other metadata. This was hardcoded and unable to change from the outside of DNF.
-With these changes, we are proposing to not download the filelists metadata by default. This can be changed through the new DNF configuration option `filelists` which is `False` by default. If it is set to `True`, filelists will be downloaded during the standard metadata synchronization procedure. Additionally, specific commands can override this behavior during the runtime using the existing demands object in the DNF and so explicitly request loading the filelists metadata. This is used when a filename spec (other than rpm package filename) is passed as the command-line parameter.
+With these changes, we are proposing to not download the filelists metadata by default. This default behavior can be modified through the new DNF configuration option. Additionally, specific commands can override this behavior and request loading the filelists metadata at runtime using the existing demands object in DNF.
+Note that after this change, users can still use DNF without filelists metadata when querying file provides located in `/usr/bin`, `/usr/sbin` or `/etc` directories.
+The proposed behavior has already been incorporated into the future successor, DNF5 project, where they were implemented around the beginning of this year (see [https://github.com/rpm-software-management/dnf5/pull/123 this PR] for more details).
 == Feedback ==
@@ Line 60: / Line 55: @@
 == Benefit to Fedora ==
 As DNF is integral to various infrastructure tasks like package building and installation, testing environment creation, and server integration tests, this change significantly reduces processing time and resource usage for these processes.
+This change reduces the RAM requirements of the DNF process, addressing existing issues when running the Fedora system on low-memory machines such as the Raspberry Pi (see f.e. [https://bugzilla.redhat.com/show_bug.cgi?id=1907030 Bug 1907030]).
+Also, omitting the filelists metadata download overall decreases the costs of a Fedora mirror server operation.
+As the described behavior already exists in its extended form in DNF5 within the current Fedora release, allowing any optional metadata types to be conditionally loaded, and considering that DNF5 is planned to replace DNF as the main package manager for Fedora 41, implementing these changes will facilitate a smoother and more compatible transition process.
 == Scope ==
@@ Line 67: / Line 68: @@
 *** Introduce a new main configuration option to set the default behavior
 ** dnf
-*** Introduce a new demand to enable specific commands to override filelists metadata download behavior
+*** Enable configuration of filelists download from commandline, DNF commands and DNF plugins
-*** Handle demand and configuration option inputs to delegate filelists loading decision to `libdnf`
 *** Implement filename pattern argument detection heuristics
 * Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-** The implementation work is isolated only to the mentioned two DNF stack components
+** Dependencies using the existing DNF C interface may need to adapt if they expect the filelists metadata to be available and explicitly request loading filelists using the existing API due to this change:
-** There is no other development work related outside of the DNF stack
+*** PackageKit
+*** microdnf
+*** API users
 * Release engineering: N/A
@@ Line 79: / Line 81: @@
 * Policies and guidelines:
 ** Package maintainers must follow Fedora's packaging guidelines, particularly concerning file dependency specifications (see [https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directory_dependencies here])
+*** Adopting the '''MUST NOT''' rule in these guidelines would help prevent future issues with the installability of such packages.
+*** A few packages in the current Fedora developmental release are not following these rules. Pull requests have already been prepared to fix their spec files. Please refer to [https://bugzilla.redhat.com/show_bug.cgi?id=2180842 Bug 2180842] for details.
 * Trademark approval: N/A
@@ Line 87: / Line 91: @@
 == Upgrade/compatibility impact ==
-In general, applying these changes should not affect any existing user workflows and no additional manual changes are required. However, the absence of filelists might create an issue with packages that are not correctly packaged, f.e. from third-party repositories.
+In general, applying these changes should not affect any existing user workflows and no additional manual changes are required.
+However, the absence of filelists would cause issues for packages that do '''not''' follow the recommended file dependencies outlined in the [https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directory_dependencies packaging guidelines]. This change would render such packages uninstal‌lable without the presence of filelists. In the current Fedora release repository, only a few packages are affected, and none of them is critical to the system. Also, trivial pull requests have already been prepared for each, resolving the issue upon merging.
+If DNF fails to resolve a transaction due to a missing file dependency, and the filelists metadata are not currently present on the system, users will receive a hint on how to request the download of filelists from the command line. This action may assist in resolving the situation.
+For more information, refer to the [https://bugzilla.redhat.com/show_bug.cgi?id=2180842 Bug 2180842] and the [https://discussion.fedoraproject.org/t/f40-change-proposal-dnfconditionalfilelists-system-wide/94939 discussion thread] on this proposal.
 == How To Test ==
-When using DNF commands without a filename pattern passed as the argument, filelists metadata should not be downloaded from the remote repository and should not be needed for the command execution. This can be tested with the following steps:
+When using DNF commands without a filename pattern passed as the argument, filelists metadata should not be downloaded from the remote repositories and should not be needed for the command execution. This can be tested with the following steps:
 * Clean the local metadata cache (`dnf clean metadata`)
-* Run a DNF command not involving the filename spec (`dnf repoquery rpm`)
+* Run a DNF command not involving the filename spec (e.g. `dnf repoquery rpm`)
 * Verify that no `*-filelists.*` metadata files were downloaded inside the cache subdirectories (by default under the `/var/cache/dnf` for root)
 * Check the command works as expected
-The same behavior should also apply to RPM package arguments (files ending with `.rpm` extension).
+The same should also apply to RPM package arguments (files ending with `.rpm` extension).
-When using DNF commands with a filename pattern passed as the argument, filelists metadata should be downloaded from the remote repository as before.
-<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.
-Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
-A good "how to test" should answer these four questions:
-. What special hardware / data / etc. is needed (if any)?
-. How do I prepare my system to test this change? What packages
-need to be installed, config files edited, etc.?
-. What specific actions do I perform to check that the change is
-working like it's supposed to?
-. What are the expected results of those actions?
--->
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+When using DNF commands with a filename pattern passed as the argument, filelists metadata should be downloaded from the remote repositores as before.
 == User Experience ==
@@ Line 131: / Line 125: @@
 == Dependencies ==
-<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
+No changes should be required for any package depending on DNF to implement this behavior.
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 == Contingency Plan ==
+* Contingency mechanism: Change the configuration option to download the filelists by default
-<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
+* Contingency deadline: Branch Fedora Linux 40 from Rawhide
-* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Blocks release? No
-<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
-* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
-* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 == Documentation ==
-<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
+New configuration option `optional_metadata_types` was added to allow requesting filelists metadata on demand, see configuration docs [https://dnf.readthedocs.io/en/latest/conf_ref.html#optional-metadata-types-label here].
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-N/A (not a System Wide Change)
 == Release Notes ==

Search

Changes/DNFConditionalFilelists: Difference between revisions