Changes/DNFConditionalFilelists

Comments and Explanations
The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.
Copy the source to a new page before making changes! DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.

Guidance
For details on how to fill out this form, see the documentation.

Report issues
To report an issue with this template, file an issue in the pgm_docs repo.

DNF: Do not download filelists by default

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Change the DNF behavior to not download filelists by default. These metadata, which describe all the files contained within each package, are unnecessary in the majority of use cases. Additionally, these metadata files can be large in size, leading to a significant slowdown in the user experience.

Owner

• Name: Jan Kolarik
• Email: jkolarik@redhat.com

Current status

• Targeted release: Fedora Linux 40
• Last updated: 2023-10-30
• [<will be assigned by the Wrangler> devel thread]
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Until now, filelists were always downloaded together with other metadata. This was hardcoded and unable to change from the outside of DNF.

With these changes, we are proposing to not download the filelists metadata by default. This default behavior can be modified through the new DNF configuration option. Additionally, specific commands can override this behavior and request loading the filelists metadata at runtime using the existing demands object in DNF.

Benefit to Fedora

As DNF is integral to various infrastructure tasks like package building and installation, testing environment creation, and server integration tests, this change significantly reduces processing time and resource usage for these processes.

Scope

• Proposal owners:
  • libdnf
    • Modify the Repo object to enable conditional filelists metadata download
    • Introduce a new main configuration option to set the default behavior
  • dnf
    • Introduce a new demand to enable specific commands to override filelists metadata download behavior
    • Handle demand and configuration option inputs to delegate filelists loading decision to libdnf
    • Implement filename pattern argument detection heuristics

• Other developers:
  • Dependencies using the existing DNF C interface may need to adapt and explicitly request filelist loading due to this change:
    • PackageKit
    • microdnf
    • API users

• Release engineering: N/A

• Policies and guidelines:
  • Package maintainers must follow Fedora's packaging guidelines, particularly concerning file dependency specifications (see here)

• Trademark approval: N/A

• Alignment with Community Initiatives: N/A (no currently active initiatives)

Upgrade/compatibility impact

In general, applying these changes should not affect any existing user workflows and no additional manual changes are required. However, the absence of filelists might create an issue with packages that are not correctly packaged, f.e. from third-party repositories.

How To Test

When using DNF commands without a filename pattern passed as the argument, filelists metadata should not be downloaded from the remote repositories and should not be needed for the command execution. This can be tested with the following steps:

• Clean the local metadata cache (dnf clean metadata)
• Run a DNF command not involving the filename spec (e.g. dnf repoquery rpm)
• Verify that no *-filelists.* metadata files were downloaded inside the cache subdirectories (by default under the /var/cache/dnf for root)
• Check the command works as expected

The same should also apply to RPM package arguments (files ending with .rpm extension).

When using DNF commands with a filename pattern passed as the argument, filelists metadata should be downloaded from the remote repositores as before.

User Experience

Large filelists could be over 200MB in size. It could take 1-2 minutes to download which is greatly slowing down the user experience.

For many operations the filelists metadata are not needed, so downloading them is wasting the resources. Without filelists being downloaded, DNF performance will be improved significantly, mainly regarding the network, CPU and disk space resources. Metadata download size will be reduced by about 60%. The improvement includes deployments of customer built RPMS to containers that have no need for filelists level dependencies.

Dependencies

No changes should be required for any package depending on DNF.

Contingency Plan

• Contingency mechanism: Change the configuration option to download the filelists by default
• Contingency deadline: Branch Fedora Linux 40 from Rawhide
• Blocks release? No

Documentation

Links to the relevant DNF CLI and API documentation sections will be provided here once the related pull request is created.