From Fedora Project Wiki
mNo edit summary
m (Put names and email addresses on one line)
 
(5 intermediate revisions by 2 users not shown)
Line 9: Line 9:
This should link to your home wiki page so we know who you are.  
This should link to your home wiki page so we know who you are.  
-->
-->
* Name: [[User:Jdieter| Jonathan Dieter]]
* Name: [[User:Jdieter| Jonathan Dieter]], [[User:Ngompa| Neal Gompa]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: jdieter@gmail.com
* Email: jdieter@gmail.com, ngompa13@gmail.com
* Name: [[User:Ngompa| Neal Gompa]]
* Email: ngompa13@gmail.com
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
Line 34: Line 32:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug:  [https://bugzilla.redhat.com/show_bug.cgi?id=1601595 1601595]
* Tracker bug:  [https://bugzilla.redhat.com/show_bug.cgi?id=1601595 #1601595]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/285 #285]


== Detailed Description ==
== Detailed Description ==
Line 54: Line 53:
*** <s>https://github.com/rpm-software-management/libdnf/pull/478</s>
*** <s>https://github.com/rpm-software-management/libdnf/pull/478</s>
*** <s>https://github.com/rpm-software-management/createrepo_c/pull/92</s>
*** <s>https://github.com/rpm-software-management/createrepo_c/pull/92</s>
**<s>Create a new package for Fedora's zchunk dictionaries</s> - See https://bugzilla.redhat.com/show_bug.cgi?id=1656627
** <s>Create a new package for Fedora's zchunk dictionaries</s> - See https://bugzilla.redhat.com/show_bug.cgi?id=1656627


* Other developers: Fedora Infrastructure needs to start creating zchunked metadata - In progress
* Other developers:
** <s>Fedora Infrastructure needs to start creating zchunked metadata</s> Generated since 2018-12-27
** <s>Build zchunk-enabled libdnf</s> (see PR at https://github.com/rpm-software-management/libdnf/pull/663)
** <s>Build librepo with patch to properly calculate download size for zchunk metadata</s> (see PR at https://github.com/rpm-software-management/librepo/pull/138)
** <s>Build createrepo_c with patch to generate zchunked metadata for all records</s> (see PR at https://github.com/rpm-software-management/createrepo_c/pull/115)


* Release engineering: [https://pagure.io/releng/issue/7600 #7600]
* Release engineering: [https://pagure.io/releng/issue/7600 #7600]
Line 86: Line 89:
# Remove /var/cache/dnf/*
# Remove /var/cache/dnf/*
# Run dnf update
# Run dnf update
# Check that the files in /var/cache/dnf/updates-*/repodata have .zck extensions
# Check that the files in /var/cache/dnf/rawhide-*/repodata have .zck extensions


Check that zchunk metadata won't be redownloaded when it matches the current metadata:
Check that zchunk metadata won't be redownloaded when it matches the current metadata:
# Remove /var/cache/dnf/updates-*/repodata/repomd.xml
# Remove /var/cache/dnf/rawhide-*/repodata/repomd.xml
# Run dnf update
# Run dnf update
# Check that the repodata download size matches the size of ''just'' repomd.xml
# Check that the repodata download size matches the size of ''just'' repomd.xml


Check that only different chunks are being downloaded:<br/>
Check that only different chunks are being downloaded:<br/>
''Todo: setup two snapshots of updates that are a few days apart''
# Run dnf update
# Create /etc/yum.repos.d/test.repo that contains the following lines:
# Wait 24-48 hours
#:[test]
# Run dnf update again
#:name=Fedora $releasever - $basearch - Test
# Validate that per-repository metadata download is less than 60MB (should be less than 5MB)
#:failovermethod=priority
#:baseurl=first_snapshot_from_todo
#:enabled=0
#:gpgcheck=0
#:skip_if_unavailable=True
#:
# Run dnf --enablerepo=test update
# Change /etc/yum.repos.d/test.repo as follows:
#: baseurl=second_snapshot_from_todo
#:
# Run dnf --enablerepo=test update
# Check that the repodata download size matches ''delta size from todo'' bytes


== User Experience ==
== User Experience ==
Line 126: Line 117:
*** First disable the downloading of zchunk metadata
*** First disable the downloading of zchunk metadata
*** If that doesn't fix it, we will revert the zchunk patches
*** If that doesn't fix it, we will revert the zchunk patches
* Contingency deadline: 2018-08-28 (Beta Freeze)
* Contingency deadline: 2019-03-05 (Beta Freeze)
* Blocks release? No, as long as the contingency is enacted
* Blocks release? No, as long as the contingency is enacted
* Blocks product? No
* Blocks product? No

Latest revision as of 19:20, 25 February 2019

Zchunk Metadata

Summary

All dnf repository metadata will be compressed with the zchunk format in addition to xz or gzip.

Owner

Current status

Detailed Description

Currently Fedora's repository metadata is compressed using the xz and gzip formats. Zchunk is a new compression format designed to allow for highly efficient deltas. When Fedora's metadata is compressed using zchunk, dnf will download only the differences between any earlier copies of the metadata and the current version.

Benefit to Fedora

Dnf and related tools will see significant reductions in the size of the metadata they download, especially if they are run on a regular basis.

Scope

  • Policies and guidelines: Packaging guidelines are not affected by this change.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Old-style xz and gzip metadata will continue to be generated by createrepo_c, so old versions of Fedora will use the xz/gzip metadata while upgrading to the new version, and then use the new zchunk metadata after the upgrade.

How To Test

Check that zchunk metadata is being downloaded:

  1. Remove /var/cache/dnf/*
  2. Run dnf update
  3. Check that the files in /var/cache/dnf/rawhide-*/repodata have .zck extensions

Check that zchunk metadata won't be redownloaded when it matches the current metadata:

  1. Remove /var/cache/dnf/rawhide-*/repodata/repomd.xml
  2. Run dnf update
  3. Check that the repodata download size matches the size of just repomd.xml

Check that only different chunks are being downloaded:

  1. Run dnf update
  2. Wait 24-48 hours
  3. Run dnf update again
  4. Validate that per-repository metadata download is less than 60MB (should be less than 5MB)

User Experience

This change will allow Fedora users to see significant reductions in the size of the metadata they download. If they check for updates daily, they should see a savings of up to 95%.

Dependencies

Upstream needs to merge the zchunk pull requests, and Fedora 29's dnf, libdnf, librepo, libsolv and createrepo_c will need to pull the patches in.

Contingency Plan

  • Contingency mechanism:
    • If there are problems generating zchunked metadata, Fedora Infrastructure will need to disable zchunk metadata generation
    • If there are problems with the zchunk dnf/libdnf integration, we will:
      • First disable the downloading of zchunk metadata
      • If that doesn't fix it, we will revert the zchunk patches
  • Contingency deadline: 2019-03-05 (Beta Freeze)
  • Blocks release? No, as long as the contingency is enacted
  • Blocks product? No

Documentation

Introduction to zchunk
What is zchunk and how does it work?
How to use the zchunk utilities
The zchunk file format

Release Notes