From Fedora Project Wiki
No edit summary
(Change was previously announced on mailing list)
Line 200: Line 200:
-->
-->


[[Category:ChangeReadyForWrangler]]
[[Category:ChangeAnnounced]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->

Revision as of 16:52, 13 January 2020


Reduce installation media size by improving the compression ratio of SquashFS filesystem

Summary

Improve compression ratio of SquashFS filesystem on the installation media.

Owner

Current status

  • Targeted release: Fedora 32
  • Last updated: 2020-01-13
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

As of Fedora 31, the LiveOS/squashfs.img file on the installation image, is compressed with default settings of mksquashfs. The standard configuration is set to XZ algorithm with block size of 128k and BCJ filter enabled. Those parameters can be adjusted which will lead to a better compression ratio and/or reduction of the CPU usage at build time. Comparison of different SquashFS compression options

Unless other specified, all tests use the default configuration plus additional parameters listed. In column A

  • The ext4 filesystem contains duplicate files. I used hardlink utility to de-duplicate them. 12.43s of hardlink run time is included
    • I was unable to boot plain SquashFS images. Perhaps an issue with dracut initialization scripts: unable to find /dev/mapper/live-rw

This is simple to achieve. Recently, Lorax has gotten support[1] for adjusting the compression options for mksquashfs via the configuration file. The file should be altered as following:

[compression]
bcj = yes
args = -b 1M -Xdict-size 1M -no-recovery

Where -b 1M and -Xdict-size 1M are block and dictionary sizes respectively. Could be adjusted.

The compression algorithm is to be selected. Below you can see the impact of the compression algorithms on the install time. With current kernel configuration:

Different compression options and their impact on the installation time

Benefit to Fedora

  • Reduction of the installation media size and the cost of storing and distributing Fedora.
  • Reduction of the CPU usage at build time. Depending on which compression parameters chosen.

Scope

  • Proposal owners:

The build environment should have support for adjusting the Lorax configuration file. Lorax is a program that produces the LiveOS/squashfs.img file on the installation media.

One of the way to enable such customization is to introduce support in Pungi to pass -c option to Lorax.

  • Other developers:

The pungi utility should support passing the custom configuration file location to the Lorax utility. This option should apply during buildInstall phase of pungi.

  • Release engineering: [1]
  • Policies and guidelines: Not required.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

This change comes at a cost of higher memory usage during the installation. Based on my personal estimations, this should not be the issue. Since the decompression should require up to 1MiB per thread. At the moment, the decompression of SquashFS is single-threaded.

N/A (not a System Wide Change)

How To Test

mkdir -p /mnt/new /mnt/old
sudo mount -o loop,ro FedoraInstallationOld.iso
sudo mount -o loop,ro FedoraInstallationNew.iso
ls -l /mnt/{new,old}/LiveOS/squashfs.img

And then calculate the size difference.

User Experience

  • Decreasing the installation image size will reduce cost of mirroring and storing Fedora installation images.
  • Decreasing the installation image size will reduce the download time.
  • Increasing the block size on the current configuration with EXT4 file system, should increase latency while accessing the EXT4 filesystem. The exact impact is to be evaluated.
  • The impact of latency will be reduced, if the plain SquashFS option is be choosen.

Dependencies

Pungi, a utility that builds the compose, should include new functionality mentioned above. Alternatively, the /etc/lorax/lorax.conf should be altered in the environment where Lorax is running.

Contingency Plan

N/A

Documentation

https://pagure.io/releng/issue/9127.
mksquashfs(1)
lorax(1)
https://docs.pagure.org/pungi

Release Notes