From Fedora Project Wiki
(Add results of informal discussion in FPC meeting today)
No edit summary
 
(63 intermediate revisions by 9 users not shown)
Line 1: Line 1:
{{autolang|base=yes}}
= Move all to /usr =
= Move all to /usr =


== Summary ==
== Summary ==
Refer to http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge first. 
Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.
Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.


/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.
/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.


Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem an /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.
Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem and /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.


== Owner ==
== Owner ==
Line 15: Line 20:


== Current status ==
== Current status ==
* Targeted release: [[Releases/17 | Fedora 17 ]]  
* Targeted release: [[Releases/17 | Fedora 17 ]]
* Last updated: 2011-09-21
* Last updated: 2012-02-07
* Percentage of completion: 5%
* Percentage of completion: 100%


== Detailed Description ==
== Detailed Description ==
Line 32: Line 37:
* /usr - installed system; shareable; possibly read-only
* /usr - installed system; shareable; possibly read-only
* /etc - config data; non-shareable
* /etc - config data; non-shareable
* /var - persistent data; non-shareable;  
* /var - persistent data; non-shareable;
* /run - volatile data; non-shareable; mandatory tmpfs filesystem
* /run - volatile data; non-shareable; mandatory tmpfs filesystem
In the process of moving /bin and /sbin to /usr/bin, /usr/sbin can be moved also to /usr/bin.


<pre>
<pre>
Line 42: Line 45:
|-- usr
|-- usr
|  |-- bin
|  |-- bin
|  |-- sbin
|  |-- lib
|  |-- lib
|  `-- lib64
|  `-- lib64
Line 47: Line 51:
|-- var
|-- var
|-- bin -> usr/bin
|-- bin -> usr/bin
|-- sbin -> usr/bin
|-- sbin -> usr/sbin
|-- lib -> usr/lib
|-- lib -> usr/lib
`-- lib64 -> usr/lib64
`-- lib64 -> usr/lib64
Line 53: Line 57:


== Benefit to Fedora ==
== Benefit to Fedora ==
* Clear separation of operating system and host specific resources.  
* Simpler and cleaner overall file system layout, with full compatibility.
* /usr can be read-only and shareable.  
* Clear separation of operating system and host specific resources.
* /usr can be easily snapshotted.
* Improve compatibility with other Unixes/linux, no confusion about tools install locations, no $PATH fiddling, all possible paths to a binary will always work. All binaries will be available on both /usr and / thus minimizing compatibility problems.
* Improve compatibility with build systems such as GNU autotools who never have been aware of the /usr split in the first place
* Minimize difference to other Unixes, such as Solaris, which already did the same move
* Isolate the vendor-supplied mostly read-only operating system resources from the rest, thus allow snapshotting of the OS, and easy lightweight container OS duplication


== Scope ==
== Scope ==
The ability to share /usr is especially useful for clusters and virtual machines.
* The ability to share /usr is especially useful for clusters and virtual machines.
The ability to mount /usr read-only (e.g. on read-only media) adds to the security of the machine.
* The ability to mount /usr read-only (e.g. on read-only media) can add to the security of the machine.
* The entire /usr can safely be snapshotted during upgrades.


== How To Test ==
== How To Test ==
<!-- This does not need to be a full-fledged document.  Describe the dimensions of tests that this feature is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.  
* instructions on how to test via yum update:
https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum#Fedora_16_-.3E_Fedora_17


Remember that you are writing this how to for interested testers to use to check out your feature - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your feature.
* install a fresh F17 with the usrmove feature included (not yet available)
 
-> see symbolic toplevel links:
A good "how to test" should answer these four questions:
<pre>
/lib -> usr/lib
/lib64 -> usr/lib64
/sbin -> usr/sbin
/bin -> usr/bin
</pre>


0. What special hardware / data / etc. is needed (if any)?
Ensure that basic shell operations work.
1. How do I prepare my system to test this feature? What packages
# /sbin/ifconfig |/bin/grep -i ip
need to be installed, config files edited, etc.?
2. What specific actions do I perform to check that the feature is
working like it's supposed to?
3. What are the expected results of those actions?
-->


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
* less toplevel directories
* fewer toplevel directories


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->


* initramfs (dracut)
* initramfs ([https://fedoraproject.org/wiki/Dracut dracut])
* changes in selinux policies
* changes in selinux policies
* repackaging of packages with content in /bin, /sbin, /lib*
* repackaging of packages with content in /bin, /sbin, /lib*
* drop consolehelper to move /usr/sbin/* to /usr/bin
* alternatives symlinks?
* alternatives symlinks?
* filesystem rpm, toplevel symlinks
* filesystem rpm, toplevel symlinks
* anaconda update support https://bugzilla.redhat.com/show_bug.cgi?id=787893


== Roadmap ==
== Roadmap ==
* Begin changing rpm packages with files in /bin, /sbin, /usr/sbin, /lib, /lib64.
* Provide a [https://fedoraproject.org/wiki/Dracut dracut] module to move all content from /bin, /sbin, /lib, /lib64 to /usr.
* Make backward compat symlinks in %post and %ghost those symlinks:
* Add check to rpm+filesystem.rpm version 3, that refuses to install itself when /bin, /sbin, /lib, /lib64 is a directory. On new installation: create symlinks /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib, /lib64 -> usr/lib64
<pre>
* Change the ~32 RPM packages which install conflicting files in /bin, /sbin, /lib, /lib64 to install only into /usr, and add Conflicts: filesystem < 3.
%post
* Change the SELinux policies.
# create compat symlink for tools as long as root directories are not converted to symlinks
* Make sure dracut is able to mount needed filesystems specifies in /etc/fstab before starting systemd.
if ! test -L /bin; then
 
    ln -s ../usr/bin/foo /bin/foo
=== Mock Transition ===
    ln -s ../usr/bin/bar /bin/bar
* Disable mock root cache in buildsystem
fi
* Step 0: update rpm on the mock system to at least:
if ! test -L /sbin; then
** RHEL-6: 4.8.0-19.el6.0.usrmove.1 ([http://people.redhat.com/harald/downloads/rpm/4.8.0-19.el6.0.usrmove.1/ Repository] until RHEL-6.3 is out)
    ln -s ../usr/bin/buz /sbin/buz
** F-15: rpm-4.9.1.2-3.fc15.3
fi
** F-16: rpm-4.9.1.2-4.fc16.1
** F-17: rpm-4.9.1.2-8.fc17
 
* Step 1: Build without filesystem conflict
  acl
  attr
  db4
  findutils
  gawk
  gettext
  gzip
  nss-softokn
  policycoreutils
  libdb
  coreutils
* Step 2: Build
  filesystem
  util-linux
  bash
* Step 3: Build
  udev
  systemd
* Step 4: Build
  alsa-utils
  davfs2
  ethtool
  fuse
  iputils
  isdn4k-utils
  libselinux
  nano
  nspr
  ncpfs
  ntfs-3g
  plymouth
  psacct
  rp-pppoe
  vim
* Step 5: Build with filesystem conflict
  acl
  attr
  db4
  findutils
  gawk
  gettext
  gzip
  libdb
  nss-softokn
  policycoreutils
  coreutils
* Turn root cache back in buildsystem mock
 
=== Koji Transition ===
* Step 0: update rpm on the koji systems to at least:
** RHEL-6: 4.8.0-19.el6.0.usrmove.1 ([http://people.redhat.com/harald/downloads/rpm/4.8.0-19.el6.0.usrmove.1/ Repository] until RHEL-6.3 is out)
** F-15: rpm-4.9.1.2-3.fc15.3
** F-16: rpm-4.9.1.2-4.fc16.1
** F-17: rpm-4.9.1.2-8.fc17
 
* Step 1: build without tagging the packages in the koji repo
  acl
  attr
  db4
  findutils
  gawk
  gettext
  gzip
  nss-softokn
  policycoreutils
  libdb
  coreutils
  filesystem
  gcc (gcc may not compile after filesystem >= 3 is installed)
 
* Step 2: tag all of the above packages in the koji repo


%files
* Step 3: build
%ghost %attr(777, root, root) /bin/foo
  util-linux
%ghost %attr(777, root, root) /bin/bar
  bash
%ghost %attr(777, root, root) /sbin/buz
  udev
</pre>
  systemd
* RPM: [http://harald.fedorapeople.org/Package-list-root.txt 257 packages] that install files in the root filesystem.
  alsa-utils
* Change SELinux policies.
  davfs2
* On new installation: create symlinks /bin -> usr/bin, /sbin -> usr/bin, /lib -> usr/lib, /lib64 -> usr/lib64, /usr/sbin -> bin. These links will take care that installed packages do not install compat symlinks in %post.
  ethtool
* Make sure dracut is able to mount needed filesystems specifies in /etc/fstab before starting systemd.
  fuse
* Drop consolehelper to enable the /usr/sbin -> /usr/bin move.
  iputils
  isdn4k-utils
  libselinux
  nano
  nspr
  ncpfs
  ntfs-3g
  plymouth
  psacct
  rp-pppoe
  vim


== Contingency Plan ==
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->


* We do not support to bootup with an empty /usr today, so moving things to /usr and have compat links in the rootfs should be low risk. If things turn out to get difficult, we can delay the creation of the /bin /sbin /lib lib64 compat links to a later release.
* We do not support to bootup with an empty /usr today, so moving things to /usr and have compat links in the rootfs should be low risk.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
*
* This page is the primary source of documentation
* [http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge] Freedesktop page listing the benefits
* [http://docs.oracle.com/cd/E23824_01/html/E24456/userenv-1.html Solaris] (has the same model for ages)
* [http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken Why /usr on a separate partition is broken currently]
* [http://thread.gmane.org/gmane.linux.redhat.fedora.devel/155511/focus=155547 Discussion on fedora-devel]
* [http://en.opensuse.org/openSUSE:Usr_merge openSUSE status (wiki) page for the merging project]
* [http://lists.opensuse.org/opensuse-factory/2011-11/msg01431.html Proposal for openSUSE]
* [http://thread.gmane.org/gmane.linux.debian.devel.general/165785 Discussion on debian-devel]
* [http://thread.gmane.org/gmane.linux.gentoo.devel/72128 Discussion on gentoo-devel] and several following threads about udev and /usr
* [http://linux.slashdot.org/story/11/11/02/2129237/fedora-aims-to-simplify-linux-filesystem Discussion on slashdot]
* Media
** [http://www.itworld.com/it-managementstrategy/218847/fedora-simplify-filesystem-hierarchy itworld.com]
** [http://www.osnews.com/story/25289/Fedora_To_Simplify_Filesystem_Hierarchy_Move_Everything_to_usr osnews.com]
** [http://h-online.com/-1369642 Heise]
** [http://heise.de/-1371441 Heise - german only]
** [http://www.linux.com/news/featured-blogs/196:zonker/540696:the-ever-changing-linux-filesystems-merging-directoris-into-usr Linux.com feature/blog]


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
*
* With this release, packages will not install files anymore in the following directories: /bin /sbin /lib /lib64 and /usr/sbin.
* Fresh installations of this release, will have the following symbolic links in the toplevel directory:
  /bin -> usr/bin
  /sbin -> usr/sbin
  /lib -> usr/lib
and for 64bit architectures
  /lib64 -> usr/lib64
 
Steps to upgrade to Fedora 17 using yum directly:
https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum#Fedora_16_-.3E_Fedora_17


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/UsrMove]] <!-- This adds a link to the "discussion" tab associated with your page. This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
* See [[Talk:Features/UsrMove]]
 
=== FAQ ===
 
==== What problem are you trying to solve? ====
We want to make /usr shareable in a sane way.
 
Additional benefits of this feature are:
* less clutter across the filesystem
* if you snapshot /usr before updating, you have snapshotted the OS at once.
 
==== What is currently broken with having /usr as a separate partition? ====
http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken
 
==== I don’t have /usr as a separate partition. What changes for me? ====
Nothing changes in functionality. All the old paths are reachable, because there a compat symlinks in place, which will not go away (at least not in the near future). All your scripts and binaries should work, like they did before. For the upgrade process to work, you will find /sbin, /bin, /lib and /lib64 mostly containing symbolic links. As soon, as these directories '''only''' contain symbolic links, the whole directory is replaced by only one symbolic link. These three or four toplevel symbolic links will stay there as long as the linux elf loader ABI is defined with “/lib/ld-linux.so.2” or their architecture specific counterpart like “/lib64/ld-linux-x86-64.so.2”, and as long as scripts use “#!/bin/sh”.
 
==== I have /usr as a separate partition. What changes for me? ====
Not sure, how you managed to do that. In general, having /usr as a separate partition does not really work right now.
See http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken.
But with this feature implemented, things will now come back to a sane and supported way of having a /usr mount point.
 
==== Why don’t you fix the /usr situation by putting all the relevant binaries in /bin /sbin /lib and /lib64? ====
Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.
 
==== So, why don’t you just mount /usr from the initramfs and leave the files where they are? ====
Ok, so imagine you have a /usr mounted from a network location and you want to update a package. So maybe you mount the master copy of /usr on your master machine and update /usr with your package manager. Then you provide a new copy of the master /usr to the other machines, when they reboot. They all have the new updated /usr now. But what about /sbin /bin /lib and /lib64? They still have the old binaries. No glibc security update for them. So, every machine has to update these directories via rsync or such (rpm will not work with a readonly /usr). This doubles the maintenance to keep both parts of the system in sync.
 
==== You are doing it wrong! /bin and /sbin are there to rescue a broken /usr! ====
The most critical filesystem is /boot, because the kernel lives there. So the purpose of having /bin and /sbin for /usr repairing relied on _two_ working filesystems ( / and /boot). If either of them was broken, you were not able to rescue /usr. The role of the rescue system can easily be fulfilled by a rescue initramfs.  So having the rescue initramfs in /boot, which contains the fsck utils, is in the same danger of becoming corrupted as the kernel. Now you only have to pull out your rescue CD, if /boot is corrupted and not if / is corrupted. 
 
==== Then, let’s share /bin /sbin /lib /lib64 and /usr and mount them all from the initramfs! ====
Now, you get a feeling, that moving everything to /usr might make things easier....
 
==== Why don’t you move all /usr contents to / and forget about /usr? ====
Because this introduces a '''lot''' of new toplevel directories, which all have to be mount points then to be shared across other hosts.


* The contingency plan is flawed.  Disregarding the compat symlinks is not an option as those will be needed for third party scripts, FHS compliance, user comfort, etc.
==== Ok, but what about a root filesystem on the network and mounting local filesystems only? ====
* I would strongly suggest considering the /sbin/ => /bin/ and /{s,}}bin => /usr/{s,}bin separately.  There's different considerations for each (for instance, the consolehelper porting is only necessary for the /sbin/ => /bin/ move, not for the / to /usr/ move.)
Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.
* The statement "Historically /bin, /sbin, /lib had the purpose to contain the utilities to mount /usr" is only one reason for the split.  The FHS specifies other reasons: http://pathname.com/fhs/pub/fhs-2.3.html#PURPOSE2.  The one that sticks out as unaddressed is "Disk errors that corrupt data on the root filesystem are a greater problem than errors on any other partition. A small root filesystem is less prone to corruption as the result of a system crash."  This is both helped (the root filesystem will indeed be smaller) and hindered (but the root filesystem is essential in part because it contains the tools necessary to recover from disk errors... which would no longer bethe case) by this change.
* Additionally, the FHS is not only specifying these directories for purely technical reasons but also for organizational reasons.  For instance, /bin contains "Essential" user command binaries. The section of FHS pointed at above would seem to indicate a definition of "Essential" that includes not only what's necessary to boot (what this feature is saying is no longer necessary) but also the commands necessary to recover, repair, or restore a system.  The /bin and /sbin split is even more a separation due to organization of binaries rather than a need to do that to prevent breaking the system.
* Many of the proposed benefits of this feature, such as "/usr can be read-only and shareable. " are already present in the current /usr.  For all such benefits, the benefit is actually that more files are going to be included on /usr (or the inverse, that less will be included on /).
* For dependencies, the Feature also needs to make sure that it works for the non-programming/library files that are stored in those directories on /. For instance, /lib/udev, /lib/systemd, /lib/terminfo....  These may all be fine but they need to be considered as dependencies that may need changing.
* FPC should probably be involved as obeying the FHS is a packaging mandate (they'd want to evaluate whether any of this violates the FHS or not and whether it seems justified to make an exception if so) and also, if individual packages (like coreutils) were to implement symlinks, then how to deal with that (making symlinks for the major directories in filesystem may sidestep, that, though).
** FPC took an informal look at this today.  There's a consensus that we'd be against the /usr/bin, /usr/sbin merge.  Merging /bin, /sbin, /lib, /lib64 into their /usr/ counterparts and keeping the compat symlinks will probably take some Guidelines updates but out initial view is that whether or not to go forward with that portion is up to FESCo.
* anaconda and the docs team are additional dependencies as documentation and code that recommends relative partition sizes would need to be updated.


[[Category:FeatureReadyForWrangler]]
[[Category:FeatureAcceptedF17]]
<!-- When your feature page is completed and ready for review -->
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 12:52, 29 March 2012

Move all to /usr

Summary

Refer to http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge first.

Provide a simple way of mounting almost the entire installed operating system read-only, atomically snapshot it, or share it between multiple hosts to save maintenance and space. Instead of spreading RPM package content all over the place in the filesystem, and artificially separate /bin from /usr/bin and /lib from /usr/lib, move all content to /usr and provide only symlinks in the root filesystem.

/usr on its own filesystem provides a lot of valuable options in custom setups. For historic reasons, we split-off more and more tools from /usr and put them in /. But, advanced features in today's systems can not really bootup with an empty /usr anymore. More and more fails in subtle ways in such setups.

Instead of moving more tools to /, we today already require /usr to be mounted from inside the initramfs, to be available before the real 'init' starts. The split of the root filesystem and /usr serves no purpose in Linux anymore and only complicates or prevents simple and more flexible setups.

Owner

Current status

  • Targeted release: Fedora 17
  • Last updated: 2012-02-07
  • Percentage of completion: 100%

Detailed Description

There is no way to reliably bring up a modern system with an empty /usr, there are two alternatives to fix it: copy /usr back to the rootfs or use an initramfs which can hide the split-off from the system.

Historically /bin, /sbin, /lib had the purpose to contain the utilities to mount /usr. This role can now be taken by the initramfs. Because the initramfs knows, where to find the root partition (which includes /etc), it can parse /etc/fstab and other configuration files and mount /usr before it finally switches the root partition and executes /usr/bin/init. From this point on init mounts the remaining partitions in /etc/fstab and the system starts as usual.

The long-term plan is to clean up the mess and confusion the current split of / vs. /usr has created. All tools will move back to /usr where they belong, and the rootfs will only contain compat-symlinks into /usr. Almost the entire system installed by packages will reside in /usr. This will split all non-host specific data to /usr. /usr can then be seen as the Unix System Resources partition (/System), which defines the base operating system (e.g. F18 or RHEL-7).

This new /usr could be mounted read-only by default, while the rootfs is read-write and contains only empty mount points, compat-symlinks to /usr and the host-specific data like /etc, /root, /srv. Compared to today's setups, the rootfs will be very small. The new /usr could also easily be shared read-only across several systems, and it would contain almost the entire system. Such setups are more efficient, can optionally provide a lot more security, are more flexible, provide more sane options for custom setups, and are much simpler to setup and maintain.

This leaves us with the following well-defined directories, which compose the base of the system:

  • /usr - installed system; shareable; possibly read-only
  • /etc - config data; non-shareable
  • /var - persistent data; non-shareable;
  • /run - volatile data; non-shareable; mandatory tmpfs filesystem
/
|-- etc
|-- usr
|   |-- bin
|   |-- sbin
|   |-- lib
|   `-- lib64
|-- run
|-- var
|-- bin -> usr/bin
|-- sbin -> usr/sbin
|-- lib -> usr/lib
`-- lib64 -> usr/lib64

Benefit to Fedora

  • Simpler and cleaner overall file system layout, with full compatibility.
  • Clear separation of operating system and host specific resources.
  • Improve compatibility with other Unixes/linux, no confusion about tools install locations, no $PATH fiddling, all possible paths to a binary will always work. All binaries will be available on both /usr and / thus minimizing compatibility problems.
  • Improve compatibility with build systems such as GNU autotools who never have been aware of the /usr split in the first place
  • Minimize difference to other Unixes, such as Solaris, which already did the same move
  • Isolate the vendor-supplied mostly read-only operating system resources from the rest, thus allow snapshotting of the OS, and easy lightweight container OS duplication

Scope

  • The ability to share /usr is especially useful for clusters and virtual machines.
  • The ability to mount /usr read-only (e.g. on read-only media) can add to the security of the machine.
  • The entire /usr can safely be snapshotted during upgrades.

How To Test

  • instructions on how to test via yum update:

https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum#Fedora_16_-.3E_Fedora_17

  • install a fresh F17 with the usrmove feature included (not yet available)

-> see symbolic toplevel links:

 /lib -> usr/lib
 /lib64 -> usr/lib64
 /sbin -> usr/sbin
 /bin -> usr/bin

Ensure that basic shell operations work.

# /sbin/ifconfig |/bin/grep -i ip

User Experience

  • fewer toplevel directories

Dependencies

Roadmap

  • Provide a dracut module to move all content from /bin, /sbin, /lib, /lib64 to /usr.
  • Add check to rpm+filesystem.rpm version 3, that refuses to install itself when /bin, /sbin, /lib, /lib64 is a directory. On new installation: create symlinks /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib, /lib64 -> usr/lib64
  • Change the ~32 RPM packages which install conflicting files in /bin, /sbin, /lib, /lib64 to install only into /usr, and add Conflicts: filesystem < 3.
  • Change the SELinux policies.
  • Make sure dracut is able to mount needed filesystems specifies in /etc/fstab before starting systemd.

Mock Transition

  • Disable mock root cache in buildsystem
  • Step 0: update rpm on the mock system to at least:
    • RHEL-6: 4.8.0-19.el6.0.usrmove.1 (Repository until RHEL-6.3 is out)
    • F-15: rpm-4.9.1.2-3.fc15.3
    • F-16: rpm-4.9.1.2-4.fc16.1
    • F-17: rpm-4.9.1.2-8.fc17
  • Step 1: Build without filesystem conflict
 acl
 attr
 db4
 findutils
 gawk
 gettext
 gzip
 nss-softokn
 policycoreutils
 libdb
 coreutils
  • Step 2: Build
 filesystem
 util-linux
 bash
  • Step 3: Build
 udev
 systemd
  • Step 4: Build
 alsa-utils
 davfs2
 ethtool
 fuse
 iputils
 isdn4k-utils
 libselinux
 nano
 nspr
 ncpfs
 ntfs-3g
 plymouth
 psacct
 rp-pppoe
 vim
  • Step 5: Build with filesystem conflict
 acl
 attr
 db4
 findutils
 gawk
 gettext
 gzip
 libdb
 nss-softokn
 policycoreutils
 coreutils
  • Turn root cache back in buildsystem mock

Koji Transition

  • Step 0: update rpm on the koji systems to at least:
    • RHEL-6: 4.8.0-19.el6.0.usrmove.1 (Repository until RHEL-6.3 is out)
    • F-15: rpm-4.9.1.2-3.fc15.3
    • F-16: rpm-4.9.1.2-4.fc16.1
    • F-17: rpm-4.9.1.2-8.fc17
  • Step 1: build without tagging the packages in the koji repo
 acl
 attr
 db4
 findutils
 gawk
 gettext
 gzip
 nss-softokn
 policycoreutils
 libdb
 coreutils
 filesystem
 gcc (gcc may not compile after filesystem >= 3 is installed)
  • Step 2: tag all of the above packages in the koji repo
  • Step 3: build
 util-linux
 bash
 udev
 systemd
 alsa-utils
 davfs2
 ethtool
 fuse
 iputils
 isdn4k-utils
 libselinux
 nano
 nspr
 ncpfs
 ntfs-3g
 plymouth
 psacct
 rp-pppoe
 vim

Contingency Plan

  • We do not support to bootup with an empty /usr today, so moving things to /usr and have compat links in the rootfs should be low risk.

Documentation

Release Notes

  • With this release, packages will not install files anymore in the following directories: /bin /sbin /lib /lib64 and /usr/sbin.
  • Fresh installations of this release, will have the following symbolic links in the toplevel directory:
 /bin -> usr/bin
 /sbin -> usr/sbin
 /lib -> usr/lib

and for 64bit architectures

 /lib64 -> usr/lib64

Steps to upgrade to Fedora 17 using yum directly: https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum#Fedora_16_-.3E_Fedora_17

Comments and Discussion

FAQ

What problem are you trying to solve?

We want to make /usr shareable in a sane way.

Additional benefits of this feature are:

  • less clutter across the filesystem
  • if you snapshot /usr before updating, you have snapshotted the OS at once.

What is currently broken with having /usr as a separate partition?

http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken

I don’t have /usr as a separate partition. What changes for me?

Nothing changes in functionality. All the old paths are reachable, because there a compat symlinks in place, which will not go away (at least not in the near future). All your scripts and binaries should work, like they did before. For the upgrade process to work, you will find /sbin, /bin, /lib and /lib64 mostly containing symbolic links. As soon, as these directories only contain symbolic links, the whole directory is replaced by only one symbolic link. These three or four toplevel symbolic links will stay there as long as the linux elf loader ABI is defined with “/lib/ld-linux.so.2” or their architecture specific counterpart like “/lib64/ld-linux-x86-64.so.2”, and as long as scripts use “#!/bin/sh”.

I have /usr as a separate partition. What changes for me?

Not sure, how you managed to do that. In general, having /usr as a separate partition does not really work right now. See http://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken. But with this feature implemented, things will now come back to a sane and supported way of having a /usr mount point.

Why don’t you fix the /usr situation by putting all the relevant binaries in /bin /sbin /lib and /lib64?

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.

So, why don’t you just mount /usr from the initramfs and leave the files where they are?

Ok, so imagine you have a /usr mounted from a network location and you want to update a package. So maybe you mount the master copy of /usr on your master machine and update /usr with your package manager. Then you provide a new copy of the master /usr to the other machines, when they reboot. They all have the new updated /usr now. But what about /sbin /bin /lib and /lib64? They still have the old binaries. No glibc security update for them. So, every machine has to update these directories via rsync or such (rpm will not work with a readonly /usr). This doubles the maintenance to keep both parts of the system in sync.

You are doing it wrong! /bin and /sbin are there to rescue a broken /usr!

The most critical filesystem is /boot, because the kernel lives there. So the purpose of having /bin and /sbin for /usr repairing relied on _two_ working filesystems ( / and /boot). If either of them was broken, you were not able to rescue /usr. The role of the rescue system can easily be fulfilled by a rescue initramfs. So having the rescue initramfs in /boot, which contains the fsck utils, is in the same danger of becoming corrupted as the kernel. Now you only have to pull out your rescue CD, if /boot is corrupted and not if / is corrupted.

Then, let’s share /bin /sbin /lib /lib64 and /usr and mount them all from the initramfs!

Now, you get a feeling, that moving everything to /usr might make things easier....

Why don’t you move all /usr contents to / and forget about /usr?

Because this introduces a lot of new toplevel directories, which all have to be mount points then to be shared across other hosts.

Ok, but what about a root filesystem on the network and mounting local filesystems only?

Then you would share the toplevel directory hierarchy among all hosts. Hosts would need to mount /etc and /var for host-only versions. Especially /etc/fstab is not accessible, without adding information to the initramfs on how to mount it. For every host-only additional top level directory like /opt and /srv, you would have to have a mountpoint.