User:Roshi/QA/Atomic Release Criteria

Draft
This page is currently a draft. Once it's vetted through QA and the Atomic WG, it'll move to a more permanent location

Atomic WG Objectives

The objectives of the Atomic WG are to:

1. Release a new OStree and images every two weeks
2. Ensure that upgrades to the latest release work
3. Ensure that rolling back to a previous OStree works

2 Week Release Criteria

Installer iso must work

All Fedora Atomic Host installer images must boot in their supported configurations.

What are "supported configurations?"

The Fedora Atomic Host installer should follow all the same installation criteria used for Workstation and Server (where it makes sense).

What do you mean by "where it makes sense?"

A good example of something that Atomic would be allowed to break is "Package Set Selection" in the Beta criteria. Because of what ostree is, we don't care about installing packages with anaconda, just that the ostree gets written.

Atomic host must setup storage

All Fedora Atomic images (Installer and the various images) must correctly instantiate the storage pools for the installed container runtime.

Container runtime

Currently the targeted container runtime is Docker, and docker-storage-setup is used to prepare the storage for containers. This criterion is worded such that if we change the runtime in the future, we don't have to change the criterion.

"Correctly?"

Fedora Atomic host uses the devicemapper backend for docker storage. For Fedora 26 onwards, we'll be moving to overlayfs to handle storage. Each of these requires a different setup, so the intent here is that whatever backend we ship gets set up properly.

Upgrade

When a new ostree is composed, it must be possible to upgrade to it with an existing host via both the atomic host upgrade and rpm-ostree upgrade commands.

Rollback

After upgrading to the latest tree, it must be possible to downgrade back to the previous tree with both the atomic host rollback and rpm-ostree rollback commands.

Only on existing installs

Because an installer image or recently generated qcows/vagrant/raw image only ships the latest tree, it doesn't make sense to require these instances to be able to downgrade. This is only meant for existing Fedora Atomic Hosts that have installed a new tree.

Deploy

After installing or upgrading to the latest tree, it must be possible to deploy an older version with the atomic host deploy and rpm-ostree deploy commands.

Deploying older content

This should be possible from a system that thas been upgraded or a freshly installed system.

Package Layering

After installing or upgrading to the latest tree, it must be possible to layer additional packages using the atomic host install and rpm-ostree install commands.

Layering packages

Package layering is a key feature in the rpm-ostree stack, so we should make sure it works. This should work regardless if the host was upgraded or freshly installed.

Cloud-init

On virtualized atomic host images cloud-init must support the following features:

1. pass user defined ssh key into the guest
2. set hostname
3. run arbitrary user defined scripts

Why only those features?

cloud-init has a large variety of options that it can work with - not all of those options make sense when it comes to an Atomic host (like package installation). These are listed as the bare minimum of features that have to work for base functionality.

Container runtime

On any atomic host the installed container runtime (which is currently docker) must be able to run containers.

What does "run containers" mean?

On an installed system, you should be able to execute docker run <docker container> as well as atomic run <container> with no errors.

Container Service

The container runtime service must start properly.

References

• Stolen and adapted from Fedora Final release criterion.
• Test case: QA:Testcase_Services_start

System Containers

The Atomic Host should be able to run 'system containers' without problems. This is a two-step process - pulling the system container to ostree storage and then installing the container to the host. A good example of a system container is the etcd image.

# atomic pull --storage ostree registry.fedoraproject.org/f25/etcd:latest
# atomic install --system --name etcd registry.fedoraproject.org/f25/etcd:latest
# systemctl start etcd
# systemctl status etcd

What are system containers?

System containers are OCI images that are pulled to ostree storage and then started using the runc container runtume.