From Fedora Project Wiki

< Changes

Revision as of 22:22, 12 April 2017 by Bowlofeggs (talk | contribs) (→‎Proposal: Update the proposal.)


Fedora Scale-Out Docker Registry

Summary

This is a proposal for a change to the Fedora Infrastructure and Fedora Release Engineering tooling to provide a scalable container registry solution for Fedora that is integrated with the Fedora Docker Layered Image Build Service.

Owner

  • Name: Adam Miller and Randy Barlow
  • Email: maxamillion@fedoraproject.org and bowlofeggs@fedoraproject.org
  • Release notes owner:

Current status

Detailed Description

Background

The Fedora project wishes to begin distributing new types of content than it has in the past. One of the types that has been identified as a goal are container images. Adam Miller has already done the work that will allow packagers to build container images, but we still need a way to distribute those builds to Fedora's users. Adam Miller's implementation helpfully drops the builds we want into a container registry.

registry: a collection of container image repositories

repository: named after an image and is a collection of multiple tags of an that image

tag: an arbitrary string assigned to a specific container image (identified by the image's sha256 checksum) NOTE: The "latest" tag is special and is assumed if no tag is provided. This is true also for a 'docker pull' operation and an image tagged "latest" will be the default image pulled by users.

Proposal

In summary, the proposal is to deploy the docker distribution registry at

registry.fedoraproject.org

, which will serve the container registry API to Fedora's users. Users will fetch all API data from this endpoint, except for the container blobs. Fedora will serve 302 redirects for all requests for container blobs to

cdn.registry.fedoraproject.org

. The CDN will handle serving the large blob files to the users.

  • Docker Distribution is the defacto standard open source implementation of the Docker Registry V2 API spec.

Workflow

  • OSBS will perform Builds, as these builds complete they will be pushed to the docker-distribution (v2) registry, these will be considered "candidate images". These will be stored in candidate repositories on the docker-distribution registry.
  • Testing will occur using the "candidate images" (details of how we want to handle that are outside the scope of this proposal).
  • A "candidate image" will be marked stable once it's criteria have been satisfied to do so. (This is vague because this is a topic of ongoing discussion and work to decide what criteria an image will need to abide by before being considered "stable" and promoted as such)
  • Once stable, the images will be pushed into stable repositories in the docker-distribution registry.
  • The new Python tool will split that content and sync the image layers along with their metadata to Mirror Manager master mirror. It will also sync the repo metadata to somewhere Mirror List can pick it up.
  • Mirror Manager will distribute to the mirrors the image layers.
  • The docker clients will request Manifests from Mirror List. Mirror list will return schema 2 Manifests that contain lists of URLs to the mirrors where the client can retrieve the blobs.

Mirror List

Users will be pointing their docker clients at Mirror List when they docker pull Fedora's Docker images. In order for this to work, we will need to make two changes to Mirror List so that it can respond to the docker client properly. The first change is that Mirror List will need to respond with a special header and a body of "{}" when the docker client sends a GET request for /v2/. The second change is that it will need to return a Docker Manifest schema 2 document containing a list of mirrors that have the requested blobs when the client makes additional requests, so that the clients can be retrieve the blobs from a list of mirrors near their locations, similar to how it does with the dnf client today.

The docker client typically connects to port 5000. We could run a second instance of Mirror List on port 5000 if we wanted to isolate it from the current instance. We can also have the docker client pull from 443 as dnf does if we want to keep the deployment simpler.

Mirror Manager

We will need to make a few changes to Mirror Manager as well. We will need to provide an interface to allow mirror admins to opt in/out of mirroring Docker content. We will also need to modify the curler to detect whether a given mirror is up to date or not. We will need to make sure that UMDL is updated when content changes.

There was some discussion about how the Docker content would be organized on the master mirror. We could either give an all-or-nothing Docker module for mirrors to choose from, or we could split the Docker content by arch (or perhaps even primary vs. secondary). To allow multi-arch support, Mirror List will use Docker Manifest Lists and allow the client to request the needed Manifest by digest. This would mean that the second request from the client would be when Mirror List could pick a list of mirrors that it knew had the requested content. We will not create a module per Docker repository since there could be hundreds or thousands of them.

There is an open question about how we would deal with archived data.

docker

The new schema 2 manifest that Docker has defined has a new feature that was not part of schema 1: a list of URLs for each Blob can be listed in the Manifest[7].

Unfortunately, the feature only works for the Windows version of the Docker client at this time. The original pull request[8] was submitted by Microsoft and only worked for the Windows client. Later, Antonio Murdaca submitted a pull request[6] to expand the support for other operating systems, but it was not accepted. In response to all that, he opened an issue[8] to request the feature be expanded.

The plan is to use the patch by Antonio in Fedora's docker client.

New Tool

The last piece that is needed is a tool that can create the filesystem tree that we want to synchronize out to the mirrors. The mirrors only need to carry blobs, so the tool needs only to pull these binaries out of the registry that Adam Miller has set up and write them to disk in a particular structure. For optimization, we could use hardlinks for blobs that are common across the various images (for example, the Fedora base blob will be the same in all images) to save rsync time and mirror disk space.

Additionally, we will need a playbook to run this new tool in response to fedmsgs. We may be able to use Adam Miller's loopabull project to run such a playbook at the right times.

Signing

For the initial implementation of the Fedora Docker registry, we will not be signing the images. This will still be safe for our users, as the manifests will be served by Mirror List only (the mirrors will not be serving manifests or any metadata) and only over TLS. Docker manifests reference the blob layers by checksum and the client does verify the checksums of the layers it downloads. Thus we will rely on TLS to safely transmit the checksums of the blobs to the end user, and we will rely on their client to validate the checksums of the blobs it downloads from the mirrors.

We may revisit signing in the future when there are more available choices for us to use as an added layer of security.

Optional Mirror Registries

A notable drawback to this proposal is that users will not be able to point their docker client directly at a mirror and docker pull. This is due to the docker client not supporting a path to the docker v2 API, and to it expecting to see certain headers in the response. Instead, users will always have to point their clients at mirror list so that it can send them the manifest with URLs to the blobs on the mirrors.

However, we could have a "phase 2" plan, where we ask mirrors to consider running their own full registries for users to pull from. Of course, this would require opt-in and hands on work by the mirror admins (similar to how some mirrors support ftp or rsync, but not all do). Without a registry on the mirror, there isn't a good way to allow users to docker pull directly from a specific mirror.

General Notes

A couple of things to note about maintenance and uptime considerations:

The Intermediate docker-distribution registry is needed for builds in koji+OSBS.

Much of the current design was discussed on the infrastructure mailing list[1].

All new components in this design should be able to be locked down, similar to the "Fedora internal" components like koji (builders, etc) and bodhi (signing, etc).

Benefit to Fedora

This will allow for Fedora to provide packages, software, and other content in the form of a Docker Image as an officially released artifact from the Fedora Project that is released and hosted much in the same way RPMs are today. These images can then be included in the distribution in various ways. This could potentially be used by the Modularization effort or by any other part of the Fedora.next initiative that may arise.

Scope

Proposal owners

Proposal owners shall have to:

  • Implement the proposed Design of a Scaled-Out Docker Registry
    • Build the new Python Tool to pull blobs out of the registry
    • Deploy Docker-Distribution Registry
    • Integrate with MirrorManager for content distribution
  • Document the system

Task matrix

This is a RACI matrix for tasks required to implement the RelEng Automation Workflow Engine. Work is tracked in Taiga: http://taiga.cloud.fedoraproject.org/project/acarter-fedora-docker-atomic-tooling/wiki/home


Is this current?

It is, as of 2017-04-12

Definitions

Here, we're using what Wikipedia calls "RACI (alternative scheme)":


Responsible
The person responsible for the performance of the task. There should be exactly one person with this assignment for each task.
Assists
Those who assist completion of the task.
Consulted
Those whose opinions are sought; and with whom there is two-way communication.
Informed
Those who are kept up-to-date on progress; and with whom there is one-way communication.

Task Table

Note.png
This an early cut. Please feel free to add new tasks as appropriate — just let one of the coordinators know.
Task Subtask Responsible Assists Consulted Informed Current Status
Implement the proposed design of a Scaled-Out Docker Registry Adam Miller 0%
Deploy solution, including ansible playbooks added for Fedora Infrastructure Ansible repo Adam Miller 0%
Deploy docker-distribution registry Adam Miller 0%
Integrate with MirrorManager for content distribution Adam Miller 0%
Document the system Adam Miller 0%

Glossary of Nicknames

Various Task Notes

Functional Requirements

The following features are functional requirements

  • Users must be able to perform a
    docker pull registry.fedoraproject.org/fedora
    and have the actual image layer data come from a local mirror via mirrormanager.

Other developers

  • (anything here)?

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

Once the service is deployed, users can perform the following on their systems to test.

$ dnf -y install docker
$ systemctl start docker
$ docker pull registry.fedoraproject.org/fedora

N/A (not a System Wide Change)

User Experience

N/A (not a System Wide Change)

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? No (not a System Wide Change)
  • Blocks product? N/A

Documentation

FIXME

Release Notes