Infrastructure/CoreExtrasMerge

Merger of Core and Extras

This is finally upon us. Below you will find a rough outline of the mechanics of the merger.

What will be done?

Our current plan is to merge just the development trees for now (aka rawhide). This prevents us from breaking updates for existing releases. Once the development merger has been smoothed out and other parts of the puzzle are ready, we will merge Fedora 6 content as well.

Tasks

These are the necessary tasks to complete the merger. I've put a name/names by the task for whom I think should be responsible for making it happen:

• Lock CVS/Buildsystem for devel on Wednesday morning - Jesse / Dennis DONE
• Rsync latest core package sets to koji storage - Jesse DONE
• Sync up package listings, import builds, tag builds - Jesse In progress
• Work on a koji owners to owners.list translation tool - Bill DONE
• Sync Core CVS - Jeremy/Bill DONE
• Import/tag Extras builds - Dennis/Jesse DONE
• Make koji the default build target in Makefile.common for devel/ - Dennis DONE
• Create new mirror directories and setup symlinks to content - Jesse Rescheduled
• Work on getting a rawhide to compose - In progress
• Publish an FAQ regarding the merger - Warren In progress
• Open CVS/Buildsystem for building again - Jesse / Dennis DONE
• Work on a 'needsign' repo replacement for up to the moment builds access - DONE
• Work on a package signing method - Jesse / ?

Locking CVS / Buildsystem

Devel branchs in Core and Extras will be locked down to prevent any further checkins. Core build system will not allow any builds to be tagged for dist-fc6-updates(-candidate) as well as dist-fc7. Extras will ?

Rsyncing core package sets

The i386, x86_64, ppc, ppc64, and srpm builds of core packages will be rsynced over to /mnt/koji/packages/ so that they can be imported in place by koji. At this time we don't plan on rsyncing over the build logs.

Sync up package listings, import builds, tag builds

Once the rpm files are in place under /mnt/koji/packages, the script will make sure that the package is added to the right collection in koji, adding it if it isn't there. It will import in place the build which will make it available in koji for tagging, and it will tag the build for the right collection. This will make the build available to be used in buildroots and for composes.

Sync koji owners to owners.list

Current owners of Core packages will have to be added to owners.list so that proper ACLs can be generated for the CVS roots, as well as populating bugzilla with package information. This should be a one time sync as when packages are added in the future, they will be added to owners.list at the same time they are added to koji. Ownership changes will change both as well.

Sync core and extras CVS

See Sync Core CVS

Import/tag Extras builds

Much in the way that Core builds need to be imported/tagged, Extras packages need to as well. They will be tagged into the same collection(s) that Core packages are tagged for.

Makefile.common changes

Instead of plague being used when 'make build' is called, koji will be used instead.

New Mirror Layout

Currently the mirror is laid out as such:

pub/
<code>-- fedora
<code>-- linux
|-- core
|   |-- 5
|   |-- 6
|   |-- development
|   |-- test
|   |   |-- 6.90
|   |   |-- 6.91
|   |   <code>-- 6.92
|   <code>-- updates
|       |-- 5
|       |-- 6
|       <code>-- testing
|           |-- 5
|           <code>-- 6
<code>-- extras
|-- 5
|-- 6
<code>-- development

With the merger of core and extras, things will change a bit. I'm proposing a new layout:

pub/
<code>-- fedora
<code>-- linux
|-- development
|-- releases
|   |-- 6
|   |-- 7
|   <code>-- test
|       |-- 6.90
|       |-- 6.91
|       <code>-- 6.92
<code>-- updates
|-- 5
|-- 6
|-- 7
<code>-- testing
|-- 5
|-- 6
<code>-- 7

This seperates things out nicely and allows mirrors to sync specific trees at appropriate times. To aid mirrors in the transition, I propose a series of symlinks be created during transition that points to content in old locations, and leaving old data around for a period of time, reverse symlinks after a spell, and finally remove old content that exists in new directories. Old Extras content may be moved into an archive directory to get it out of the name space, but that decision will be made later. The inital layout would look like:

pub/
<code>-- fedora
<code>-- linux
|-- core
|   |-- 5
|   |-- 6
|   |-- development -> ../../development
|   |-- test
|   |   |-- 6.90
|   |   |-- 6.91
|   |   <code>-- 6.92
|   <code>-- updates
|       |-- 5
|       |-- 6
|       <code>-- testing
|           |-- 5
|           <code>-- 6
|-- development
|-- extras
|   |-- 5
|   |-- 6
|   <code>-- development -> ../../development
|-- releases
|   |-- 6 -> ../core/6
|   |-- 7
|   <code>-- test
|       |-- 6.90 -> ../../core/test/6.90
|       |-- 6.91 -> ../../core/test/6.91
|       <code>-- 6.92 -> ../../core/test/6.92
<code>-- updates
|-- 5 -> ../core/updates/5
|-- 6 -> ../core/updates/6
|-- 7
<code>-- testing
|-- 5 -> ../../core/updates/testing/5
|-- 6 -> ../../core/updates/testing/6
<code>-- 7

As you can see, that's a little busy and hard to follow. I propose a README file get dropped into the linux/ directory that explains what is going on.

When will it happen?

Merger is in progress. New build hosts are on order/in route, and storage issues are being taken care of. In the interest of getting things done, we're going ahead with the merge and will pick up the new resources as they come available.

Who is making it happen?

The following people are playing an active role in making the merger happen:

• JesseKeating
• DennisGilmore
• JeremyKatz
• MikeBonnet
• BillNottingham
• WarrenTogami
• MikeMcGrath

Many many other people are helping out in various ways as well.