From Fedora Project Wiki
(Created page with "This is a proposal for SCL dist-git structure on CentOS, i.e git repositories where future upstream development of Software Collections will take place. It focuses on easy col...")
 
 
(11 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
SCL SIG has been established and approved by CentOS board for driving this effort: http://wiki.centos.org/SpecialInterestGroup/SCLo
 
SCL SIG has been established and approved by CentOS board for driving this effort: http://wiki.centos.org/SpecialInterestGroup/SCLo
  
== Currently synchronized content of RHSCL ==
+
== Synchronized content of RHSCL being done so far==
  
Currently, source RPMs of all released RHSCL packages are synchronized to git.centos.org as well, but these are not subject of this proposal. They are imported into git.centos.org under rpm project and use SCL prefix in their name.
+
Currently, source RPMs of all released RHSCL packages are synchronized to git.centos.org as well, but these are not subject of this proposal. They are imported into git.centos.org under ''rpms'' project and use SCL prefix in their component name.
  
 
== Components vs. git repositories ==
 
== Components vs. git repositories ==
Line 11: Line 11:
 
Similar to Fedora, Red Hat uses one git repository for one component internally. It allows to set permissions with enough granularity.
 
Similar to Fedora, Red Hat uses one git repository for one component internally. It allows to set permissions with enough granularity.
  
Open question: who will be able to commit to the dist-git
+
Only members of SCL SIG will be able to commit to the dist-git, it is a requirement from [[http://wiki.centos.org/HowTos/CommunityBuildSystem#head-00dad77f5720f3a984b0fac9f9bacac52047f73f|CentOS policy]]
* Proposed variant #1: members of SCL SIG
 
* Proposed variant #2 (preferred): users approved by SCL SIG
 
  
Structure of the repository will be the same as it is used in Fedora and Red Hat internally. It means every repository will contain at least RPM specfile and sources file which includes list of links to look-aside cache. Optionally, there may be patches and other sources directly in git repository's root, i.e. No directories are used.
+
Structure of the repository will be the same as it is used in Fedora and Red Hat internally. It means every repository will contain at least RPM ''specfile'' and ''sources'' file which includes list of links to look-aside cache (with checksum). Optionally, there may be patches and other sources directly in git repository's root. No directories are used.
  
 
== Branches ==
 
== Branches ==
  
When working with Software Collections it is common that same package is part of more collections. Typical example is a python module, that is available in python27 and python33 software collection as well. Changes between RPM sources for python27 and python33 are typically small or even none.
+
When working with Software Collections it is common that one package is part of more software collections. A typical example is some python module, that is available in ''python27'' and ''python33'' software collection as well. Changes between RPM sources for ''python27'' and ''python33'' are typically small or even none.
  
On the other hand, even though in some cases the same source is used in all branches, dist-git needs to be flexible and make it easy for maintainers to include different code for different collections. Typical use case will be Ruby on Rails (RoR) collection, where we find different versions of components in various RoR versions. Requiring the same sources for those packages would result in ugly spec files with many %if, which is not desired.
+
On the other hand, even though in some cases the same source is used in all branches, dist-git needs to be flexible and make it easy for maintainers to include different code for different collections. Typical use case will be Ruby on Rails (RoR) collection, where we find different versions of components in various RoR versions. Requiring the same sources for those packages in all branches would result in complicated and ugly spec files with many %ifs, which is not desired.
  
A similar scenario works for sources for different platforms -- CentoOS 6 and CentOS 7. Thus, it make sense to have a branch for specific CentOS version as well.
+
A similar scenario works for sources for different platforms -- CentOS 6 and CentOS 7. Thus, it make sense to have a separate branches for specific CentOS versions as well.
  
To summarize it, we will have one branch for all supported combinations of CentOS version and SCL, where a component is build to, so we will have for example these branches (for packages that are available in python27 and python33 collection):
+
To summarize it, we will have one branch for each supported combination of CentOS version and SCL, where a component is build for. For example, we will have these branches (for packages that are available in python27 and python33 collection):
 
* scl-python27-el6
 
* scl-python27-el6
 
* scl-python27-el7
 
* scl-python27-el7
Line 31: Line 29:
 
* scl-python33-el7
 
* scl-python33-el7
  
Prefix 'scl' is used to easily identify that a branch is related to SCL.
+
Prefix ''scl'' is used to easily identify that a branch is related to Software collections.
  
 
== Repositories naming ==
 
== Repositories naming ==
Line 41: Line 39:
 
* python-meta
 
* python-meta
 
* python-setuptools
 
* python-setuptools
 +
 +
Git repositories for meta packages components will be called differently than it is done internally in Red Hat. Meta packages will use '-meta' suffix after the ''name of the collection without version''. Using this layout it is easier to maintain meta packages for similar collections.
  
 
== Content of repositories ==
 
== Content of repositories ==
  
The most often reason for having SCLs on CentOS is to provide newer versions of some component. Thus, content of SCL dist-git will often correspond with content in Fedora, there will only be SCL-related macros added, which is not allowed in Fedora currently.
+
The most often reason for having SCLs on CentOS is to provide newer versions of some component. Thus, content of SCLs in dist-git will often correspond with content in a stable Fedora. The difference will be in SCL-related macros added, which is not allowed in Fedora currently.
  
 
== Naming of built packages ==
 
== Naming of built packages ==
  
When built in koji, names of resulting RPM packages need to stay the same as packages in RHSCL, so they are compatible with them. This is required by SCL consumers who do not want to differ between SCLs provided by CentOS and Red Hat. From the same reason, SCLs will use '''rh''' as a vendor, which is used in SCL concept.
+
When built in koji, names of resulting RPM packages need to stay the same as packages in RHSCL, so they are compatible between each other. This is required by SCL consumers who do not want to differ between SCLs provided by CentOS and Red Hat.
 +
 
 +
From the same reason, SCLs will use '''rh''' as a vendor, which is used in RHSCL.
  
 
This is an example of packages that will be built in koji for particular branch:
 
This is an example of packages that will be built in koji for particular branch:
Line 55: Line 57:
 
! component          !! sclo-python33-el6          !! sclo-python34-el6
 
! component          !! sclo-python33-el6          !! sclo-python34-el6
 
|-
 
|-
| python-meta        || python33                    || python34
+
| python-meta        || python33                    || rh-python34 *
 
|-
 
|-
| python             || python33-python            || python34-python
+
| python             || python33-python            || rh-python34-python *
 
|-
 
|-
| python-setuptools   || python33-python-setuptools  || python34-python-setuptools
+
| python-setuptools || python33-python-setuptools  || rh-python34-python-setuptools *
 
|-
 
|-
| python-pytest       || python33-python-pytest      || python34-python-pytest
+
| python-pytest     || python33-python-pytest      || rh-python34-python-pytest *
 
|}
 
|}
  
Git repositories for meta packages components will be called differently than it is done internally in Red Hat. Meta packages will use '-meta' suffix after the name of the collection without version. Using this layout it is easier to maintain meta packages for similar collections.
+
{{admon/note|* Note about '''rh-''' prefix in the name|The prefix '''rh-''' is a planned feature for new collections so they won't be mixed with non-SCL packages called the same (e.g. mysql55 packaged usually may be mixed up with meta package for mysql55 collection. Already released SCLs will not use vendor in the packages names in order to stay compatible.}}
  
 
== koji setup ==
 
== koji setup ==
  
koji setup for every branch includes setting up:
+
''koji'' setup for every branch includes setting up:
 
* a target/tag for every branch
 
* a target/tag for every branch
 
* buildroot that includes the following packages:
 
* buildroot that includes the following packages:
Line 74: Line 76:
 
** colname00-build
 
** colname00-build
  
Open question: Who will be able to setup the koji buildroots, tags, targets?
+
Members of SCL SIG will be able to setup the koji: buildroots, tags, targets.
* Proposed variant #1 (preferred):  members of SCL SIG
 
* Proposed variant #2: somebody dedicated for koji setup and SIG will ask for setting up koji
 
  
== CentOS look-aside cache? ==
+
== CentOS look-aside cache ==
  
 
Sources that are not part of git are uploaded into CentOS look-aside cache, similar to what is in Fedora.
 
Sources that are not part of git are uploaded into CentOS look-aside cache, similar to what is in Fedora.
Line 86: Line 86:
 
== Example of work for internal RHSCL maintainers ==
 
== Example of work for internal RHSCL maintainers ==
 
<pre>
 
<pre>
rhpkg clone python
+
# getting sources from centos into working directory of RH dist-git
cd python
+
$ rhpkg clone python
git remote add centos https://git.centos.com/scls/python.git
+
$ cd python
git fetch centos
+
$ git remote add centos https://git.centos.com/scls/python.git
git checkout -b scl-python33-el7 --track centos/scl-python33-el7
+
$ git fetch centos
[change/commit]
+
 
git push centos scl-python33-el7:scl-python33-el7
+
# switch into centos branch
 +
$ git checkout -b scl-python33-el7 --track centos/scl-python33-el7
 +
 
 +
# now it is time to do local commits
 +
$ vim *.spec
 +
$ rhpkg commit
 +
 
 +
# you may also upload files into centos look-aside cache
 +
$ centpkg new-sources new.tar.gz
 +
 
 +
# pushing changes back to centos (do not forget about 'centos' here)
 +
$ git push centos
 +
</pre>
 +
 
 +
Or alternative scenario using '''centpkg''':
 +
<pre>
 +
# do the changes directly in the centos in one directory
 +
$ centpkg clone python
 +
$ cd python
 +
$ vim *.spec
 +
$ centpkg commit
 +
$ centpkg push
 +
 
 +
# then merge the changes in a directory cloned using `rhpkg`
 +
$ rhpkg clone python
 +
$ cd python
 +
$ rhpkg remote fetch-sources centos # if RHBZ#973550 is implemented
 +
$ git cherry-pick 7705c460
 +
$ rhpkg push
 
</pre>
 
</pre>
 +
 +
== FAQ ==
 +
 +
* Q: Where the packages will go (which repository)?
 +
* A: answered bellow
 +
* Q: How user will see the difference between "stable" packages and "development" packages?
 +
* A: Proposal: there will be two repositories. One repository with stable '''downstream''' rebuild of RHSCL, another with '''upstream''' SCLs that include fixes above RHSCL and have quality compared to Fedora.

Latest revision as of 13:24, 11 December 2014

This is a proposal for SCL dist-git structure on CentOS, i.e git repositories where future upstream development of Software Collections will take place. It focuses on easy collaboration of Red Hat engineers, that will be able to merge changes from CentOS into internal dist-git or cherry-pick particular commits as easy as possible.

SCL SIG has been established and approved by CentOS board for driving this effort: http://wiki.centos.org/SpecialInterestGroup/SCLo

Synchronized content of RHSCL being done so far

Currently, source RPMs of all released RHSCL packages are synchronized to git.centos.org as well, but these are not subject of this proposal. They are imported into git.centos.org under rpms project and use SCL prefix in their component name.

Components vs. git repositories

Similar to Fedora, Red Hat uses one git repository for one component internally. It allows to set permissions with enough granularity.

Only members of SCL SIG will be able to commit to the dist-git, it is a requirement from [policy]

Structure of the repository will be the same as it is used in Fedora and Red Hat internally. It means every repository will contain at least RPM specfile and sources file which includes list of links to look-aside cache (with checksum). Optionally, there may be patches and other sources directly in git repository's root. No directories are used.

Branches

When working with Software Collections it is common that one package is part of more software collections. A typical example is some python module, that is available in python27 and python33 software collection as well. Changes between RPM sources for python27 and python33 are typically small or even none.

On the other hand, even though in some cases the same source is used in all branches, dist-git needs to be flexible and make it easy for maintainers to include different code for different collections. Typical use case will be Ruby on Rails (RoR) collection, where we find different versions of components in various RoR versions. Requiring the same sources for those packages in all branches would result in complicated and ugly spec files with many %ifs, which is not desired.

A similar scenario works for sources for different platforms -- CentOS 6 and CentOS 7. Thus, it make sense to have a separate branches for specific CentOS versions as well.

To summarize it, we will have one branch for each supported combination of CentOS version and SCL, where a component is build for. For example, we will have these branches (for packages that are available in python27 and python33 collection):

  • scl-python27-el6
  • scl-python27-el7
  • scl-python33-el6
  • scl-python33-el7

Prefix scl is used to easily identify that a branch is related to Software collections.

Repositories naming

Since we use branches to distinguish source code for particular CentOS version and SCL, no prefix is used in git repository name.

For example, dist-git will include the following git repositories:

  • python
  • python-meta
  • python-setuptools

Git repositories for meta packages components will be called differently than it is done internally in Red Hat. Meta packages will use '-meta' suffix after the name of the collection without version. Using this layout it is easier to maintain meta packages for similar collections.

Content of repositories

The most often reason for having SCLs on CentOS is to provide newer versions of some component. Thus, content of SCLs in dist-git will often correspond with content in a stable Fedora. The difference will be in SCL-related macros added, which is not allowed in Fedora currently.

Naming of built packages

When built in koji, names of resulting RPM packages need to stay the same as packages in RHSCL, so they are compatible between each other. This is required by SCL consumers who do not want to differ between SCLs provided by CentOS and Red Hat.

From the same reason, SCLs will use rh as a vendor, which is used in RHSCL.

This is an example of packages that will be built in koji for particular branch:

component sclo-python33-el6 sclo-python34-el6
python-meta python33 rh-python34 *
python python33-python rh-python34-python *
python-setuptools python33-python-setuptools rh-python34-python-setuptools *
python-pytest python33-python-pytest rh-python34-python-pytest *
Note.png
* Note about rh- prefix in the name
The prefix rh- is a planned feature for new collections so they won't be mixed with non-SCL packages called the same (e.g. mysql55 packaged usually may be mixed up with meta package for mysql55 collection. Already released SCLs will not use vendor in the packages names in order to stay compatible.

koji setup

koji setup for every branch includes setting up:

  • a target/tag for every branch
  • buildroot that includes the following packages:
    • scl-utils-build
    • colname00-build

Members of SCL SIG will be able to setup the koji: buildroots, tags, targets.

CentOS look-aside cache

Sources that are not part of git are uploaded into CentOS look-aside cache, similar to what is in Fedora.

Contributors will be able to upload any sources into CentOS look-aside cache by a tool provided by CentOS (similar to fedpkg in Fedora). Everybody who is able to commit to SCL dist-git is able to upload sources into the look-aside cache.

Example of work for internal RHSCL maintainers

# getting sources from centos into working directory of RH dist-git
$ rhpkg clone python
$ cd python
$ git remote add centos https://git.centos.com/scls/python.git
$ git fetch centos

# switch into centos branch
$ git checkout -b scl-python33-el7 --track centos/scl-python33-el7

# now it is time to do local commits
$ vim *.spec
$ rhpkg commit

# you may also upload files into centos look-aside cache
$ centpkg new-sources new.tar.gz

# pushing changes back to centos (do not forget about 'centos' here)
$ git push centos

Or alternative scenario using centpkg:

# do the changes directly in the centos in one directory
$ centpkg clone python
$ cd python
$ vim *.spec
$ centpkg commit
$ centpkg push

# then merge the changes in a directory cloned using `rhpkg`
$ rhpkg clone python
$ cd python
$ rhpkg remote fetch-sources centos # if RHBZ#973550 is implemented
$ git cherry-pick 7705c460
$ rhpkg push

FAQ

  • Q: Where the packages will go (which repository)?
  • A: answered bellow
  • Q: How user will see the difference between "stable" packages and "development" packages?
  • A: Proposal: there will be two repositories. One repository with stable downstream rebuild of RHSCL, another with upstream SCLs that include fixes above RHSCL and have quality compared to Fedora.