From Fedora Project Wiki
(Created page with "== EPEL 7 in Python 3 Plan Draft == === Why Python 3 in EPEL 7, Why not SCLs === Due to Fedora plans for migration to Python 3 [1], it is becoming necessary to have Python 3...")
 
(Add a proposal for handling specfiles)
Line 24: Line 24:
 
** python3X+1 package will be created for EPEL ASAP and all extension packages should be built also for this new python3X+1 stack. (While the python3X stack would stay untouched the whole time.)
 
** python3X+1 package will be created for EPEL ASAP and all extension packages should be built also for this new python3X+1 stack. (While the python3X stack would stay untouched the whole time.)
 
** When all packages are rebuilt for python3X+1, the old python3X stack will be retired after some amount of time (TODO: how long?, TODO: would this align with the EPIC proposal somehow?).
 
** When all packages are rebuilt for python3X+1, the old python3X stack will be retired after some amount of time (TODO: how long?, TODO: would this align with the EPIC proposal somehow?).
** "/usr/bin/python3" will belong to the "stable" python3X stack, hence it should be only changed to point to the new stack once the old is obsoleted. Usage of "/usr/bin/python3" should be discouraged in favour of using /usr/bin/python3.X explicitly (TODO: or should we not provide "/usr/bin/python3" at all?)
+
** "/usr/bin/python3" will belong to the "stable" python3X stack, hence it should be only changed to point to the new stack once the old is obsoleted. Usage of "/usr/bin/python3" should be discouraged in favour of using /usr/bin/python3.X explicitly (TODO: at which point should "/usr/bin/python3" be switched from python3X to python3X+1?)
  
 
==== Packaging Parallel python3X stacks ====
 
==== Packaging Parallel python3X stacks ====
Line 37: Line 37:
 
** Or we could do the above, but create special epel7-python3-foo dist-git repos to keep things more organized and separated from standard Python repos (I'd prefer this).
 
** Or we could do the above, but create special epel7-python3-foo dist-git repos to keep things more organized and separated from standard Python repos (I'd prefer this).
  
 +
==== What Will a Specfile Look Like? ====
 +
 +
I experimented with several possibilities and this one seems to be the best:
 +
 +
<pre>
 +
%global with_python3 1
 +
%if 0%{?rhel} > 6
 +
%global python3_pkgversion_nodots 34
 +
%global with_python3_other 1
 +
%global python3_other_pkgversion_nodots 35
 +
%else
 +
%global python3_pkgversion_nodots 3
 +
%endif
 +
 +
Name:          python-X
 +
Version:        1
 +
Release:        1%{?dist}
 +
Summary:        X
 +
Source:        X
 +
 +
License:        MIT
 +
URL:            http://X
 +
%if 0%{?with_python3}
 +
# BR: python%{python3_pkgversion_nodots}-devel
 +
# and other BR
 +
%endif
 +
 +
%if 0%{?with_python3_other}
 +
# BR: python%{python3_other_pkgversion_nodots}-devel
 +
# and other BR
 +
%endif
 +
 +
%description
 +
X.
 +
 +
%if 0%{?with_python3}
 +
%package -n python%{python3_pkgversion_nodots}-X
 +
Summary: python%{python3_pkgversion_nodots} build of X
 +
 +
%description -n python%{python3_pkgversion_nodots}-X
 +
python%{python3_pkgversion_nodots} build of X.
 +
%endif
 +
 +
%if 0%{?with_python3_other}
 +
%package -n python%{python3_other_pkgversion_nodots}-X
 +
Summary: python%{python3_other_pkgversion_nodots} build of X
 +
 +
%description -n python%{python3_other_pkgversion_nodots}-X
 +
python%{python3_other_pkgversion_nodots} build of X.
 +
%endif
 +
 +
%prep
 +
%setup -q -n X-%{version}
 +
 +
%if 0%{?with_python3}
 +
# do what you do for Fedora's Python 3 stack
 +
%endif
 +
 +
%if 0%{?with_python3_other}
 +
# do the same as above, just with pytohn3_other macro set
 +
%endif
 +
 +
%build
 +
# do it like in Fedora, add with_python3_other
 +
 +
%install
 +
# do it like in Fedora, add with_python3_other
 +
 +
%if 0%{?with_python3}
 +
%files -n python%{python3_pkgversion_nodots}-X
 +
%endif
 +
 +
%if 0%{?with_python3_other}
 +
%files -n python%{python3_other_pkgversion_nodots}-X
 +
%endif
 +
 +
%changelog
 +
</pre>
 +
 +
===== Explanation =====
 +
 +
For start, packagers will be able to merge specfile from Fedora and use it - all macros like <code>%__python3</code> or <code>%python3_sitelib</code> will be provided by the current python3X-devel. It'll even be possible to use the same specfile for Fedora and EPEL with minimal effort. When merging specfiles from Fedora, packagers will need to:
 +
 +
* do initial Fedora import
 +
* add conditionalized defintions of <code>python3_pkgversion_nodots</code>, <code>with_other_python3</code> and <code>python3_other_pkgversion_nodots</code> (see the top of the specfile)
 +
* change Fedora's <code>python3-X</code> subpacke name to <code>python%{python3_pkgversion_nodots}-X</code>
 +
* add defintion and %prep, %build and %install steps of <code>python%{python3_other_pkgversion_nodots}-X</code> subpackage. which looks exactly the same as <code>python%{python3_pkgversion_nodots}-X</code> subpackage, it just uses different macros
 +
 +
Changing stacks when a new python3X+1 build is introduced or python3X is retired:
 +
* during periods when only a single Python 3 runtime is in EPEL 7, packages will be built with <code>%with_python3</code> enabled and with <code>%with_python3_other</code> disabled
 +
* when a new python3X+1 is built in EPEL - let's say that there is python34 and python35 has just been introduced:
 +
** <code>%with_python3_other</code> is enabled in all packages, <code>%python3_pkgversion_nodots</code> is still set to 34, <code>%python3_other_pkgversion_nodots</code> is still set to 35
 +
** mass rebuild is run to build as much of the new stack possible automatically
 +
** at a certain point at time, annoucement is made that python34 is to be retired and python35 is to be *the* one - at this point, python34 and python35 will be rebuilt; python34-devel will provide the "other" macros from now on, while python35 will provide the "default" macros (note that all the specfiles still build just fine as they are)
 +
** some time after python34 and python35 are rebuilt, <code>%python3_pkgversion_nodots</code> is set to "35", <code>%python3_other_pkgversion_nodots</code> is set to "36" and <code>%with_python3_other</code> is disabled
 +
** no rebuild is needed at this point, but all packages build just python35-X subpackages
 +
** now the whole python34 stack is disabled
 +
 +
All the painful parts in the above process can be done automatically (enabling/disabling "with" macros, changing values of "pkgversion" macros, mass rebuilding).
  
 
[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default
 
[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default
 
[2] https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#A_major_version_update
 
[2] https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#A_major_version_update

Revision as of 15:30, 15 January 2015

EPEL 7 in Python 3 Plan Draft

Why Python 3 in EPEL 7, Why not SCLs

Due to Fedora plans for migration to Python 3 [1], it is becoming necessary to have Python 3 accessible in some form in EPEL 7. This will allow upstream projects to move to Python 3 (dropping Python 2 support) and still be usable through EPEL repositories. Some key stakeholders in this regard are Fedora Infra and DNF.

Another important reason is further promotion of Python 3 and raising awareness for enterprise-level Python users.

Because of key stakeholders like Fedora Infra and DNF, SCLs as a solution to this problem were ruled out. SCLs are currently not allowed in EPEL (and it seems they won't be allowed in near future) and since these projects need/strongly prefer to be in EPEL itself, SCLs are not a viable alternative for them.

Some Facts

  • Python 3 is actively developed by upstream and new minor versions are released approximately every year. Latest Python 3 minor is 3.4.
  • Python minor versions are not ABI compatible, rebuilds are needed (both because of libpython soname change and because of bytecode magic numbers).
  • Python has a good support for parallel installable stacks, even for minor versions of one major version (e.g. 3.3 and 3.4 can easily be installable alongside).
  • EPEL policies try to discourage updates that break ABI and require rebuilds of other packages - due to the above fact, I consider Python minor version update a "major version update" as specified in EPEL update guidelines [2]. And as the guidelines say, this should be avoided if at all possible.

Proposal

  • We will package Python 3 as python3X parallel-installable package (X is minor version, e.g. "4" ATM).
  • All extension packages will be named python3X-foo. They will follow Fedora's python3 packaging guidelines with the exception that name contains both major and minor Python version (in Fedora it's only major).
  • The whole python3X stack will be parallel installable with any other python3Y stack.
  • In a situation when python3X is in EPEL and 3.X+1 is released upstream, the following should happen:
    • python3X+1 package will be created for EPEL ASAP and all extension packages should be built also for this new python3X+1 stack. (While the python3X stack would stay untouched the whole time.)
    • When all packages are rebuilt for python3X+1, the old python3X stack will be retired after some amount of time (TODO: how long?, TODO: would this align with the EPIC proposal somehow?).
    • "/usr/bin/python3" will belong to the "stable" python3X stack, hence it should be only changed to point to the new stack once the old is obsoleted. Usage of "/usr/bin/python3" should be discouraged in favour of using /usr/bin/python3.X explicitly (TODO: at which point should "/usr/bin/python3" be switched from python3X to python3X+1?)

Packaging Parallel python3X stacks

  • Automatic dependency generators and bytecompilation hooks should do fine.
  • SRPM/RPM naming:
    • As mentioned above, binary RPMs will be named python3X for Python itself and python3X-foo for extension packages.
    • Source RPMs should also be named python3X-foo.srpm, although we could create python3-foo.srpm and build python3X-foo.rpm and python3X+1-foo.rpm out of it. Doing the latter is pretty similar to what we're doing in Fedora with parallel python2/python3 stacks and may be easier to maintain. (We could even create just python-foo.srpm, but that might be confusing, since the same SRPMs could exist in RHEL/CentOS 7, which are used for the Python 2 RHEL 7 stack.)
  • Dist-git possibilities:
    • Having python3X-foo dist-git repo for every package. Easy and obvious solution, but packages would have to be re-reviewed and new repos created with every Python 3 minor release. This would align with the first SRPM variant (python3X-foo SRPMS).
    • Using current dist-git repos, e.g. branch epel-7 in python-foo (or python3-foo, if such Fedora dist-git exists). With this approach, we'd create python3-foo.srpm and build python3X and possibly python3X+1 RPMs out of it. The disadvantage is that some packages would be in python-foo and others in python3-foo dist git repos, which could get confusing. Also, their packaging could in time diverge a lot, so it might not make sense to do this.
    • Or we could do the above, but create special epel7-python3-foo dist-git repos to keep things more organized and separated from standard Python repos (I'd prefer this).

What Will a Specfile Look Like?

I experimented with several possibilities and this one seems to be the best:

%global with_python3 1
%if 0%{?rhel} > 6
%global python3_pkgversion_nodots 34
%global with_python3_other 1
%global python3_other_pkgversion_nodots 35
%else
%global python3_pkgversion_nodots 3
%endif

Name:           python-X
Version:        1
Release:        1%{?dist}
Summary:        X
Source:         X

License:        MIT
URL:            http://X
%if 0%{?with_python3}
# BR: python%{python3_pkgversion_nodots}-devel
# and other BR
%endif

%if 0%{?with_python3_other}
# BR: python%{python3_other_pkgversion_nodots}-devel
# and other BR
%endif

%description
X.

%if 0%{?with_python3}
%package -n python%{python3_pkgversion_nodots}-X
Summary: python%{python3_pkgversion_nodots} build of X

%description -n python%{python3_pkgversion_nodots}-X
python%{python3_pkgversion_nodots} build of X.
%endif

%if 0%{?with_python3_other}
%package -n python%{python3_other_pkgversion_nodots}-X
Summary: python%{python3_other_pkgversion_nodots} build of X

%description -n python%{python3_other_pkgversion_nodots}-X
python%{python3_other_pkgversion_nodots} build of X.
%endif

%prep
%setup -q -n X-%{version}

%if 0%{?with_python3}
# do what you do for Fedora's Python 3 stack
%endif

%if 0%{?with_python3_other}
# do the same as above, just with pytohn3_other macro set
%endif

%build
# do it like in Fedora, add with_python3_other

%install
# do it like in Fedora, add with_python3_other

%if 0%{?with_python3}
%files -n python%{python3_pkgversion_nodots}-X
%endif

%if 0%{?with_python3_other}
%files -n python%{python3_other_pkgversion_nodots}-X
%endif

%changelog
Explanation

For start, packagers will be able to merge specfile from Fedora and use it - all macros like %__python3 or %python3_sitelib will be provided by the current python3X-devel. It'll even be possible to use the same specfile for Fedora and EPEL with minimal effort. When merging specfiles from Fedora, packagers will need to:

  • do initial Fedora import
  • add conditionalized defintions of python3_pkgversion_nodots, with_other_python3 and python3_other_pkgversion_nodots (see the top of the specfile)
  • change Fedora's python3-X subpacke name to python%{python3_pkgversion_nodots}-X
  • add defintion and %prep, %build and %install steps of python%{python3_other_pkgversion_nodots}-X subpackage. which looks exactly the same as python%{python3_pkgversion_nodots}-X subpackage, it just uses different macros

Changing stacks when a new python3X+1 build is introduced or python3X is retired:

  • during periods when only a single Python 3 runtime is in EPEL 7, packages will be built with %with_python3 enabled and with %with_python3_other disabled
  • when a new python3X+1 is built in EPEL - let's say that there is python34 and python35 has just been introduced:
    • %with_python3_other is enabled in all packages, %python3_pkgversion_nodots is still set to 34, %python3_other_pkgversion_nodots is still set to 35
    • mass rebuild is run to build as much of the new stack possible automatically
    • at a certain point at time, annoucement is made that python34 is to be retired and python35 is to be *the* one - at this point, python34 and python35 will be rebuilt; python34-devel will provide the "other" macros from now on, while python35 will provide the "default" macros (note that all the specfiles still build just fine as they are)
    • some time after python34 and python35 are rebuilt, %python3_pkgversion_nodots is set to "35", %python3_other_pkgversion_nodots is set to "36" and %with_python3_other is disabled
    • no rebuild is needed at this point, but all packages build just python35-X subpackages
    • now the whole python34 stack is disabled

All the painful parts in the above process can be done automatically (enabling/disabling "with" macros, changing values of "pkgversion" macros, mass rebuilding).

[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default [2] https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#A_major_version_update