FinalizingFedoraSwitchtoPython3

Mostly done
Things described on this page are mostly done. The remaining Python 2 exceptions are tracked in Python 2 Dropping Database. This page is no longer updated regularly. Talk to the Python SIG if you want to discuss it.

Summary

Currently, in Fedora package names, executables etc., python without a version number generally means Python 2. Given that the upstream support for Python 2 will end soon, we want to prepare Fedora for a transition to Python 3 as a default. Before switching python to refer to Python 3, a lot of preparatory work needs to be done and this page explains our plan to achieve this.

Wait, wasn't Python 3 as Default already done in Fedora 23? Yes it was, but part of that change proposal was that /usr/bin/python and unversioned package names starting with python will still mean Python 2, in compliance with upstream PEP 394. This change is about what happens once PEP 394 is updated (and we will drive that update if needed):

• /usr/bin/python will mean Python 3
• dnf install python will mean Python 3
• dnf install python-foo will mean Python 3 version of foo
• pip, virtualenv, flask, pytest, flake8, sphinx-build and others like so will all run the Python 3 version, the Python 2 version will need to be run using the -2 (or 2) suffix

Owner

• Name: Iryna Shcherbina
• Name: Miro Hrončok
• Name: Petr Viktorin
• Email: python-maint at redhat.com

Current status

• Last updated: 2020-05-06

Currently (in Fedora 27):

• /usr/bin/python means Python 2
• dnf install python means Python 2
• dnf install python-foo means Python 2 version of foo
• pip, virtualenv, flask, pycodestyle, flake8, sphinx-build and others like so (almost) all run the Python 2 version, the Python 3 version needs to be run using the -3 (or 3) suffix

/usr/bin/python

/usr/bin/python is a symbolic link to /usr/bin/python2 and belongs to the python2 package. This also means that packages that require /usr/bin/python (usually due to shebangs) de facto require Python 2.

The python package

Since Fedora 26, there is no python package, only python3 and python2. python is virtually provided from python2.

Packages with Python modules

According to the guidelines, built (binary) RPM packages with python in their names MUST use the python2- or python3- prefix. python- prefix (or -python suffix) is forbidden. The name with the python- prefix shall be provided from the python2- package using the %python_provide macro. This macro shall be used in both Python 2 and 3 subpackages, while it is currently a no-op on Python 3, but allows an easy switch in the future.

However, it wasn't always so and plenty of packages do not follow those guidelines yet.

The current situation in Fedora is as follows ("correct" means "follows current packaging guidelines"):

• Correct package supporting Python 2 and 3 builds these binary RPMs:
  • python2-foo (provides python-foo as a virtual provide via %python_provides)
  • python3-foo (uses %python_provides, which is currently no-op)

• "Misnamed" package supporting py2/3 builds these binary RPMs:
  • python-foo (the py2 version; may provide python2-foo virtually, but usually doesn't)
  • python3-foo

• Correct py2-only packages are named:
  • python2-foo (provides python-foo as a virtual provide via %python_provides)

• Correct py3-only packages are named:
  • python3-foo (uses %python_provides, which is currently no-op)

• "Misnamed" py2-only package might be named:
  • python-foo, or
  • pyfoo, or
  • foo-python, etc.

• There are (close to) no "misnamed" py3-only packages

We are tracking statistics about this in the Fedora Python 3 Porting Database. There is close to a thousand packages that violate the naming guidelines by the time of Fedora 26.

Detailed Description

While the default version of Python in Fedora is Python 3, the python command still invokes a Python 2 interpreter, and whenever one installs a python-foo package, they get a Python 2 version of this package. Similarly, when a Python developer wants to run a familiar tool, such as virtualenv, twine, flake8, pytest and others, the Python 2 version is run. This has all been decided in long discussions around the Python 3 as Default Change, where the main argument was PEP 394 that says: for the time being, all distributions should ensure that python refers to [...] python2.

However, Python 2 is slowly approaching its end of existence and will no longer be supported upstream after 2020-01-01. Therefore we would like to start the transition to a state where python means python3. PEP 394 says: there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2. We want to be ready for this update, and furthermore we want to make this update happen as soon as possible, but not later than 2020-01-01.

This requires a lot of preparatory work before we do the actual switch. We need to make sure that nothing uses python to imply Python 2 neither in binary RPM names nor in dependencies' names or shebangs, but rather explicitly specifies a Python version by using either python2 or python3. The state we want to achieve in Fedora to be ready for the switch, is where python-foo package names are only virtually provided with the %python_provide macro, so we can switch the macro to mean no-op on Python 2 and create the virtual provide on Python 3 packages.

Other Linux Distributions and OSes

Some Linux distributions like Arch Linux (2010) or OpenMandriva (2017) have already made /usr/bin/python pointing to Python 3, while others like openSUSE or Debian/Ubuntu are working towards handling the transition in a more conservative way. Gentoo uses a configurable option with Python 3 as default.

Open source package manager homebrew for macOS X has renamed the python3 package to just python on 2018-03-01.

Transition Steps

Phase 1: Eliminate the python- prefix from the Python 2 package names and /usr/bin/python from shebangs

• Targeted release for completion: Fedora 28 ~= first half of 2018
• Fedora Change: Not needed
• Description: Ensure that all python-foo names are virtual provides declarations, with the actual RPMs themselves being named python2-foo. python-foo names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the /usr/bin/python shebang and no packages require /usr/bin/python. All scripts shall explicitly use /usr/bin/python2 or /usr/bin/python3 and all packages shall explicitly require those as well. (This also applies to the /usr/bin/env python shebang.) This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with Phase 2.
• Status: Mostly done, stray packages exist due to FTBFSes, and maintainers reverting our changes.

• Steps:
  • Identify the packages with naming and shebangs issues. Those include packages with misnamed binary RPMs and dependencies.
  • Inform the package maintainers about the issues. Gather feedback and encourage fixing the packages.
  • Rename Python 2 binary RPMs from python-foo to python2-foo. This is already partially done by applying automated patches as announced in Mass package change (python2- binary package renaming). The packages which still need to be fixed are tracked in PortingDB (at this point, most of these fail to build due to unrelated problems).
  • Fix ambiguous Python 2 dependency declarations, i.e using python2-foo instead of python-foo in requirements. We have started opening automatic Pull Requests with patches as announced in Mass Pull Request filing for ambiguous Python 2 requirements. The packages which will be targeted are tracked in PortingDB.
  • Ensure that /usr/bin/python or /usr/bin/env python shebangs are replaced with either /usr/bin/python2 or /usr/bin/python3. This is currently covered by a Taskotron check. Automation is in place as well.
  • See also: Update comps to use python3 ( Fedora 29)
  • See also: Make ambiguous python shebangs error ( Fedora 30)

Phase 2: Mass Python 2 Package Removal

• Targeted release: Fedora 30, continue in Fedora 31
• Fedora Change: Mass Python 2 Package Removal and F31 Mass Python 2 Package Removal
• Description: (Sub-)packages only providing python2 importable modules without additional functionality will be removed from Fedora unless some other package(s) depends on them. Python 2 will be deprecated in Fedora. Packagers can mark any other Python 2 packages as deprecated as well. See details in the change. This is done so we can focus on the remaining bits for Phase 4.

Interlude: Update PEP 394 (not a Fedora Change)

• Targeted date: Before Fedora 31 beta freeze
• Formal proposal: https://github.com/python/peps/pull/989
• Description: Work with CPython upstream to change the PEP 394 so that it says: distributions [are free to choose if] python refers to [...] python3.
• Note: This was already attempted in July 2017 but failed. However changes were made in the PEP that said: "In controlled environments aimed at expert users, where being explicit is valued over user experience (for example, in test environments and package build systems), distributions may choose to not provide the python command even if python2 is available. (All software in such a controlled environment must use python3 or python2 rather than python, which means scripts that deliberately use python need to be modified for such environments.)"
• The updates from July 2017 led to Move /usr/bin/python into a separate package Fedora 29 change.

Phase 3: Switch python to refer to python3

• Targeted release: Fedora 31
• Fedora Change: Changes/Python_means_Python3 (proposed)
• Description: Make python mean Python 3 in Fedora. This would mean that installing a python-foo package will imply a Python 3 version on the package, and invoking python or e.g. pytest will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in PEP 394. Note that after Phase 1, nothing in Fedora uses /usr/bin/python any more, so it should be safe for sysadmins to flip the symblink back to python2 and we should provide a documented and supported way of doing so.

• Steps:
  • Switch %python_provides macro to provide python-foo from python3-foo instead of python2-foo
  • Switch /usr/bin/python to point to /usr/bin/python3
  • Switch /usr/bin/pip, pytest, nose... to point tot the Python 3 version

Phase 4: Retire Python 2 for other packages, but keep it around for developers and users

• Targeted release: Fedora 32
• Fedora Change: Retire Python 2
• Description: The python2 package and all its subpackages will be removed from Fedora 32. A legacy python27 package for developers and users will be provided. All packages in Fedora that need Python 2 to run will be removed from Fedora 32 regardless of their dependencies. All packages in Fedora that need Python 2 to build will be removed from Fedora 32 regardless of their dependencies. Exceptions can be granted by FESCo.

Benefit to Fedora

Fedora has long been a one of the leaders of the Make Python 3 the Default Python initiative, having Python 3 as the "default Python" since Fedora 23 including a Fedora Workstation without Python 2, tracking the status of Python 3 compatibility and actively getting involved in porting upstream projects, such as (but not limited to) Samba. However, there are other distros (such as Arch) where python means Python 3 for quite some time.

This transition plan simplifies the final transition to Python 3 when the upstream support for Python 2 ends.

How can I help?

Go trough the packages you maintain and see if they FAIL some taskotron checks. If so, please fix your packages to follow the new rules. You can also see a list of packages breaking the naming policy on PortingDB.

Scope

• Proposal owners:

Proposal owners are to communicate the idea to the packagers and rename the affected packages as well as fix the requirements via Mass_package_changes procedure.

• Other developers:

Package maintainers are expected to fix package requirements generated from shebangs by making sure that no scripts in their packages use /usr/bin/python or /usr/bin/env python shebangs. Package maintainers will also be asked to review Pagure PRs with automated patches fixing the ambiguous Python 2 dependency declarations.

• Policies and guidelines:
  • Using python2/python3 instead of python in binary RPM package name guidelines, fpc
  • Using python2/python3 instead of python in (Build)Requires, guidelines, fpc
  • Using /usr/bin/python2 instead of /usr/bin/python in shebangs and Requires, guidelines, fpc

How To Test

We have implemented taskotron checks for Phase 1 changes. The checks are being run each time a Python package is build in Koji, and you may view the results in taskotron resultsdb or when you submit the update in Bodhi.

Dependencies

Each phase depends on the phase before it.

Documentation

• Discussion on Fedora devel mailing list
• PEP 394
• Mass package change (python2- binary package renaming), part 1
• MASS CHANGE announcement: python2- prefix renaming, part 2
• Mass Pull Request filing for ambiguous Python 2 requirements.

Q&A

• I don't care which Python version my package runs on. What do I do? I just want to say python and let the build system to decide what's that.

Unfortunately that is not possible. According to the Python packaging guidelines, if the software is Python 3 compatible, it must be packaged for Python 3. Thus, you should update your package to Python 3 immediately. If you need more instructions, a guide for porting Python-based RPMs is available.

• I want to use the same specfile for both Fedora and EPEL. By forcing me to use python2- prefix on Fedora, you are making my life harder, because I cannot use it in EPEL. What do I do?

This can be done with the following code:

%if 0%{?rhel}
%global py2_prefix python
%else
%global py2_prefix python2
%endif

...

BuildRequires: %{py2_prefix}-requests

If this would make your specfile unbearably ugly, please contact us at the python-devel mailing list. We can add a %py2_prefix macro to the build root for you, but we don't think it's necessary given common specfiles usually already have such if-else blocks. You can even do:

BuildRequires:  python%{?fedora:2}-requests