From Fedora Project Wiki
m (Change RPMBUILD to RPM_BUILD_ROOT, the first doesn't exist)
(obsoleted)
 
(34 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Avoid /usr/bin/python in RPM build}}
{{DISPLAYTITLE:Avoid /usr/bin/python in RPM build}}


{{admon/important|Almost-a-Change|This page not a [[Changes/Policy#\|Fedora Change]] only because it does not affect end users, so it doesn't need release notes and associated communication.}}
== Summary ==


== Summary ==
{{admon/important|Obsoleted|This change was obsoleted by [[Changes/Move_usr_bin_python_into_separate_package|Move /usr/bin/python into a separate package]] }}


Deprecate, and later disable, running <code>/usr/bin/python</code> (as opposed to <code>/usr/bin/python3</code> or <code>/usr/bin/python2</code>) during RPM build.
Deprecate, and later disable, running <code>/usr/bin/python</code> (as opposed to <code>/usr/bin/python3</code> or <code>/usr/bin/python2</code>) during RPM build.
Line 13: Line 13:
* Name: [[User:pviktori| Petr Viktorin]]
* Name: [[User:pviktori| Petr Viktorin]]
* Name: [[User:churchyard| Miro Hrončok]]
* Name: [[User:churchyard| Miro Hrončok]]
* Email: python-devel at lists.fedoraproject.org
* Email: python-devel at lists.fedoraproject.org, pviktori at fp.o, churchyard at fp.o
 
== Current status ==
* Targeted release: [[Releases/28 | Fedora 28 ]]
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
Bugzilla states meaning as usual:
NEW -> change proposal is submitted and announced
ASSIGNED -> accepted by FESCo with on going development
MODIFIED -> change is substantially done and testable
ON_QA -> change is code completed and could be tested in the Beta release (optionally by QA)
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1537244 #1537244]


== Detailed Description ==
== Detailed Description ==
Line 32: Line 45:
=== Specification ===
=== Specification ===


Python 2, when called as <code>python</code> or <code>/usr/bin/python</code> at RPM build time (as identified by the <code>RPM_BUILD_ROOT</code> environment variable), will print a deprecation warning to <em>stdout</em>.
Python 2, when called as <code>python</code> or <code>/usr/bin/python</code> will check an environment variable <code>PYTHON_DISALLOW_AMBIGUOUS_VERSION</code>. According to it's value, it will:
 
* do nothing special (value <code>0</code>, empty or unset)
* print a deprecation warning to <em>stderr</em> (value <code>warn</code>)
* print error and exit with positive exit code (value <code>1</code>)
 
(Any program that invokes <code>/usr/bin/python</code> transitively will cause the above to happen as well.)
 
We'll set <code>PYTHON_DISALLOW_AMBIGUOUS_VERSION=warn</code> for Fedora's build environment.


A new Taskotron check will be implemented to look for this warning and fail if it's found.
A new Taskotron check will be implemented to look for the warning and fail if it's found.
We will look at the Taskotron results and work with packagers to switch to update the affected packages. (We'll look at the results to determine if we'll use automated pull requests, mass bug filing, or something else.)
We will look at the Taskotron results and work with packagers to switch to update the affected packages. (We'll look at the results to determine if we'll use automated pull requests, mass bug filing, or something else.)


The warning itself may cause some packages to fail to build, for example if a test relies on exact <em>stderr</em> contents.
The warning itself may cause some packages to fail to build, for example if a test relies on exact <em>stderr</em> contents.
For these cases, it will be possible to turn the warning off using an environment variable, but we ask packagers that use of this workaround is tracked in Bugzilla. See [[#Quick Opt-Out]] below.
For these cases, it will be possible to turn the warning off by changing the environment variable, but we ask packagers that use of this workaround is tracked in Bugzilla. See [[#Quick Opt-Out]] below.


After all packages that BuildRequire python2 are re-built with this check passing, <code>python</code> will be switched to fail after printing the message.
After all packages that BuildRequire python2 are re-built with this check passing, <code>python</code> will be switched to fail after printing the message, i.e. we'll set <code>PYTHON_DISALLOW_AMBIGUOUS_VERSION=1</code> .


The warning will not be effective if stderr output is hidden. So, after switching <code>/usr/bin/python</code> to fail, some packages may start failing to build. We will work with the packagers to fix these. (We'll look at results from the "warnings phase" to see how proactive we'll need to be here.)
The warning will not be effective if stderr output is hidden. So, after switching <code>/usr/bin/python</code> to fail, some packages may start failing to build. We will work with the packagers to fix these. (We'll look at results from the "warnings phase" to see how proactive we'll need to be here.)
Line 50: Line 71:
     Use /usr/bin/python3 or /usr/bin/python2
     Use /usr/bin/python3 or /usr/bin/python2
     /usr/bin/python will be removed or switched to Python 3 in the future.
     /usr/bin/python will be removed or switched to Python 3 in the future.
     If you cannot make the switch now, please follow instructions at https://fedoraproject.org/wiki/XXX#Quick_Opt-Out
     If you cannot make the switch now, please follow instructions at https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out
</pre>
</pre>


Line 57: Line 78:
=== Quick Opt-Out ===
=== Quick Opt-Out ===


In case switching to <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> is non-trivial, you can do two things:
In case switching to <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> is non-trivial, do these two things:  


* Set the environment variable <code>DISABLE_AMBIGUOUS_PYTHON_VERSION_WARNING=1</code> when calling <code>/usr/bin/python</code>
* Set the environment variable <code>PYTHON_DISALLOW_AMBIGUOUS_VERSION=0</code> when calling <code>/usr/bin/python</code>.
* File a Bugzilla, and make it block our tracking bug (XXX - link)
** The variable is reset at the beginning of every RPM section (<code>%prep, %build, %install, %check</code>, etc.). Change it, therefore, in every relevant section—ideally right before the invocation of <code>/usr/bin/python</code>.
* File a Bugzilla, and make it block our tracking bug [https://bugzilla.redhat.com/show_bug.cgi?id=1537244 #1537244]


All such bugs will need to be fixed before we make <code>/usr/bin/python</code> fail hard.
All such bugs will need to be fixed before we make <code>/usr/bin/python</code> fail hard.


==== Python bytecompilation  ====
In case the deprecation warning somehow breaks the automatic bytecompilation outside of <code>/usr/lib(64)?/pythonX.Y</code>, use the following to switch to  <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> (where X is either 2 or 3):
%global __python %{__pythonX}


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 73: Line 100:


== Scope ==
== Scope ==
* Proposal owners: Patch <code>python2</code>, write the Taskotron check, deal with warnings and failures.
* Proposal owners: Patch <code>python2</code><sup>[https://src.fedoraproject.org/rpms/python2/c/7143d8af3a3148044422d3a96fc52877e4fa6c21?branch=master commit]</sup>, set a value for <code>PYTHON_DISALLOW_AMBIGUOUS_VERSION</code><sup>[https://src.fedoraproject.org/rpms/rpm/c/0670d684d58c7ceee6655cc1cd50974e6eb1ae7d?branch=master commit]</sup>, write the Taskotron check<sup>[https://github.com/fedora-python/taskotron-python-versions/pull/40 pull request]</sup>, deal with warnings and failures.
* Other developers: Switch to using <code>/usr/bin/python3</code> or <code>/usr/bin/python2</code> explicitly at RPM build time (with help from Proposal owners if needed).
* Other developers: Switch to using <code>/usr/bin/python3</code> or <code>/usr/bin/python2</code> explicitly at RPM build time (with help from Proposal owners if needed). Also tools that are invoked during build time of other packages that themselves use  <code>/usr/bin/python</code> will need to be fixed.
* Release engineering: N/A <!-- [https://pagure.io/releng/issues #Releng issue number] (XXX: a check of an impact with Release Engineering is needed) --> (Note: Depending on the timing, separate targeted rebuilds may be needed, but we can do these as Proven Packagers) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
* Release engineering: [https://pagure.io/releng/issue/7257 #7257] <!-- (XXX: a check of an impact with Release Engineering is needed) --> (Note: Depending on the timing, separate targeted rebuilds may be needed, but we can do these as Proven Packagers, no side-tags are required) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A
* Policies and guidelines: N/A (XXX: really?)
* Policies and guidelines: Already existing "packages in Fedora ... MUST call the proper executable for the needed python major version directly, either <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> as appropriate" from [[Packaging:Python#Multiple_Python_Runtimes]]
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
* Trademark approval: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
Line 86: Line 113:


== How To Test ==
== How To Test ==
* Build a package that invokes <code>/usr/bin/python</code>, <code>python</code> or <code>%{__python}</code> in <code>%prep</code>, <code>%build</code>, <code>%install</code> or <code>%check</code>.
* Build a package that invokes <code>/usr/bin/python</code>, <code>python</code> or <code>%{__python}</code> in any section of the specfile that is executed during the build.
* Observe a warning in the build logs
* Observe a warning in the build logs
* If already enabled system-wide: observe that the build fails
* If already enabled system-wide: observe that the build fails
Line 99: Line 126:


== Contingency Plan ==
== Contingency Plan ==
* Contingency mechanism: If something goes unexpectedly wrong, and the workaround above doesn't work, the patch to <code>python2</code> will be reverted. Any packages already changed due to the warning will remain changed.
* Contingency mechanism: If something goes unexpectedly wrong, and the workaround above doesn't work, the patch to <code>python2</code> will be reverted. Any packages already changed due to the warning will remain changed. If everything goes smoothly, but there is still a significant number of packages failing the check, the switch that would render all of them FTBFS will happen in the next release.
* Contingency deadline: Beta Freeze (or any time this causes a FTBFS that can't be solved promptly)
* Contingency deadline: Beta Freeze (or any time this causes a FTBFS that can't be solved promptly)
* Blocks release? No
* Blocks release? No
Line 111: Line 138:




<!-- In case this needs to be a Change after all: -->
[[Category:ChangePageIncomplete]]
<!-- [[Category:ChangePageIncomplete]] -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 119: Line 145:


<!-- Select proper category, default is Self Contained Change -->
<!-- Select proper category, default is Self Contained Change -->
<!-- [[Category:SelfContainedChange]] -->
[[Category:SelfContainedChange]]
<!-- [[Category:SystemWideChange]] -->
<!-- [[Category:SystemWideChange]] -->

Latest revision as of 15:01, 8 June 2018


Summary

Important.png
Obsoleted
This change was obsoleted by Move /usr/bin/python into a separate package

Deprecate, and later disable, running /usr/bin/python (as opposed to /usr/bin/python3 or /usr/bin/python2) during RPM build.

Changes will be driven by Python SIG, but a few packages may fail to build (with the failure message providing an easy workaround).

Owner

Current status

Detailed Description

Motivation

Currently in Fedora (package names, executable names, etc.), python means Python 2. We would like to change it to mean Python 3, but to do that, we need to free it of the current meaning. This means explicitly using either "python2" or "python3" throughout Fedora. This is a multi-release effort tracked in Python SIG's "Finalizing Fedora Switch to Python3" document. This page describes a very focused subset of it.

Renaming packages (and associated changes to, for example, Requires: directives) is relatively straightforward, but making all Fedora-provided code avoid /usr/bin/python (as opposed to /usr/bin/python2) is both harder to achieve and harder to keep track of.

We would like to start deprecating /usr/bin/python (as opposed to /usr/bin/python2) at RPM build time. RPM build is a controlled environment: changes to it will not be felt by end users, breakages are almost immediately visible, and output of Koji builds can be nicely tracked and analyzed.

Specification

Python 2, when called as python or /usr/bin/python will check an environment variable PYTHON_DISALLOW_AMBIGUOUS_VERSION. According to it's value, it will:

  • do nothing special (value 0, empty or unset)
  • print a deprecation warning to stderr (value warn)
  • print error and exit with positive exit code (value 1)

(Any program that invokes /usr/bin/python transitively will cause the above to happen as well.)

We'll set PYTHON_DISALLOW_AMBIGUOUS_VERSION=warn for Fedora's build environment.

A new Taskotron check will be implemented to look for the warning and fail if it's found. We will look at the Taskotron results and work with packagers to switch to update the affected packages. (We'll look at the results to determine if we'll use automated pull requests, mass bug filing, or something else.)

The warning itself may cause some packages to fail to build, for example if a test relies on exact stderr contents. For these cases, it will be possible to turn the warning off by changing the environment variable, but we ask packagers that use of this workaround is tracked in Bugzilla. See #Quick Opt-Out below.

After all packages that BuildRequire python2 are re-built with this check passing, python will be switched to fail after printing the message, i.e. we'll set PYTHON_DISALLOW_AMBIGUOUS_VERSION=1 .

The warning will not be effective if stderr output is hidden. So, after switching /usr/bin/python to fail, some packages may start failing to build. We will work with the packagers to fix these. (We'll look at results from the "warnings phase" to see how proactive we'll need to be here.)

The warning text will be:

DEPRECATION WARNING: python2 invoked with /usr/bin/python.
    Use /usr/bin/python3 or /usr/bin/python2
    /usr/bin/python will be removed or switched to Python 3 in the future.
    If you cannot make the switch now, please follow instructions at https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

(Using the Wiki link allows us to easily revise the instructions.)

Quick Opt-Out

In case switching to /usr/bin/python2 or /usr/bin/python3 is non-trivial, do these two things:

  • Set the environment variable PYTHON_DISALLOW_AMBIGUOUS_VERSION=0 when calling /usr/bin/python.
    • The variable is reset at the beginning of every RPM section (%prep, %build, %install, %check, etc.). Change it, therefore, in every relevant section—ideally right before the invocation of /usr/bin/python.
  • File a Bugzilla, and make it block our tracking bug #1537244

All such bugs will need to be fixed before we make /usr/bin/python fail hard.

Python bytecompilation

In case the deprecation warning somehow breaks the automatic bytecompilation outside of /usr/lib(64)?/pythonX.Y, use the following to switch to /usr/bin/python2 or /usr/bin/python3 (where X is either 2 or 3):

%global __python %{__pythonX}

Benefit to Fedora

This change brings us one step closer to a seamless transition to Python 3.

Also, we hope to gain experience with changing packages to avoid /usr/bin/python, to prepare us for making user-visible changes later.


Scope

  • Proposal owners: Patch python2commit, set a value for PYTHON_DISALLOW_AMBIGUOUS_VERSIONcommit, write the Taskotron checkpull request, deal with warnings and failures.
  • Other developers: Switch to using /usr/bin/python3 or /usr/bin/python2 explicitly at RPM build time (with help from Proposal owners if needed). Also tools that are invoked during build time of other packages that themselves use /usr/bin/python will need to be fixed.
  • Release engineering: #7257 (Note: Depending on the timing, separate targeted rebuilds may be needed, but we can do these as Proven Packagers, no side-tags are required)
  • Policies and guidelines: Already existing "packages in Fedora ... MUST call the proper executable for the needed python major version directly, either /usr/bin/python2 or /usr/bin/python3 as appropriate" from Packaging:Python#Multiple_Python_Runtimes
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

N/A (the effect is at RPM build time only)


How To Test

  • Build a package that invokes /usr/bin/python, python or %{__python} in any section of the specfile that is executed during the build.
  • Observe a warning in the build logs
  • If already enabled system-wide: observe that the build fails
  • Look at XXX to see a failed Taskotron check

User Experience

N/A – does not affect end users

Dependencies

May affect packages that use Python 2 to build (including in tests). Any breakage should be minimal, with an easy workaround if things go wrong.

Contingency Plan

  • Contingency mechanism: If something goes unexpectedly wrong, and the workaround above doesn't work, the patch to python2 will be reverted. Any packages already changed due to the warning will remain changed. If everything goes smoothly, but there is still a significant number of packages failing the check, the switch that would render all of them FTBFS will happen in the next release.
  • Contingency deadline: Beta Freeze (or any time this causes a FTBFS that can't be solved promptly)
  • Blocks release? No
  • Blocks product? None

Documentation

This page is the documentation.

Release Notes

N/A (Not an end-user visible change)