From Fedora Project Wiki
(Move the list of macros to detailed description)
(packaging guidelines PR)
 
(35 intermediate revisions by 4 users not shown)
Line 2: Line 2:


= Python: Add -P to default shebangs <!-- The name of your change proposal --> =
= Python: Add -P to default shebangs <!-- The name of your change proposal --> =
{{Change_Proposal_Banner}}


== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->


The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P` flag] will be added to the Python shebang macros (`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that adhere to those macros will change their Python shbanegs from `#! /usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result, will no longer have the directory of the script (such as `/usr/bin`) in `sys.path`. An opt-out mechanism exists.
The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P` flag] will be added to the Python shebang macros (`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that adhere to those macros will change their Python shebangs from `#! /usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result, will no longer have the directory of the script (such as `/usr/bin`) in `sys.path`. An opt-out mechanism exists.


== Owner ==
== Owner ==
Line 15: Line 13:
This should link to your home wiki page so we know who you are.  
This should link to your home wiki page so we know who you are.  
-->
-->
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]]
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Ksurma|Karolina Surma]], [[User:Vstinner|Victor Stinner]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: python-maint@redhat.com <!-- your email address so we can contact you, invite you to meetings, etc. Please provide your Bugzilla email address if it is different from your email in FAS -->
* Email: python-maint@redhat.com <!-- your email address so we can contact you, invite you to meetings, etc. Please provide your Bugzilla email address if it is different from your email in FAS -->
Line 21: Line 19:
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
-->
-->


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF37]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 42: Line 39:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EHNPGB66VZC3OVYU2WNDACOE6RFL4Q64/ devel thread]
* Tracker bug: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2795 #2795]
* Release notes tracker: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2090866 #2090866]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/842 #842]


== Detailed Description ==
== Detailed Description ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
All Python 3 shebang RPM macros will be changed to contain one more flag: `-P`. Previously, they contained `-s`, now they will contain `-sP`.
From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P documentation for the `-P` option]:
:Don’t prepend a potentially unsafe path to `sys.path`:
:
:* `python -m module` command line: Don’t prepend the current working directory.
:* `python script.py` command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
:* `python -c code` and `python` (REPL) command lines: Don’t prepend an empty string, which means the current working directory.
In shebangs, only the middle option (''don’t prepend the script’s directory'') is relevant.
Consider the following executbale script installed as `/usr/bin/let-there-be-fun`:
#! /usr/bin/python3 -s
import abc
...
When the script is directly executed (e.g. by running `let-there-be-fun` from the console), the script's directory (`/usr/bin`) is prepended to `sys.path`. Python tries to locate an importable `abc` module in `/usr/bin` first. This can cause real issues: [https://bugzilla.redhat.com/2057340 python3-notebook: ImportError: bad magic number in six] and [https://github.com/benjaminp/six/issues/359 bad magic number in six].
When the shebang includes `-P`:
#! /usr/bin/python3 -sP
import abc
...
...the script's directory (`/usr/bin`) is '''not''' prepended to `sys.path`. The change owners consider this approach safer for the majority of Fedora's RPM packages.
By default, '''all standardly RPM-packaged Python packages with scripts in `/usr/bin` will gain the `-P` flag in their shebang''', assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.)


=== List of RPM macros that will gain `-P` ===
=== List of RPM macros that will gain `-P` ===
Line 57: Line 85:
* `%{py_shbang_opts_nodash}`
* `%{py_shbang_opts_nodash}`
* `%{py_shebang_flags}`
* `%{py_shebang_flags}`
=== List of source RPM packages that will gain `-P` in some of the built package(s) ===
Queried on 2022-07-07 in https://copr.fedorainfracloud.org/coprs/g/python/python-safe-path/
* GConf2
* ansible-core
* ansible-lint
* antlr4-project
* argparse-manpage
* asciinema
* astrometry
* asv
* auditwheel
* autodownloader
* autorandr
* bandit
* blender
* bodhi-client
* bodhi-server
* borgbackup
* borgmatic
* btrbk
* certbot
* cffconvert
* cfn-lint
* clang
* compiler-rt
* container-workflow-tool
* dblatex
* doge
* emacs-jedi
* ephemeral-port-reserve
* esptool
* fedfind
* fedora-easy-karma
* fedora-review
* fio
* gajim
* gi-docgen
* git-pull-request
* glances
* global
* gnofract4d
* go2rpm
* gtkpod
* hatch
* httpie
* ilua
* ini2toml
* inksmoto
* jack-mixer
* jello
* jrnl
* libsvm
* linode-cli
* llvm
* marshalparser
* mediawiki
* menulibre
* mkdocs
* monsterz
* mu
* mycli
* mygnuhealth
* netbox
* nicotine+
* nml
* noggin
* nox
* ntpsec
* nx-libs
* obserware
* ocrmypdf
* octave-miscellaneous
* offlineimap
* oraculum
* partio
* pcsc-lite
* pg_activity
* pipx
* playitagainsam
* poetry
* ptpython
* pynag
* pyp2spec
* pyplane
* pytest
* python-Automat
* python-PyLEMS
* python-TestSlide
* python-ZEO
* python-ZODB
* python-ajsonrpc
* python-ansi2html
* python-autopep8
* python-b4
* python-black
* python-bluepy
* python-bluepyopt
* python-blurb
* python-boutdata
* python-build
* python-cartopy
* python-cffsubr
* python-chardet
* python-charset-normalizer
* python-cheetah
* python-compreffor
* python-confluent-kafka
* python-cookiecutter
* python-cs
* python-cssutils
* python-cysignals
* python-daphne
* python-datadog
* python-datalad
* python-diff-cover
* python-dipy
* python-distro
* python-django
* python-dns-lexicon
* python-docutils
* python-doxytag2zealdb
* python-ephyviewer
* python-esphomeflasher
* python-exabgp
* python-ezdxf
* python-fabric
* python-fastavro
* python-fiona
* python-flake8
* python-flit
* python-fsleyes
* python-fslpy
* python-geotiler
* python-ghp-import
* python-git-changelog
* python-glymur
* python-gpxpy
* python-hatchling
* python-hdfs
* python-hdmf
* python-igor
* python-ipyparallel
* python-jenkins-job-builder
* python-jira
* python-josepy
* python-jsondiff
* python-jsonschema
* python-jupyter-client
* python-jupyter-console
* python-jupyter-core
* python-kafka
* python-keyring
* python-markdown-it-py
* python-molecule
* python-molmass
* python-myrepos-utils
* python-myst-parser
* python-name-that-hash
* python-nb2plots
* python-nbformat
* python-neurom
* python-niaarm
* python-nibabel
* python-nitrate
* python-nixio
* python-nose2
* python-notebook
* python-nudatus
* python-os-testr
* python-patatt
* python-pdfminer
* python-pint
* python-pipdeptree
* python-port-for
* python-pox
* python-ppft
* python-pudb
* python-pulp
* python-pybtex
* python-pycec
* python-pyct
* python-pygments
* python-pykwalify
* python-pypng
* python-pyside2
* python-pytest-bdd
* python-rasterio
* python-relatorio
* python-rq
* python-samloader
* python-sciunit
* python-scss
* python-shortuuid
* python-shtab
* python-sphinx
* python-sphinx-autobuild
* python-sphinx-theme-builder
* python-spyking-circus
* python-sshtunnel
* python-stem
* python-streamlink
* python-sure
* python-system-calls
* python-tldextract
* python-tmuxp
* python-toml-adapt
* python-towncrier
* python-tox
* python-tqdm
* python-twine
* python-twisted
* python-userpath
* python-versioningit
* python-virtualenv-clone
* python-watchgod
* python-websocket-client
* python-wloc
* python-xmlschema
* python-xnat
* python-zbase32
* python-zope-testrunner
* pythran
* rabbitvcs
* rdopkg
* roca-detect
* rofimoji
* salt-lint
* scribus-generator
* shybrid
* src
* sssd
* stomppy
* syslog-ng
* terminator
* thefuck
* thonny
* tldr
* tuna
* tuptime
* urlscan
* usd
* vips
* visidata
* wavextract
* weasyprint
* yt-dlp
* yubikey-manager
==== Components known to need opt-out ====
* llvm (files in `/usr/share/opt-viewer` are executable Python scripts with shebangs that import each other)
=== Opting out ===
If the new behavior is not desirable to your package, you can undefine the `%_py3_shebang_P` macro, like this:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%undefine _py3_shebang_P
Alternatively, you can amend the shebang flag macro (e.g. with `sed`) to remove the `P` flag.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ current Python packaging guidelines], e.g. `%pyproject_wheel` and `%pyproject_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python packaging guidelines], e.g. `%py3_build` and `%py3_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)
(The only difference is the name of the macro.)
=== Opting in ===
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ current Python packaging guidelines], e.g. `%pyproject_wheel` and `%pyproject_install`, the standard set of Python shebang flags is applied to all files with Python shebangs installed in `/usr/bin/`.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python packaging guidelines], e.g. `%py3_build` and `%py3_install`, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software.
If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_shebang_fix the `%py3_shebang_fix` macro].
=== What if the packager changes `%__python3` to an older version of Python ===
The `-P` flag was introduced in Python 3.11. When `%__python3` is redefined to an older Python version, e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this:
<nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")</nowiki>
=== What if the admin/user changes `/usr/bin/python3` to an older version of Python ===
The `-P` flag was introduced in Python 3.11. When an admin/user changes `/usr/bin/python3` to point to an older version of Python, e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would break the scripts.
However, changing `/usr/bin/python3` to a different Python would ''brick'' a Fedora system even now. So we don't consider that an issue. See an example that changes `/usr/bin/python3` to Python 3.9 (don't try this at home):
[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3
[root@086a2804411a /]# dnf --version
4.12.0
...
[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3
[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
  File "/usr/bin/dnf", line 61, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'
=== Risks ===
As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change.
The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ 201x-era Python RPM macros] set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that:
# not all Python scripts in `/usr/bin` will have the `-P` flag automatically
# some scripts '''not''' in `/usr/bin` might gain the `-P` flag as well
The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in `$PATH`. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the `%pyproject_install` macro, leaving packages with the "legacy" macros intact.


== Feedback ==
== Feedback ==
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
* Relevant upstream discussion: [https://discuss.python.org/t/13896 Should console_scripts entry points exclude the scripts directory from sys.path?]
* Adding `-P` to Python: [https://github.com/python/cpython/issues/57684 GitHub issue], [https://mail.python.org/archives/list/python-dev@python.org/thread/IU5Q2AXAURFVDPRWNU3BDFVKV2QX5NOR/ email thread]
* [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestreview-891177878 dnf maintainers response for doing this explicitly in dnf first]
Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts.
=== Feeedback from the Fedora's devel list ===
People asked why `-P`-like behavior isn't the default for Python instead. While the change owners agree that that would indeed be a saner default, Python has ''just now'' added the `-P` option. Changing the default behavior is not easy. That approach might be considered too disruptive to users. This proposal at least gets the security benefits for all system shipped stuff, without breaking anything the user has been using from non-packaged locations. (Adapted from [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/WEIETOO34KUZNNQSIZU5YVIHRP2RATCN/ a reply by Daniel P. Berrangé].)


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 89: Line 451:
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
-->
-->
Python programs in `/usr/bin` will be less fragile to other random files being present in `/usr/bin`.
Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340 python3-notebook: ImportError: bad magic number in six] and [https://github.com/benjaminp/six/issues/359 bad magic number in six] will not happen.


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
** Test everything in Copr
** If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild
** Provide guidance to packagers, fix bugs if needed
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
** Observe their packages, find and report bugs, opt-out if needed
** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or by converting their packages to `%pyproject_install`
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: [https://pagure.io/releng/issue/10784 #10784] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->


* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: The new flag needs to be documented in the Python packaging guidelines: https://pagure.io/packaging-committee/pull-request/1191 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: not needed for this Change
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


* Alignment with Objectives:  
* Alignment with Objectives: no
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->


Line 114: Line 484:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
No impact is anticipated.


== How To Test ==
== How To Test ==
Line 133: Line 503:
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


* Low level: Examine the value of the changed RPM macros, it should contain the `-P` flag
* Middle level: Examine the shebang lines of RPM-installed Python scripts in `/usr/bin`, it should contain the `-P` flag
* High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from `/usr/bin`


== User Experience ==
== User Experience ==
Line 145: Line 518:
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
-->
-->
Users of RPM-installed scripts should get a safer experience by default.
Users of Python should not observe a difference, the behavior is not a new default: the `-P` flag needs to be explicitly used.


== Dependencies ==
== Dependencies ==
Line 151: Line 526:
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


We need [[Changes/Python3.11]] first.


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: defer to F38; only add `-P` to shebangs in `%pyproject_install` to keep backward compatibility of the old macros; revert and rebuild <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: 1 week before the beta freeze <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 


== Documentation ==
== Documentation ==
Line 166: Line 541:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
* This page
* TBD Updated Python guidelines
* https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
* https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights (Security improvements)


== Release Notes ==
== Release Notes ==
Line 174: Line 552:
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
-->
-->
TBD

Latest revision as of 15:55, 20 July 2022


Python: Add -P to default shebangs

Summary

The -P flag will be added to the Python shebang macros (%{py3_shbang_opts}, %{py3_shebang_flags}, ...). Packages that adhere to those macros will change their Python shebangs from #! /usr/bin/python3 -s to #! /usr/bin/python3 -sP and as a result, will no longer have the directory of the script (such as /usr/bin) in sys.path. An opt-out mechanism exists.

Owner

Current status

Detailed Description

All Python 3 shebang RPM macros will be changed to contain one more flag: -P. Previously, they contained -s, now they will contain -sP.

From the documentation for the -P option:

Don’t prepend a potentially unsafe path to sys.path:
  • python -m module command line: Don’t prepend the current working directory.
  • python script.py command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
  • python -c code and python (REPL) command lines: Don’t prepend an empty string, which means the current working directory.

In shebangs, only the middle option (don’t prepend the script’s directory) is relevant.

Consider the following executbale script installed as /usr/bin/let-there-be-fun:

#! /usr/bin/python3 -s
import abc
...

When the script is directly executed (e.g. by running let-there-be-fun from the console), the script's directory (/usr/bin) is prepended to sys.path. Python tries to locate an importable abc module in /usr/bin first. This can cause real issues: python3-notebook: ImportError: bad magic number in six and bad magic number in six.

When the shebang includes -P:

#! /usr/bin/python3 -sP
import abc
...

...the script's directory (/usr/bin) is not prepended to sys.path. The change owners consider this approach safer for the majority of Fedora's RPM packages.

By default, all standardly RPM-packaged Python packages with scripts in /usr/bin will gain the -P flag in their shebang, assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.)

List of RPM macros that will gain -P

  • %{py3_shbang_opts}
  • %{py3_shbang_opts_nodash}
  • %{py3_shebang_flags}
  • %{py_shbang_opts}
  • %{py_shbang_opts_nodash}
  • %{py_shebang_flags}

List of source RPM packages that will gain -P in some of the built package(s)

Queried on 2022-07-07 in https://copr.fedorainfracloud.org/coprs/g/python/python-safe-path/

  • GConf2
  • ansible-core
  • ansible-lint
  • antlr4-project
  • argparse-manpage
  • asciinema
  • astrometry
  • asv
  • auditwheel
  • autodownloader
  • autorandr
  • bandit
  • blender
  • bodhi-client
  • bodhi-server
  • borgbackup
  • borgmatic
  • btrbk
  • certbot
  • cffconvert
  • cfn-lint
  • clang
  • compiler-rt
  • container-workflow-tool
  • dblatex
  • doge
  • emacs-jedi
  • ephemeral-port-reserve
  • esptool
  • fedfind
  • fedora-easy-karma
  • fedora-review
  • fio
  • gajim
  • gi-docgen
  • git-pull-request
  • glances
  • global
  • gnofract4d
  • go2rpm
  • gtkpod
  • hatch
  • httpie
  • ilua
  • ini2toml
  • inksmoto
  • jack-mixer
  • jello
  • jrnl
  • libsvm
  • linode-cli
  • llvm
  • marshalparser
  • mediawiki
  • menulibre
  • mkdocs
  • monsterz
  • mu
  • mycli
  • mygnuhealth
  • netbox
  • nicotine+
  • nml
  • noggin
  • nox
  • ntpsec
  • nx-libs
  • obserware
  • ocrmypdf
  • octave-miscellaneous
  • offlineimap
  • oraculum
  • partio
  • pcsc-lite
  • pg_activity
  • pipx
  • playitagainsam
  • poetry
  • ptpython
  • pynag
  • pyp2spec
  • pyplane
  • pytest
  • python-Automat
  • python-PyLEMS
  • python-TestSlide
  • python-ZEO
  • python-ZODB
  • python-ajsonrpc
  • python-ansi2html
  • python-autopep8
  • python-b4
  • python-black
  • python-bluepy
  • python-bluepyopt
  • python-blurb
  • python-boutdata
  • python-build
  • python-cartopy
  • python-cffsubr
  • python-chardet
  • python-charset-normalizer
  • python-cheetah
  • python-compreffor
  • python-confluent-kafka
  • python-cookiecutter
  • python-cs
  • python-cssutils
  • python-cysignals
  • python-daphne
  • python-datadog
  • python-datalad
  • python-diff-cover
  • python-dipy
  • python-distro
  • python-django
  • python-dns-lexicon
  • python-docutils
  • python-doxytag2zealdb
  • python-ephyviewer
  • python-esphomeflasher
  • python-exabgp
  • python-ezdxf
  • python-fabric
  • python-fastavro
  • python-fiona
  • python-flake8
  • python-flit
  • python-fsleyes
  • python-fslpy
  • python-geotiler
  • python-ghp-import
  • python-git-changelog
  • python-glymur
  • python-gpxpy
  • python-hatchling
  • python-hdfs
  • python-hdmf
  • python-igor
  • python-ipyparallel
  • python-jenkins-job-builder
  • python-jira
  • python-josepy
  • python-jsondiff
  • python-jsonschema
  • python-jupyter-client
  • python-jupyter-console
  • python-jupyter-core
  • python-kafka
  • python-keyring
  • python-markdown-it-py
  • python-molecule
  • python-molmass
  • python-myrepos-utils
  • python-myst-parser
  • python-name-that-hash
  • python-nb2plots
  • python-nbformat
  • python-neurom
  • python-niaarm
  • python-nibabel
  • python-nitrate
  • python-nixio
  • python-nose2
  • python-notebook
  • python-nudatus
  • python-os-testr
  • python-patatt
  • python-pdfminer
  • python-pint
  • python-pipdeptree
  • python-port-for
  • python-pox
  • python-ppft
  • python-pudb
  • python-pulp
  • python-pybtex
  • python-pycec
  • python-pyct
  • python-pygments
  • python-pykwalify
  • python-pypng
  • python-pyside2
  • python-pytest-bdd
  • python-rasterio
  • python-relatorio
  • python-rq
  • python-samloader
  • python-sciunit
  • python-scss
  • python-shortuuid
  • python-shtab
  • python-sphinx
  • python-sphinx-autobuild
  • python-sphinx-theme-builder
  • python-spyking-circus
  • python-sshtunnel
  • python-stem
  • python-streamlink
  • python-sure
  • python-system-calls
  • python-tldextract
  • python-tmuxp
  • python-toml-adapt
  • python-towncrier
  • python-tox
  • python-tqdm
  • python-twine
  • python-twisted
  • python-userpath
  • python-versioningit
  • python-virtualenv-clone
  • python-watchgod
  • python-websocket-client
  • python-wloc
  • python-xmlschema
  • python-xnat
  • python-zbase32
  • python-zope-testrunner
  • pythran
  • rabbitvcs
  • rdopkg
  • roca-detect
  • rofimoji
  • salt-lint
  • scribus-generator
  • shybrid
  • src
  • sssd
  • stomppy
  • syslog-ng
  • terminator
  • thefuck
  • thonny
  • tldr
  • tuna
  • tuptime
  • urlscan
  • usd
  • vips
  • visidata
  • wavextract
  • weasyprint
  • yt-dlp
  • yubikey-manager

Components known to need opt-out

  • llvm (files in /usr/share/opt-viewer are executable Python scripts with shebangs that import each other)

Opting out

If the new behavior is not desirable to your package, you can undefine the %_py3_shebang_P macro, like this:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%undefine _py3_shebang_P

Alternatively, you can amend the shebang flag macro (e.g. with sed) to remove the P flag.

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

Opting in

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, the standard set of Python shebang flags is applied to all files with Python shebangs installed in /usr/bin/.

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use the %py3_shebang_fix macro.

What if the packager changes %__python3 to an older version of Python

The -P flag was introduced in Python 3.11. When %__python3 is redefined to an older Python version, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this:

%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")

What if the admin/user changes /usr/bin/python3 to an older version of Python

The -P flag was introduced in Python 3.11. When an admin/user changes /usr/bin/python3 to point to an older version of Python, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts.

However, changing /usr/bin/python3 to a different Python would brick a Fedora system even now. So we don't consider that an issue. See an example that changes /usr/bin/python3 to Python 3.9 (don't try this at home):

[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3

[root@086a2804411a /]# dnf --version
4.12.0
...

[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
  File "/usr/bin/dnf", line 61, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'

Risks

As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change.

The 201x-era Python RPM macros set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that:

  1. not all Python scripts in /usr/bin will have the -P flag automatically
  2. some scripts not in /usr/bin might gain the -P flag as well

The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in $PATH. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the %pyproject_install macro, leaving packages with the "legacy" macros intact.

Feedback

Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts.

Feeedback from the Fedora's devel list

People asked why -P-like behavior isn't the default for Python instead. While the change owners agree that that would indeed be a saner default, Python has just now added the -P option. Changing the default behavior is not easy. That approach might be considered too disruptive to users. This proposal at least gets the security benefits for all system shipped stuff, without breaking anything the user has been using from non-packaged locations. (Adapted from a reply by Daniel P. Berrangé.)

Benefit to Fedora

Python programs in /usr/bin will be less fragile to other random files being present in /usr/bin. Real hard-to-debug issues like python3-notebook: ImportError: bad magic number in six and bad magic number in six will not happen.

Scope

  • Proposal owners:
    • Test everything in Copr
    • If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild
    • Provide guidance to packagers, fix bugs if needed
  • Other developers:
    • Observe their packages, find and report bugs, opt-out if needed
    • Volunteerily opt-in by calling the %py3_shebang_fix macro and/or by converting their packages to %pyproject_install
  • Trademark approval: not needed for this Change
  • Alignment with Objectives: no

Upgrade/compatibility impact

No impact is anticipated.

How To Test

  • Low level: Examine the value of the changed RPM macros, it should contain the -P flag
  • Middle level: Examine the shebang lines of RPM-installed Python scripts in /usr/bin, it should contain the -P flag
  • High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from /usr/bin

User Experience

Users of RPM-installed scripts should get a safer experience by default. Users of Python should not observe a difference, the behavior is not a new default: the -P flag needs to be explicitly used.

Dependencies

We need Changes/Python3.11 first.

Contingency Plan

  • Contingency mechanism: defer to F38; only add -P to shebangs in %pyproject_install to keep backward compatibility of the old macros; revert and rebuild
  • Contingency deadline: 1 week before the beta freeze
  • Blocks release? No

Documentation

Release Notes

TBD