Changes/PythonSafePath

Python: Add -P to default shebangs

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

The -P flag will be added to the Python shebang macros (%{py3_shbang_opts}, %{py3_shebang_flags}, ...). Packages that adhere to those macros will change their Python shebangs from #! /usr/bin/python3 -s to #! /usr/bin/python3 -sP and as a result, will no longer have the directory of the script (such as /usr/bin) in sys.path. An opt-out mechanism exists.

Owner

• Name: Miro Hrončok, Victor Stinner
• Email: python-maint@redhat.com

Current status

• Targeted release: Fedora Linux 37
• Last updated: 2022-05-10
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

All Python 3 shebang RPM macros will be changed to contain one more flag: -P. Previously, they contained -s, now they will contain -sP.

From the documentation for the -P option:

Don’t prepend a potentially unsafe path to sys.path:

• python -m module command line: Don’t prepend the current working directory.
• python script.py command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
• python -c code and python (REPL) command lines: Don’t prepend an empty string, which means the current working directory.

In shebangs, only the middle option (don’t prepend the script’s directory) is relevant.

Consider the following executbale script installed as /usr/bin/let-there-be-fun:

#! /usr/bin/python3 -s
import abc
...

When the script is directly executed (e.g. by running let-there-be-fun from the console), the script's directory (/usr/bin) is prepended to sys.path. Python tries to locate an importable abc module in /usr/bin first. This can cause real issues: python3-notebook: ImportError: bad magic number in six and bad magic number in six.

When the shebang includes -P:

#! /usr/bin/python3 -sP
import abc
...

...the script's directory (/usr/bin) is not prepended to sys.path. The change owners consider this approach safer for the majority of Fedora's RPM packages.

By default, all standardly RPM-packaged Python packages with scripts in /usr/bin will gain the -P flag in their shebang, assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.)

List of RPM macros that will gain -P

• %{py3_shbang_opts}
• %{py3_shbang_opts_nodash}
• %{py3_shebang_flags}
• %{py_shbang_opts}
• %{py_shbang_opts_nodash}
• %{py_shebang_flags}

Opting out

If the new behavior is not desirable to your package amend the macros (e.g. with sed) to remove the P flag.

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

Opting in

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, the standard set of Python shebang flags is applied to all files with Python shebangs installed in /usr/bin/.

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use the %py3_shebang_fix macro.

What if the packager changes %__python3 to an older version of Python

The -P flag was introduced in Python 3.11. When %__python3 is redefined to an older Python version, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this:

%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")

What if the admin/user changes /usr/bin/python3 to an older version of Python

The -P flag was introduced in Python 3.11. When an admin/user changes /usr/bin/python3 to point to an older version of Python, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts.

However, changing /usr/bin/python3 to a different Python would brick a Fedora system even now. So we don't consider that an issue. See an example that changes /usr/bin/python3 to Python 3.9 (don't try this at home):

[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3

[root@086a2804411a /]# dnf --version
4.12.0
...

[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
  File "/usr/bin/dnf", line 61, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'

Risks

As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change.

The 201x-era Python RPM macros set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that:

1. not all Python scripts in /usr/bin will have the -P flag automatically
2. some scripts not in /usr/bin might gain the -P flag as well

The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in $PATH. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the %pyproject_install macro, leaving packages with the "legacy" macros intact.

Feedback

• Relevant upstream discussion: Should console_scripts entry points exclude the scripts directory from sys.path?
• Adding -P to Python: GitHub issue, email thread
• dnf maintainers response for doing this explicitly in dnf first

Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts.

Benefit to Fedora

Python programs in /usr/bin will be less fragile to other random files being present in /usr/bin. Real hard-to-debug issues like python3-notebook: ImportError: bad magic number in six and bad magic number in six will not happen.

Scope

• Proposal owners:
  • Test everything in Copr
  • If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild
  • Provide guidance to packagers, fix bugs if needed

• Other developers:
  • Observe their packages, find and report bugs, opt-out if needed
  • Volunteerily opt-in by calling the %py3_shebang_fix macro and/or by converting their packages to %pyproject_install

• Release engineering: This change does not require coordination with release engineering, #10784

• Policies and guidelines: The new flag needs to be documented in the Python packaging guidelines (old and new)

• Trademark approval: not needed for this Change

• Alignment with Objectives: no

Upgrade/compatibility impact

No impact is anticipated.

How To Test

• Low level: Examine the value of the changed RPM macros, it should contain the -P flag
• Middle level: Examine the shebang lines of RPM-installed Python scripts in /usr/bin, it should contain the -P flag
• High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from /usr/bin

User Experience

Users of RPM-installed scripts should get a safer experience by default. Users of Python should not observe a difference, the behavior is not a new default: the -P flag needs to be explicitly used.

Dependencies

We need Changes/Python3.11 first.

Contingency Plan

• Contingency mechanism: defer to F38; only add -P to shebangs in %pyproject_install to keep backward compatibility of the old macros; revert and rebuild
• Contingency deadline: 1 week before the beta freeze
• Blocks release? No

Documentation

• This page
• TBD Updated Python guidelines
• https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
• https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights (Security improvements)

Release Notes

TBD