From Fedora Project Wiki
(Description 2)
m (Add trackers)
 
(24 intermediate revisions by 3 users not shown)
Line 21: Line 21:


<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
= Python distutils to use build flags for extension builders (XXX better name?) <!-- The name of your change proposal --> =
= Avoid Fedora-specific build flags in non-RPM Python extensions <!-- The name of your change proposal --> =


== Summary ==
== Summary ==
Line 27: Line 27:
Note that motivation for the change should be in the Motivation section below, and this part should answer the question "What?" rather than "Why?". -->
Note that motivation for the change should be in the Motivation section below, and this part should answer the question "What?" rather than "Why?". -->


The build flags (<code>CFLAGS</code>, <code>CXXFLAGS</code> and <code>LDFLAGS</code>) saved in the Python's distutils module for building extension modules are switched from <code>%{build_cflags}</code>, <code>%{build_cxxflags}</code> and <code>%{build_ldflags}</code> to <code>%{extension_cflags}</code>, <code>%{extension_cxxflags}</code> and <code>%{extension_ldflags}</code>.
The build flags (<code>CFLAGS</code>, <code>CXXFLAGS</code> and <code>LDFLAGS</code>) saved in the Python's distutils module for building extension modules are switched from:
* <code>%{build_cflags}</code>,
* <code>%{build_cxxflags}</code> and
* <code>%{build_ldflags}</code>
to:
* <code>%{extension_cflags}</code>,
* <code>%{extension_cxxflags}</code> and
* <code>%{extension_ldflags}</code>.


This [https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders currently means] no GCC plugins (such as annobin) are activated and no GCC spec files (<code>-specs=</code> arguments) are used by default when building Python extension modules (e.g. with <code>python3 setup.py build</code>). That also means the {{package|python3-devel}} package can loose a runtime dependency on {{package|redhat-rpm-config}}.
This [https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders currently means] that no GCC plugins (such as annobin) are activated and no GCC spec files (<code>-specs=</code> arguments) are used by default when building Python extension modules (e.g. with <code>python3 setup.py build</code>).


The change mostly affects building extension modules by users, outside of RPM environment. The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (<code>%{build_cflags}</code> and friends), unless the package uses nonstandard methods to build the extensions.
The {{package|python3-devel}} package will lose its runtime dependency on {{package|redhat-rpm-config}} (which was only required for annobin support and GCC spec files).
 
The change will affect building extension modules by users, outside of the RPM environment. The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (<code>%{build_cflags}</code> and friends), unless the package uses nonstandard methods to build the extensions.


Only Python 3.7 ({{package|python3}}) and 3.6 ({{package|python36}}) will be changed.
Only Python 3.7 ({{package|python3}}) and 3.6 ({{package|python36}}) will be changed.
Line 63: Line 72:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1668029 #1668029]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/286 #286]


== Detailed Description ==
== Detailed Description ==
Line 82: Line 92:
The problematic flags are GCC plugins (such as annobin) and GCC spec files (<code>-specs=</code> arguments).
The problematic flags are GCC plugins (such as annobin) and GCC spec files (<code>-specs=</code> arguments).


Example: Any Python developer using Fedora automatically builds Python extension modules with annobin and hardening flags by default even if they don't need that. They might build the extension on Fedora, test it and later ship it and build it on a CI that is not based on Fedora and get a different results.
Example: Any Python developer using Fedora automatically builds Python extension modules with annobin and hardening flags by default even if they don't need that. They might build the extension on Fedora, test it and later ship it and build it on a CI that is not based on Fedora and get different results.


=== The solution ===
=== The Solution ===


The solution is not to save the problematic flags, but only the flags needed. Until recently, this would be hackish, but [https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders a designated set of flags] was created in Fedora 30, supposed to be used by extension builders (such as the Python's distutil module): <code>%{extension_cflags}</code>, <code>%{extension_cxxflags}</code> and <code>%{extension_ldflags}</code>.
The solution is not to save the problematic flags, but only the flags needed. Until recently, this would be hackish, but [https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflags.md#support-for-extension-builders a designated set of flags] was created in Fedora 30, supposed to be used by extension builders (such as the Python's distutil module): <code>%{extension_cflags}</code>, <code>%{extension_cxxflags}</code> and <code>%{extension_ldflags}</code>.
Line 99: Line 109:
</blockquote>
</blockquote>


Python already had (an incomplete) ability to specify a different set of flags for itself and for the extensions. In Python 3.7.2 and 3.6.8 the ability was completed by providing a way to do this for the <code>LDFAGS</code> (this was actually driven by Fedora's needs). Currently we are able to set a different set of flags for building Python and for building user extension modules. The code to do this can be inspected at the Fedora [https://src.fedoraproject.org/rpms/python3/pull-request/75 pull request implementing this change].
Python already had (incomplete) support for specifying different sets of flags for itself and for the extensions. In Python 3.7.2 and 3.6.8 the ability was completed by providing a way to do this for <code>LDFAGS</code> (this was actually driven by Fedora's needs). Currently we are able to set a different set of flags for building Python and for building user extension modules. The code implementing this change can be inspected at [https://src.fedoraproject.org/rpms/python3/pull-request/75 Fedora python3 PR #75] and [https://github.com/python/cpython/pull/10900 CPython PR #10900].


=== Impact on users building extension modules ===
=== Impact on users building extension modules ===


When an user compiles a Python extension module and does not specify any flags explicitly, this is what they used to had:
When an user compiles a Python extension module outside of RPMBuild and does not specify any flags explicitly, this is what they used to have:


  $ python3 setup.py build
  $ python3 setup.py build
Line 127: Line 137:
  gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so
  gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so


Users are able to provide their own flags by setting the <code>CFLAGS</code>/<code>CXXFLAGS</code> and <code>LDFLAGS</code> environment variables. They can for example opt-in for annobin if they wish to do so.
Users are able to provide their own flags by setting the <code>CFLAGS</code>/<code>CXXFLAGS</code> and <code>LDFLAGS</code> environment variables. They can, for example, opt-in for annobin if they wish to do so.


=== Impact on RPM packages building extension modules ===
=== Impact on RPM packages building extension modules ===
Line 153: Line 163:


Worst case: Some package won't have annobin.
Worst case: Some package won't have annobin.
=== Python interpreters affected by this change ===
This change is fully possible (without backporting) in Python 3.6.8 / 3.7.2 or newer. Hence we'll only change the {{package|python3}} and {{package|python36}} packages.
We'll keep older Pythons intact and let them finish their lifetime as they are. Newer Python versions will inherit this change.
PyPy 3 ({{package|pypy3}}) might eventually be updated to a version that supports this fully. Since no packages in Fedora use PyPy 3, we'll update it then without a change proposal and we'll document the fact on this Change page. (If any package starts using PyPy 3 untill then, we'll coordinate with the maintainer/s).


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 183: Line 200:
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
-->
-->
Python developers will get more upstream-like experience when building Python extension modules.
Python developers won't need {{package|redhat-rpm-macros}} (and dozens of other language-specific packages that go with that) installed just to do so.
New decisions made about the distro packages won't necessarily affect Python developers building their extension modules.


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
** Review, merge and build the [https://src.fedoraproject.org/rpms/python3/pull-request/75 pull request with the implementation].
** Go through the packages manually invoking <code>setup.py</code> to build extension modules and provide fixes. Update to <code>%py3_build</code> where possible, set the flags manually otherwise.


* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: Are encouraged to switch to <code>%py3_build</code> (but they don't need to do anything, this is not a System Wide Change). <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
* Release engineering: [https://pagure.io/releng/issue/8027 #8027] (mass rebuild not needed, no releng impact anticipated). <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->


* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: already in place. <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
* Trademark approval: N/A (not needed for this Change)
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


== Upgrade/compatibility impact ==
== Upgrade/compatibility impact ==
Line 207: Line 225:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.


== How To Test ==
== How To Test ==
Line 225: Line 243:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
 
=== For users (Python developers) ===
 
# build your favorite Python extension module in venv or outside venv with <code>setup.py build</code> or  <code>setup.py build_ext</code>
# observe the used flags and check annobin and <code>-specs</code> '''are not there''', report bugs for {{package|python3}} otherwise (and block our tracking bug)
# check if the extension works as expected
 
=== For packagers (Fedora contributors)  ===
 
# build your favorite RPM package with Python extension module
# observe the used flags and check annobin and <code>-specs</code> '''are there''', report bugs for that package otherwise (and block our tracking bug)
# check if the package works as expected


== User Experience ==
== User Experience ==
Line 238: Line 267:
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
-->
-->
See '''Benefit to Fedora''' above.


== Dependencies ==
== Dependencies ==
Line 243: Line 274:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
Changes in {{package|redhat-rpm-config}} are already done.
 
We will check dependent Fedora packages.


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Change owners can revert the change at any point. If removing the {{package|redhat-rpm-config}} dependency turns out to be problematic, it can be added back until the problem is fixed. <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: final freeze <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
* Blocks product? None <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation ==
Line 259: Line 292:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
This page is the documentation.


== Release Notes ==
== Release Notes ==
Line 268: Line 301:
-->
-->


[[Category:ChangePageIncomplete]]
TBD.
 
Note: We should properly (explicitly) communicate this in the release notes, i.e. highlight that the binaries of user-compiled extensions will not be position independent anymore by default and explain how users could get the behavior again.
 
[[Category:ChangeAcceptedF30]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->

Latest revision as of 17:58, 21 January 2019


Avoid Fedora-specific build flags in non-RPM Python extensions

Summary

The build flags (CFLAGS, CXXFLAGS and LDFLAGS) saved in the Python's distutils module for building extension modules are switched from:

  • %{build_cflags},
  • %{build_cxxflags} and
  • %{build_ldflags}

to:

  • %{extension_cflags},
  • %{extension_cxxflags} and
  • %{extension_ldflags}.

This currently means that no GCC plugins (such as annobin) are activated and no GCC spec files (-specs= arguments) are used by default when building Python extension modules (e.g. with python3 setup.py build).

The Package-x-generic-16.pngpython3-devel package will lose its runtime dependency on Package-x-generic-16.pngredhat-rpm-config (which was only required for annobin support and GCC spec files).

The change will affect building extension modules by users, outside of the RPM environment. The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (%{build_cflags} and friends), unless the package uses nonstandard methods to build the extensions.

Only Python 3.7 (Package-x-generic-16.pngpython3) and 3.6 (Package-x-generic-16.pngpython36) will be changed.

Owner

Current status

Detailed Description

The Problem

When Python is built, it saves the flags (CFLAGS, CXXFLAGS and LDFLAGS) for further use when building extension modules into a designated Makefile. The distutils module (a component responsible for building Python packages and extension modules) then reads the file and applies the flags. You can see the file at /usr/lib64/python3.7/config-3.7m-x86_64-linux-gnu/Makefile in Package-x-generic-16.pngpython3-libs. This is mostly done to make user-built extension modules binary compatible with the Python interpreter they are being built for.

Traditionally (=before this change), the Package-x-generic-16.pngpython3 package was created in a way that it simply saved the same set of flags used for building itself. This proved problematic as the flags used to build Fedora packages grew specific things (not actually needed for binary compatibility of the extension modules) and several workarounds needed to be made, most specifically the Package-x-generic-16.pngpython3-devel package got a runtime dependency on Package-x-generic-16.pngredhat-rpm-config:

The problematic flags are GCC plugins (such as annobin) and GCC spec files (-specs= arguments).

Example: Any Python developer using Fedora automatically builds Python extension modules with annobin and hardening flags by default even if they don't need that. They might build the extension on Fedora, test it and later ship it and build it on a CI that is not based on Fedora and get different results.

The Solution

The solution is not to save the problematic flags, but only the flags needed. Until recently, this would be hackish, but a designated set of flags was created in Fedora 30, supposed to be used by extension builders (such as the Python's distutil module): %{extension_cflags}, %{extension_cxxflags} and %{extension_ldflags}.

The documentation for the flags currently reads:

The current set of differences are:

  • No GCC plugins (such as annobin) are activated.
  • No GCC spec files (-specs= arguments) are used.

Additional flags may be removed in the future if they prove to be incompatible with alternative toolchains.

Python already had (incomplete) support for specifying different sets of flags for itself and for the extensions. In Python 3.7.2 and 3.6.8 the ability was completed by providing a way to do this for LDFAGS (this was actually driven by Fedora's needs). Currently we are able to set a different set of flags for building Python and for building user extension modules. The code implementing this change can be inspected at Fedora python3 PR #75 and CPython PR #10900.

Impact on users building extension modules

When an user compiles a Python extension module outside of RPMBuild and does not specify any flags explicitly, this is what they used to have:

$ python3 setup.py build
running build
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c demo.c -o build/temp.linux-x86_64-3.7/demo.o
creating build/lib.linux-x86_64-3.7
gcc -pthread -shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so

After the change, they will get:

$ python3 setup.py build
running build
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c demo.c -o build/temp.linux-x86_64-3.7/demo.o
creating build/lib.linux-x86_64-3.7
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so

Users are able to provide their own flags by setting the CFLAGS/CXXFLAGS and LDFLAGS environment variables. They can, for example, opt-in for annobin if they wish to do so.

Impact on RPM packages building extension modules

When the spec file of the package sets the CFLAGS/CXXFLAGS and LDFLAGS variables to the expected Fedora values (e.g. CFLAGS=${RPM_OPT_FLAGS} LDFLAGS=${RPM_LD_FLAGS} or %{build_cflags}, %{build_cxxflags}, %{build_ldflags}, %{optflags} etc.) everything works as expected. The currently documented way of building Python packages (%py3_build and %py3_install) does this for you, we recommend using it.

There might be some differences in the log, because the flags concatenate. Here is the example from the Package-x-generic-16.pngpython-psycopg2 build.log.

Before this change:

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000 -DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include -I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement

After this change:

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000 -DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include -I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement

If the package does python3 setup.py build to build the extension modules manually without setting the flags, the flags are not set properly. This is probably very rare. Even before the %py3_build and %py3_install macros, the preferred way was to call:

CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build  # from EL4

However, LDFLAGS are not set in that example.

There are 485 packages that have the %{python3_sitearch} macro in them and hence are most likely to build Python extension modules. Out of those, 173 have setup.py directly in the spec. 65 have setup.py build. Change owners will go trough the list and provide fixes. Packages are encouraged to switch to the %py3_build macro themselves.

Worst case: Some package won't have annobin.

Python interpreters affected by this change

This change is fully possible (without backporting) in Python 3.6.8 / 3.7.2 or newer. Hence we'll only change the Package-x-generic-16.pngpython3 and Package-x-generic-16.pngpython36 packages. We'll keep older Pythons intact and let them finish their lifetime as they are. Newer Python versions will inherit this change.

PyPy 3 (Package-x-generic-16.pngpypy3) might eventually be updated to a version that supports this fully. Since no packages in Fedora use PyPy 3, we'll update it then without a change proposal and we'll document the fact on this Change page. (If any package starts using PyPy 3 untill then, we'll coordinate with the maintainer/s).

Benefit to Fedora

Python developers will get more upstream-like experience when building Python extension modules. Python developers won't need Package-x-generic-16.pngredhat-rpm-macros (and dozens of other language-specific packages that go with that) installed just to do so. New decisions made about the distro packages won't necessarily affect Python developers building their extension modules.

Scope

  • Proposal owners:
    • Review, merge and build the pull request with the implementation.
    • Go through the packages manually invoking setup.py to build extension modules and provide fixes. Update to %py3_build where possible, set the flags manually otherwise.
  • Other developers: Are encouraged to switch to %py3_build (but they don't need to do anything, this is not a System Wide Change).
  • Release engineering: #8027 (mass rebuild not needed, no releng impact anticipated).
  • Policies and guidelines: already in place.

Upgrade/compatibility impact

Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.

How To Test

For users (Python developers)

  1. build your favorite Python extension module in venv or outside venv with setup.py build or setup.py build_ext
  2. observe the used flags and check annobin and -specs are not there, report bugs for Package-x-generic-16.pngpython3 otherwise (and block our tracking bug)
  3. check if the extension works as expected

For packagers (Fedora contributors)

  1. build your favorite RPM package with Python extension module
  2. observe the used flags and check annobin and -specs are there, report bugs for that package otherwise (and block our tracking bug)
  3. check if the package works as expected

User Experience

See Benefit to Fedora above.

Dependencies

Changes in Package-x-generic-16.pngredhat-rpm-config are already done.

We will check dependent Fedora packages.

Contingency Plan

  • Contingency mechanism: Change owners can revert the change at any point. If removing the Package-x-generic-16.pngredhat-rpm-config dependency turns out to be problematic, it can be added back until the problem is fixed.
  • Contingency deadline: final freeze
  • Blocks release? No
  • Blocks product? None

Documentation

This page is the documentation.

Release Notes

TBD.

Note: We should properly (explicitly) communicate this in the release notes, i.e. highlight that the binaries of user-compiled extensions will not be position independent anymore by default and explain how users could get the behavior again.