Latest revision as of 23:38, 28 September 2023

Further reduce Fedora-specific build flags in non-RPM Python extensions

Summary

Continuing the work started with https://fedoraproject.org/wiki/Changes/Python_Extension_Flags, this change is about further reducing the build and linker flags (CFLAGS and LDFLAGS) saved internally in the Python interpreter for use by distutils and other build systems. Compiling non-RPM Python extension modules will carry only the compiler flags required for binary compatibility with the interpreter they were built against and not Fedora specific ones.

Practically that means the only Fedora derived flags will be -fexceptions and -fcf-protection and Python will apply its own upstream hardcoded ones, making the final flag set for a non-RPM compiled Python extension as follows:

-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection

Python C extensions built as rpm's will not be affected.

The current main Python interpreter on Fedora 39 will be modified (Python 3.12) and Python 3.6-3.11 will follow.

This change will affect every package that provides support for extension builders via utilizing the %{extension...flags} macros which at the time being is only Python.

Owner

Name: Charalampos Stratakis
Email: python-maint AT redhat.com

Current status

Targeted release: Fedora Linux 39
Last updated: 2023-09-28
devel thread
FESCo issue: #3026
Tracker bug: #2229725
Release notes tracker: #1020

Detailed Description

After implementing https://fedoraproject.org/wiki/Changes/Python_Extension_Flags we uncoupled some distro specific compilation and linker flags propagated to C extensions.

However with an ever increasing set of compiler flags being added and applied distro-wide, as compilers and security standards evolve (e.g. -D_FORTIFY_SOURCE=3) it becomes an increasingly complex job to vet each flag that might leak into user-built Python C extensions through the Python interpreter. Instead of removing only some flags and letting the rest follow through, we will be taking a more proactive approach by removing all the compiler and linker flags, except the ones that are required to maintain the binary compatibility with the Python interpreter the extensions were built against which are -fexceptions and -fcf-protection. We will also preserve the ones that Python hardcodes itself through the Makefile.

Similarly, when a user builds their own C programs, no compiler flags are applied by default and the user is free to making their own decision. Bringing the compilation of Python C extensions closer to that experience is the next logical step.

Currently a user-built Python C extension will be built with:

CFLAGS: -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv

LDFLAGS: '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g'

After this change:

CFLAGS: -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection

LDFLAGS: None

Feedback

The initial thread that inspired this change was https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76/#76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76

On the initial testing pull request feedback was provided that the -fcf-protection flag should also be added to the minimal set, due to potential binary compatibility issues. Hence the changes page has been updated accordingly.

Benefit to Fedora

Python developers will get more upstream-like experience when building Python extension modules and also closer to building vanilla C programs. Also new decisions made about the distro-wide compiler flags won't necessarily affect Python developers building their extension modules.

In addition any Python developer using Fedora will have the capability to build the extension on Fedora, test it and later ship it and build it on a CI or other systems that are not based on Fedora.

Scope

Proposal owners: Review, merge and build the redhat-rpm-config PR and the apply the relevant changes in the Python interpreters (python3.6 python3.7 python3.8 python3.9 python3.10 python3.11 python3.12)

Other developers: No requirements apart from welcoming testing their C extensions

Release engineering: No mass rebuild required and no releng impact anticipated.#Releng issue number

Policies and guidelines: N/A (not needed for this Change)

Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.

How To Test

For users (Python developers)

build your favorite Python extension module in venv or outside venv with your favorite build system
observe the used flags and check that the full set of flags are are not there as mentioned in the detailed description, report bugs for python3.12 otherwise (and block our tracking bug)
check if the extension works as expected

For packagers (Fedora contributors)

build your favorite RPM package with Python extension module
observe the used flags and check that the full set of flags are there and not the reduced one, report bugs for that package otherwise (and block our tracking bug)
check if the package works as expected

User Experience

See Benefit to Fedora above.

Dependencies

Changes are required in redhat-rpm-config along with the Python interpreters.

Contingency Plan

Contingency mechanism: Change owners can revert the change at any point.
Contingency deadline: final freeze (not a System Wide Change)
Blocks release? No

Documentation

N/A (not a System Wide Change)

Release Notes

@@ Line 1: / Line 1: @@
 = Further reduce Fedora-specific build flags in non-RPM Python extensions <!-- The name of your change proposal --> =
-{{Change_Proposal_Banner}}
 == Summary ==
 <!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
-Continuing the work started with https://fedoraproject.org/wiki/Changes/Python_Extension_Flags, this change is about further reducing the build and linker flags (CFLAGS, CXXFLAGS and LDFLAGS) saved internally in the Python interpreter for use by distutils and other build systems. Compiling non-RPM Python extension modules will carry only the compiler flags required for binary compatibility with the interpreter they were built against and not Fedora specific ones.
+Continuing the work started with https://fedoraproject.org/wiki/Changes/Python_Extension_Flags, this change is about further reducing the build and linker flags (CFLAGS and LDFLAGS) saved internally in the Python interpreter for use by distutils and other build systems. Compiling non-RPM Python extension modules will carry only the compiler flags required for binary compatibility with the interpreter they were built against and not Fedora specific ones.
-Practically that means the only Fedora derived flag will be <code>-fexceptions</code> and Python will apply its own upstream hardcoded ones, making the final flag set for a non-RPM compiled Python extension as follows:
+Practically that means the only Fedora derived flags will be <code>-fexceptions</code> and <code>-fcf-protection</code> and Python will apply its own upstream hardcoded ones, making the final flag set for a non-RPM compiled Python extension as follows:
-* <code>-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions</code>
+* <code>-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection</code>
+Python C extensions built as rpm's will '''not''' be affected.
 The current main Python interpreter on Fedora 39 will be modified (Python 3.12) and Python 3.6-3.11 will follow.
+This change will affect every package that provides support for extension builders via utilizing the <code>%{extension...flags}</code> macros which at the time being is only Python.
 == Owner ==
@@ Line 26: / Line 29: @@
 == Current status ==
-[[Category:ChangePageIncomplete]]
+[[Category:ChangeAcceptedF39]]
 <!-- When your change proposal page is completed and ready for review and announcement -->
 <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
@@ Line 43: / Line 46: @@
 ON_QA -> change is fully code complete
 -->
-* [<will be assigned by the Wrangler> devel thread]
+* [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/YYDLHZQCBGA6H4J7TOZIZUW6F4ERP5JS/ devel thread]
-* FESCo issue: <will be assigned by the Wrangler>
+* FESCo issue: [https://pagure.io/fesco/issue/3026 #3026]
-* Tracker bug: <will be assigned by the Wrangler>
+* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2229725 #2229725]
-* Release notes tracker: <will be assigned by the Wrangler>
+* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/1020 #1020]
 == Detailed Description ==
 After implementing https://fedoraproject.org/wiki/Changes/Python_Extension_Flags we uncoupled some distro specific compilation and linker flags propagated to C extensions.
-However with an ever increasing set of compiler flags being added and applied distro-wide, as compilers and security standards evolve (e.g. -D_FORTIFY_SOURCE=3) it becomes an increasingly complex job to vet each flag that might leak into user-built Python C extensions through the Python interpreter. Instead of removing only some flags and letting the rest follow through, we will be taking a more proactive approach by removing all the compiler and linker flags, except the ones that are required to maintain the binary compatibility with the Python interpreter the extensions were built against which is <code>-fexceptions</code>. We will also preserve the ones that Python hardcodes itself through the Makefile.
+However with an ever increasing set of compiler flags being added and applied distro-wide, as compilers and security standards evolve (e.g. -D_FORTIFY_SOURCE=3) it becomes an increasingly complex job to vet each flag that might leak into user-built Python C extensions through the Python interpreter. Instead of removing only some flags and letting the rest follow through, we will be taking a more proactive approach by removing all the compiler and linker flags, except the ones that are required to maintain the binary compatibility with the Python interpreter the extensions were built against which are <code>-fexceptions</code> and <code>-fcf-protection</code>. We will also preserve the ones that Python hardcodes itself through the Makefile.
 Similarly, when a user builds their own C programs, no compiler flags are applied by default and the user is free to making their own decision. Bringing the compilation of Python C extensions closer to that experience is the next logical step.
@@ Line 71: / Line 74: @@
 CFLAGS:
 <code>
--Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fexceptions -fexceptions
+-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection
 </code>
@@ Line 81: / Line 84: @@
 The initial thread that inspired this change was https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76/#76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76
+On the initial testing pull request [https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/252#comment-143315 feedback] was provided that the <code>-fcf-protection</code> flag should also be added to the minimal set, due to potential binary compatibility issues. Hence the changes page has been updated accordingly.
 == Benefit to Fedora ==
@@ Line 110: / Line 115: @@
       https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
 -->
+Python developers will get more upstream-like experience when building Python extension modules and also closer to building vanilla C programs. Also new decisions made about the distro-wide compiler flags won't necessarily affect Python developers building their extension modules.
+In addition any Python developer using Fedora will have the capability to build the extension on Fedora, test it and later ship it and build it on a CI or other systems that are not based on Fedora.
 == Scope ==
-* Proposal owners:
+* Proposal owners: Review, merge and build the [https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/252 redhat-rpm-config PR] and the apply the relevant changes in the Python interpreters ([https://src.fedoraproject.org/rpms/python3.6/pull-request/75 python3.6] [https://src.fedoraproject.org/rpms/python3.7/pull-request/83 python3.7] [https://src.fedoraproject.org/rpms/python3.8/pull-request/99 python3.8] [https://src.fedoraproject.org/rpms/python3.9/pull-request/149 python3.9] [https://src.fedoraproject.org/rpms/python3.10/pull-request/164 python3.10] [https://src.fedoraproject.org/rpms/python3.11/pull-request/121 python3.11] [https://src.fedoraproject.org/rpms/python3.12/pull-request/56 python3.12])
 <!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
-* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Other developers: No requirements apart from welcoming testing their C extensions<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
-* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Release engineering: No mass rebuild required and no releng impact anticipated.[https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
 The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
@@ Line 127: / Line 136: @@
 * Trademark approval: N/A (not needed for this Change)
 <!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
-* Alignment with Community Initiatives:
-<!-- Does your proposal align with the current Fedora Community Initiatives: https://docs.fedoraproject.org/en-US/project/initiatives/ ? It's okay if it doesn't, but it's something to consider -->
 == Upgrade/compatibility impact ==
@@ Line 135: / Line 141: @@
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.
 == How To Test ==
@@ Line 154: / Line 160: @@
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+=== For users (Python developers) ===
+# build your favorite Python extension module in venv or outside venv with your favorite build system
+# observe the used flags and check that the full set of flags are '''are not there''' as mentioned in the detailed description, report bugs for {{package|python3.12}} otherwise (and block our tracking bug)
+# check if the extension works as expected
+=== For packagers (Fedora contributors)  ===
+# build your favorite RPM package with Python extension module
+# observe the used flags and check that the full set of flags '''are there''' and not the reduced one, report bugs for that package otherwise (and block our tracking bug)
+# check if the package works as expected
 == User Experience ==
@@ Line 166: / Line 183: @@
   - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
 -->
+See '''Benefit to Fedora''' above.
 == Dependencies ==
@@ Line 171: / Line 189: @@
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+Changes are required in {{package|redhat-rpm-config}} along with the Python interpreters.
 == Contingency Plan ==
 <!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
-* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Contingency mechanism: Change owners can revert the change at any point.  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
-* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Contingency deadline: final freeze (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
-* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 == Documentation ==

Search

Changes/Python Extension Flags Reduction: Difference between revisions