From Fedora Project Wiki
No edit summary
(Add trackers)
 
(32 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->


= Prevent from building packages providing python3dist(...) = 0 <!-- The name of your change proposal --> =
= Prevent from building RPM packages providing python3dist(...) = 0 <!-- The name of your change proposal --> =


== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->


It sometimes happens that Python packages succeed to build with false version metadata.
It sometimes happens that Python packages succeed to build as RPM with incorrect version metadata.
They generate a wrong provide in format python3dist(...) = 0.
They generate a wrong provide in format `python3dist(...) = 0` and `python3.Xdist(...) = 0`.
While version 0 (or equal versions like 0.0 or 0.0.0) is probably technically valid, in most cases they indicates a packaging error.  
While version `0` (or equal versions like `0.0` or `0.0.0`) is probably technically valid, in most cases this indicates a packaging error.  
We propose to prevent this error from happening by explicitly erroring (and failing the build) when such provides was generated.
We propose to prevent this error from happening by explicitly failing the RPM build instead of generating such provides.


== Owner ==
== Owner ==
Line 16: Line 16:
This should link to your home wiki page so we know who you are.  
This should link to your home wiki page so we know who you are.  
-->
-->
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Ksurma|Karolina Surma]]
* Name: [[User:Ksurma|Karolina Surma]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: mhroncok@redhat.com, ksurma@redhat.com
* Email: ksurma@redhat.com
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
Line 24: Line 24:


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF38]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 42: Line 42:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/JMOBH5CIF4XNXSD2LEEF45OW2HYNUZMW/ devel thread]
* Tracker bug: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2903 #2903]
* Release notes tracker: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2150046 #2150046]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/929 #929]


== Detailed Description ==
== Detailed Description ==
Line 51: Line 52:
This change is about automatic RPM provides in the following form:
This change is about automatic RPM provides in the following form:


* `python3(.x)dist(distname) = 0`
* `python3dist(distname) = 0`
* `python3.Xdist(distname) = 0`


Where X is the Python minor version (eg. 10, 11...).
It does not affect any other provides.
It does not affect any other provides.
More about the provides: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Machine-readable-provides
More about these provides: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Machine-readable-provides


An example of the incorrect provides
The provides generated during the RPM build come from the upstream Python package metadata.
```
Some Python building backends, eg. setuptools, explicitly allow creating package with version `0.0.0` when the version used by a project is not known. This was [https://github.com/pypa/setuptools/issues/2329 discussed upstream] with conclusion that it's an intended behavior.
$ rpm -qpP python3-ssh-python-0.10.0-5.fc38.x86_64.rpm                                           
 
python-ssh-python = 0.10.0-5.fc38
In other cases, for example when building package from a particular git commit, the incorrect provide can be generated due to a packaging error.
python3-ssh-python = 0.10.0-5.fc38
We've never encountered a situation when packaging the version `0` was the package maintainers intention.
python3-ssh-python(x86-64) = 0.10.0-5.fc38
After a discussion on python-devel mailing list we agreed we'd like to prevent such situations from happening in Fedora.
python3.11-ssh-python = 0.10.0-5.fc38
 
python3.11dist(ssh-python) = 0
An example of the incorrect provides:
python3dist(ssh-python) = 0
 
```
$ rpm -qpP python3-ssh-python-0.10.0-5.fc38.x86_64.rpm                                           
python-ssh-python = 0.10.0-5.fc38
python3-ssh-python = 0.10.0-5.fc38
python3-ssh-python(x86-64) = 0.10.0-5.fc38
python3.11-ssh-python = 0.10.0-5.fc38
'''python3.11dist(ssh-python) = 0'''
'''python3dist(ssh-python) = 0'''
 
Why is it bad?
 
If any package requires `python3-ssh-python > 0.9` (correctly assuming there's `0.10.0` in Fedora's repositories), the automatic dependency generators will not discover the example package, making the other package non-installable.
 
In January 2022 the [https://bugzilla.redhat.com/showdependencytree.cgi?id=python3dist0 umbrella Bugzilla ticket] was created for Python packages providing version `0`. In all cases the automatic provide wasn't intended.
 
Affected packages (as of Nov 11 2022):
 
b43-tools
dionaea
marisa
python-podman-api
rpkg
spectrographic
python-ssh-python
python-streamlink


In January 2022 the umbrella Bugzilla ticket was created for Python packages providing this incorrect provide: https://bugzilla.redhat.com/showdependencytree.cgi?id=python3dist0
On Nov 10 2022 there are 22 linked Bugzilla tickets, 13 of which are not closed.
The change doesn't affect a big part of the Python ecosystem.
The change doesn't affect a big part of the Python ecosystem.


We aim to prevent such situation from happening by increasing the robustness of the python-rpm-generators (namely pythondistdeps.py: https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pythondistdeps.py).
We aim to prevent such situation from happening by increasing the robustness of the python-rpm-generators (namely [https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pythondistdeps.py pythondistdeps.py]).
The generator will error and fail the build if python3dist(pkgname) = 0 was to be generated.
The generator will error and fail the build if `python3dist(...) = 0` was to be generated.
 
Based on the discussion on fedora-devel mailing list, there will be a mechanism to opt out from this change.
A macro with a provisional name (TBD) `%__python_dist_allow_version_zero` will be created. If defined in the specfile, it will change the invocation of the Python RPM generators not to fail if a package providing version `0` was created. It will behave the same way as an already existing macro [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_python_no_extras_requires %_python_no_extras_requires] does.




Line 78: Line 105:
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->


The idea was posted on python-devel mailing list and received a positive feedback. No alternatives to this approach were proposed:
The idea was posted on [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/K35JCFVJLETVUOICQM634OSYBYQ3Q2WQ/ python-devel mailing list] and received a positive feedback. No alternatives to this approach were proposed.
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/K35JCFVJLETVUOICQM634OSYBYQ3Q2WQ/
 
After the Change was announced on fedora-devel, it was requested to provide a mechanism to opt out from the Change.
It was pointed out that we shouldn't dictate the upstream's decisions, especially that [https://peps.python.org/pep-0440/ PEP 440] doesn't forbid using the version `0.0.0`. OTOH [https://semver.org/ semver specification] recommends starting at `0.1.0`.
The Change was updated to provide such mechanism, see Detailed Description.
 


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 111: Line 142:


The correct metadata is essential for the whole package ecosystem. More deterministic behavior of the generators will bring those benefits:
The correct metadata is essential for the whole package ecosystem. More deterministic behavior of the generators will bring those benefits:
The packages will stop lying about the version they provide.
* The packages will stop lying about the version they provide.
The requirements generators (eg. `%pyproject_buildrequires`) will correctly evaluate the Build- and Runtime Requirements based on the correct Provides.
* The requirements generators (eg. `%pyproject_buildrequires`) will correctly evaluate the Build- and Runtime Requirements based on the correct Provides.
The package maintainers who BuildRequire %{py3dist pkgname} in their specfiles will always require the correctly evaluated version.
* The package maintainers who BuildRequire `%{py3dist pkgname} >= 0.2` in their specfiles will always require the correctly evaluated version.


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
# implement & test the change in python-rpm-generators (pythondistdeps.py)
# implement & test the change in python-rpm-generators ([https://src.fedoraproject.org/rpms/python-rpm-generators/blob/rawhide/f/pythondistdeps.py pythondistdeps.py])
# ...
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
** fix the packaging error to prevent from generating such metadata
** Fix the packaging error to generate correct metadata and successful build. There's no common way to deal with such packages. In most of the cases the issue originates from packaging errors that need to be fixed. Contact the change owners if you need help with the necessary changes to your package.
TBD how
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


Line 130: Line 159:
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->


* Policies and guidelines: not needed for this Change <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_source_files_from_pypi Python Packaging Guidelines] cover the topic of creating the version string. This will be valid even when we change the Python RPM generators behavior <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->


Line 144: Line 173:
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
None.
None.


== How To Test ==
== How To Test ==
Line 162: Line 190:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
TBD
* Prepare a Python package, set the version to `0`
or
* Use one of the known packages that provide version `0`.
* Try to build an RPM package in a regular way (eg. mockbuild)
* The build should fail.


== User Experience ==
== User Experience ==
Line 181: Line 213:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
TBD
None


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: TBD <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Revert <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: TBD <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: Beta freeze <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->


== Documentation ==
== Documentation ==
Line 197: Line 228:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
This page is the documentation of this change.


== Release Notes ==
== Release Notes ==

Latest revision as of 18:06, 1 December 2022


Prevent from building RPM packages providing python3dist(...) = 0

Summary

It sometimes happens that Python packages succeed to build as RPM with incorrect version metadata. They generate a wrong provide in format python3dist(...) = 0 and python3.Xdist(...) = 0. While version 0 (or equal versions like 0.0 or 0.0.0) is probably technically valid, in most cases this indicates a packaging error. We propose to prevent this error from happening by explicitly failing the RPM build instead of generating such provides.

Owner

Current status

Detailed Description

This change is about automatic RPM provides in the following form:

  • python3dist(distname) = 0
  • python3.Xdist(distname) = 0

Where X is the Python minor version (eg. 10, 11...). It does not affect any other provides. More about these provides: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Machine-readable-provides

The provides generated during the RPM build come from the upstream Python package metadata. Some Python building backends, eg. setuptools, explicitly allow creating package with version 0.0.0 when the version used by a project is not known. This was discussed upstream with conclusion that it's an intended behavior.

In other cases, for example when building package from a particular git commit, the incorrect provide can be generated due to a packaging error. We've never encountered a situation when packaging the version 0 was the package maintainers intention. After a discussion on python-devel mailing list we agreed we'd like to prevent such situations from happening in Fedora.

An example of the incorrect provides:

$ rpm -qpP python3-ssh-python-0.10.0-5.fc38.x86_64.rpm                                           
python-ssh-python = 0.10.0-5.fc38
python3-ssh-python = 0.10.0-5.fc38
python3-ssh-python(x86-64) = 0.10.0-5.fc38
python3.11-ssh-python = 0.10.0-5.fc38
python3.11dist(ssh-python) = 0
python3dist(ssh-python) = 0

Why is it bad?

If any package requires python3-ssh-python > 0.9 (correctly assuming there's 0.10.0 in Fedora's repositories), the automatic dependency generators will not discover the example package, making the other package non-installable.

In January 2022 the umbrella Bugzilla ticket was created for Python packages providing version 0. In all cases the automatic provide wasn't intended.

Affected packages (as of Nov 11 2022):

b43-tools
dionaea
marisa
python-podman-api
rpkg
spectrographic
python-ssh-python
python-streamlink

The change doesn't affect a big part of the Python ecosystem.

We aim to prevent such situation from happening by increasing the robustness of the python-rpm-generators (namely pythondistdeps.py). The generator will error and fail the build if python3dist(...) = 0 was to be generated.

Based on the discussion on fedora-devel mailing list, there will be a mechanism to opt out from this change. A macro with a provisional name (TBD) %__python_dist_allow_version_zero will be created. If defined in the specfile, it will change the invocation of the Python RPM generators not to fail if a package providing version 0 was created. It will behave the same way as an already existing macro %_python_no_extras_requires does.


Feedback

The idea was posted on python-devel mailing list and received a positive feedback. No alternatives to this approach were proposed.

After the Change was announced on fedora-devel, it was requested to provide a mechanism to opt out from the Change. It was pointed out that we shouldn't dictate the upstream's decisions, especially that PEP 440 doesn't forbid using the version 0.0.0. OTOH semver specification recommends starting at 0.1.0. The Change was updated to provide such mechanism, see Detailed Description.


Benefit to Fedora

The correct metadata is essential for the whole package ecosystem. More deterministic behavior of the generators will bring those benefits:

  • The packages will stop lying about the version they provide.
  • The requirements generators (eg. %pyproject_buildrequires) will correctly evaluate the Build- and Runtime Requirements based on the correct Provides.
  • The package maintainers who BuildRequire %{py3dist pkgname} >= 0.2 in their specfiles will always require the correctly evaluated version.

Scope

  • Proposal owners:
  1. implement & test the change in python-rpm-generators (pythondistdeps.py)
  • Other developers:
    • Fix the packaging error to generate correct metadata and successful build. There's no common way to deal with such packages. In most of the cases the issue originates from packaging errors that need to be fixed. Contact the change owners if you need help with the necessary changes to your package.
  • Release engineering: not needed for this Change
  • Policies and guidelines: Python Packaging Guidelines cover the topic of creating the version string. This will be valid even when we change the Python RPM generators behavior
  • Trademark approval: not needed for this Change
  • Alignment with Objectives: No

Upgrade/compatibility impact

None.

How To Test

  • Prepare a Python package, set the version to 0

or

  • Use one of the known packages that provide version 0.
  • Try to build an RPM package in a regular way (eg. mockbuild)
  • The build should fail.

User Experience

The actual users should notice no difference.

Dependencies

None

Contingency Plan

  • Contingency mechanism: Revert
  • Contingency deadline: Beta freeze
  • Blocks release? No

Documentation

This page is the documentation of this change.

Release Notes