From Fedora Project Wiki

(Fix broken links to mediawiki syntax)
No edit summary
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Python Eggs =
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}


Python packages have started to use setuptools in their package build scripts.  Packages which do this provide extra metadata about the package in the form of eggs.  This document explains how to package eggs.
{{DISPLAYTITLE:Guidelines for Python Egg packaging}}
<div style="float: right; margin-left: 0.5em" class="toclimit-2">__TOC__</div>


Python packages provide extra metadata about the package in the form of egg metadata.  This document explains how to package those metadata.


== Why Eggs ==
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}


== Why Eggs ==
Egg metadata have several uses including:
Eggs have several uses including:
<ol>
1. Allowing end users to install eggs not made from rpms or install eggs into their home directories.  This is an important feature for people working within a shared hosting environment.
<li>Allowing end users to install egg packages not made from rpms or install egg packages into their home directories.  This is an important feature for people working within a shared hosting environment.
1. Giving python packages an easy way to support plugins.
<li>Giving python packages an easy way to support plugins.
1. Giving us a way to support multiple versions of a python module for compat libraries.
<li>Giving us a way to support multiple versions of a python module for compat libraries.
</ol>


== What are Eggs ==
The egg metadata can be used at runtime so they cannot be replaced with the rpm database which is only useful at install time or by tools specifically for Fedora.
Eggs can be placed on disk in several formats:
* As a module and a file with a .egg-info extension that contains the metadata.  Created by distutils in Fedora 9's python2.5.
* As a module and a directory with a .egg-info extension that contains the metadata.  Created using the most common invocation of setup.py in our examples below.
* As a directory with a .egg extension that contains the module and egg metadata.  Created when we use easy_install -m to allow installing multiple versions of a module.
* As a single zip file with a .egg extension that contains the module and the egg metadata.


In Fedora Packages, these will be installed to %{python_sitelib} or %{python_sitearch} directories.
== When to Provide Egg Metadata ==
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}


{{Anchor|WhenEggs}}
When upstream uses setuptools to provide egg metadata it will be automatically built and installed when you use the %py{2,3}_build and %py{2,3}_install macros.
== When to Provide Eggs ==
Since eggs establish a base of functionality that upstream authors can expect, we need to be sure to include the egg files if a package builds them.  Starting with Fedora 9 any package that uses setuptools or distutils will build egg-info.  In Fedora 8 or less, only setuptools packages build egg-info.  If you need to provide egg-info for a distutils package on Fedora 8 or less, [[#Providing_Eggs_for_non-setuptools_packages|  Providing Eggs using Setuptools]]  describes a method of substituting setuptools for distutils in the build process so egg-info is created.


== Upstream Egg Packages ==
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}


{{admon/note| In the past, when there was a requirement for an egg which was not provided by upstream we would patch the requiring package to not require that package.  This behaviour is deprecated and as packages are updated maintainers should follow the below guidelines to install eggs for the required packages. Please see [[#Providing_Eggs_for_non-setuptools_packages|  Creating Eggs for Non-setuptools Packages]]
Do not distribute egg packages from upstream.  In Fedora, all packages must be
}}
rebuilt from source.  An egg package (which is different from egg ''metadata'')
contains compiled bytecode and may, if it contains a C extension, contain
compiled binary extensions as well.  These are opaque structures with no
guarantee that they were even built from the source distributed with the egg.
If you must use an egg package from upstream because they do not provide tarballs, you
need to include it as a source in your spec, unzip it in %setup, and rebuild
from the source files contained within it.


== Upstream Eggs ==
== Providing Egg Metadata Using Setuptools ==
Do not distribute eggs from upstream.  In Fedora, all packages must be rebuilt from source. An egg package contains compiled bytecode and may, if it contains a C extension, contain compiled binary extensions as well. These are opaque structures with no guarantee that they were even built from the source distributed with the egg. If you must use an egg from upstream because they do not provide tarballs, you need to include it as a source in your spec, unzip it in %setup, and rebuild from the source files contained within it.
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}


== Providing Eggs using Setuptools ==
When upstream uses setuptools to provide egg metadata it is very simple to include them in your package.  Your spec file will look something like this:
When upstream uses setuptools to provide eggs it is very simple to include them in your package.  Your spec file will look something like this:


<pre>
<pre>
BuildRequires: python-setuptools-devel
# Must have setuptools to build the package
BuildRequires: python2-setuptools


[...]  
[...]


%install
%install
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
%py2_install


[...]  
[...]


%files
%files
[...]  
[...]
%{python_sitelib}/*
# This captures both the module directory and the egg-info directory
# Something like:
# /usr/lib/python2.7/site-packages/sqlalchemy
# /usr/lib/python2.7/site-packages/SQLAlchemy-0.3.10-py2.7.egg-info
%{python2_sitelib}/sqlalchemy/
%{python2_sitelib}/*egg-info/
</pre>
</pre>
{{ /code/|  commandline argument to create egg info and an expanded directory directly in site-packages.  This is no longer necessary as --root creates things the way we want for packaging.
}}
{{Anchor|NonSetuptoolsEggs}}
== Providing Eggs for non-setuptools packages ==
{{ Template:note/| These instructions are only for distutils in Fedora <= 8.  Fedora 9 and above will automatically generate egg-info files.
}}
When we need to provide eggs in a non-setuptools package because another package requires that functionality we can modify our spec files to generate the egg-info:
<pre>
BuildRequires: python-setuptools-devel
[...]
%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} -c 'import setuptools; execfile("setup.py")' build
%install
rm -rf $RPM_BUILD_ROOT
%{__python} -c 'import setuptools; execfile("setup.py")' install --skip-build --root $RPM_BUILD_ROOT
%files
%{python_sitelib}/*egg-info
%{python_sitelib}/[MODULENAME]
</pre>
By importing setuptools before executing setup.py we override the distutils functions that process the file with their setuptools equivalents.  Those functions create the egg-info files.


== Multiple Versions ==
== Multiple Versions ==
 
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. Package multiple conflicting versions of the package instead.}}
{{ /code/| section
}}


Sometimes we want to keep an old version of a module around for compatibility.  When upstream has renamed the module for us, this is a straightforward creation of a new module.  For instance, python-psycopg and python-psycopg2.
Sometimes we want to keep an old version of a module around for compatibility.  When upstream has renamed the module for us, this is a straightforward creation of a new module.  For instance, python-psycopg and python-psycopg2.


When upstream doesn't include the version in the name, we have to find another way to parallel install two versions of the package.  Eggs give us this ability.  The latest version of a package must be installed as the python-MODULENAME and is built using the normal guidelines.  The compatibility versions of the module should be named python-$MODULENAME$DISTINGUISHINGVER and be enabled by making these spec file changes:
When upstream doesn't include the version in the name, we have to find another way to parallel install two versions of the package.  Egg metadata give us this ability.  The latest version of a package must be installed as the python-MODULENAME and is built using the normal guidelines.  The compatibility versions of the module should be named python-$MODULENAME$DISTINGUISHINGVER and be enabled by making these spec file changes:


<pre>
<pre>
Requires: python-setuptools
# Require setuptools as the consumer will need pkg_resources to use this module
Requires: python2-setuptools


%build
%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py bdist_egg
# Build an egg file that we can then install from
%py2_build_egg


%install
%install
rm -rf $RPM_BUILD_ROOT
# Install the egg so that only code that wants this particular version can get it.
mkdir -p $RPM_BUILD_ROOT%{python_sitelib}
%py2_install_egg
easy_install -m --prefix $RPM_BUILD_ROOT%{_usr} dist/*.egg
</pre>
</pre>


This creates the python egg under the %{python_sitelib}/*.egg directory.  This module is not directly usable via the import statement.  Instead, the consuming package must setup the PYTHONPATH to reference the compat version before it imports the module.  This can be done in a variety of ways.
This creates the python egg under the %{python2_sitelib}/*.egg directory.  This module is not directly usable via the import statement.  Instead, the consuming package must setup the PYTHONPATH to reference the compat version before it imports the module.  This can be done in a variety of ways.
* Manually modifying sys.path is quick if the user just wants to try out some code with the old version:
* Manually modifying sys.path is quick if the user just wants to try out some code with the old version:
<pre>
<pre>
>>> import sys
>>> import sys
>>> sys.path.insert(0, '/usr/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg/')
>>> sys.path.insert(0, '/usr/lib/python2.7/site-packages/CherryPy-2.2.1-py2.7.egg/')
>>> import cherrypy
>>> import cherrypy
</pre>
</pre>
* Using setuptools and easy_install to create "script wrappers" to invoke the programs.  Setuptools has you define an entrypoint in the program's module (basically, a main() function) and then writes a script to access that via an option in setup.py.
* Using setuptools and easy_install to create "script wrappers" to invoke the programs.  Setuptools has you define an entrypoint in the program's module (basically, a main() function) and then writes a script to access that via an option in setup.py.


It is highly recommended that any such compatibility packages install a README.fedora file explaining how to use this module.  The file should contain the above examples of how to call the module from code and explain that this is a compat package and that a newer version exists.  Here's an [[Image:Packaging_Python_Eggs_README.fedora]] example README.fedora]  to look at for ideas.
It is highly recommended that any such compatibility packages install a README.fedora file explaining how to use this module.  The file should contain the above examples of how to call the module from code and explain that this is a compat package and that a newer version exists.


There are several other methods of invoking scripts so that they might take the right version but they suffer from various problems.  They are listed here because a program you're packaging may use them and you need to know about them if they break.  If you mention them in README.fedora, please also add why they are dangerous to use.
There are several other methods of invoking scripts so that they might take the right version but they suffer from various problems.  They are listed here because a program you're packaging may use them and you need to know about them if they break.  If you mention them in README.fedora, please also add why they are dangerous to use.
Line 116: Line 98:


== Egg "Features" to avoid ==
== Egg "Features" to avoid ==
Eggs provide some features that are to be avoided as part of the packaging process for Fedora.  Some of these may provide benefit to our users but should not be used when creating system packages.
{{admon/important|Old page|This page is very very old. The information on this page is '''not accurate'''. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.}}
 
Egg metadata provide some features that are to be avoided as part of the packaging process for Fedora.  Some of these may provide benefit to our users but should not be used when creating system packages.


* Do not let easy_install download and install packages from the net to add to the build root.  This will fail on the build system as well as being bad packaging.  Packages which are downloaded are probably missing from your BuildRequires.
* Do not let easy_install download and install packages from the net to add to the build root.  This will fail on the build system as well as being bad packaging.  Packages which are downloaded are probably missing from your BuildRequires.
Line 122: Line 106:
== Links ==
== Links ==


* https://src.fedoraproject.org/rpms/pyproject-rpm-macros
* http://peak.telecommunity.com/DevCenter/PythonEggs
* http://peak.telecommunity.com/DevCenter/PythonEggs
* http://peak.telecommunity.com/DevCenter/setuptools
* http://peak.telecommunity.com/DevCenter/setuptools
* http://lists.debian.org/debian-python/2007/09/msg00004.html -- Discussion of eggs in Debian
* http://lists.debian.org/debian-python/2007/09/msg00004.html -- Discussion of eggs in Debian
* http://mail.python.org/pipermail/distutils-sig/2007-September/008181.html -- Discussion of these guidelines on the distutils list
* http://mail.python.org/pipermail/distutils-sig/2007-September/008181.html -- Discussion of these guidelines on the distutils list
[[Category:Packaging guidelines]]
[[Category:Python]]

Revision as of 11:31, 26 March 2021

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.


Python packages provide extra metadata about the package in the form of egg metadata. This document explains how to package those metadata.

Why Eggs

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.

Egg metadata have several uses including:

  1. Allowing end users to install egg packages not made from rpms or install egg packages into their home directories. This is an important feature for people working within a shared hosting environment.
  2. Giving python packages an easy way to support plugins.
  3. Giving us a way to support multiple versions of a python module for compat libraries.

The egg metadata can be used at runtime so they cannot be replaced with the rpm database which is only useful at install time or by tools specifically for Fedora.

When to Provide Egg Metadata

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.

When upstream uses setuptools to provide egg metadata it will be automatically built and installed when you use the %py{2,3}_build and %py{2,3}_install macros.

Upstream Egg Packages

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.

Do not distribute egg packages from upstream. In Fedora, all packages must be rebuilt from source. An egg package (which is different from egg metadata) contains compiled bytecode and may, if it contains a C extension, contain compiled binary extensions as well. These are opaque structures with no guarantee that they were even built from the source distributed with the egg. If you must use an egg package from upstream because they do not provide tarballs, you need to include it as a source in your spec, unzip it in %setup, and rebuild from the source files contained within it.

Providing Egg Metadata Using Setuptools

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.

When upstream uses setuptools to provide egg metadata it is very simple to include them in your package. Your spec file will look something like this:

# Must have setuptools to build the package
BuildRequires: python2-setuptools

[...]

%install
%py2_install

[...]

%files
[...]
# This captures both the module directory and the egg-info directory
# Something like:
# /usr/lib/python2.7/site-packages/sqlalchemy
# /usr/lib/python2.7/site-packages/SQLAlchemy-0.3.10-py2.7.egg-info
%{python2_sitelib}/sqlalchemy/
%{python2_sitelib}/*egg-info/

Multiple Versions

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. Package multiple conflicting versions of the package instead.

Sometimes we want to keep an old version of a module around for compatibility. When upstream has renamed the module for us, this is a straightforward creation of a new module. For instance, python-psycopg and python-psycopg2.

When upstream doesn't include the version in the name, we have to find another way to parallel install two versions of the package. Egg metadata give us this ability. The latest version of a package must be installed as the python-MODULENAME and is built using the normal guidelines. The compatibility versions of the module should be named python-$MODULENAME$DISTINGUISHINGVER and be enabled by making these spec file changes:

# Require setuptools as the consumer will need pkg_resources to use this module
Requires: python2-setuptools

%build
# Build an egg file that we can then install from
%py2_build_egg

%install
# Install the egg so that only code that wants this particular version can get it.
%py2_install_egg

This creates the python egg under the %{python2_sitelib}/*.egg directory. This module is not directly usable via the import statement. Instead, the consuming package must setup the PYTHONPATH to reference the compat version before it imports the module. This can be done in a variety of ways.

  • Manually modifying sys.path is quick if the user just wants to try out some code with the old version:
>>> import sys
>>> sys.path.insert(0, '/usr/lib/python2.7/site-packages/CherryPy-2.2.1-py2.7.egg/')
>>> import cherrypy
  • Using setuptools and easy_install to create "script wrappers" to invoke the programs. Setuptools has you define an entrypoint in the program's module (basically, a main() function) and then writes a script to access that via an option in setup.py.

It is highly recommended that any such compatibility packages install a README.fedora file explaining how to use this module. The file should contain the above examples of how to call the module from code and explain that this is a compat package and that a newer version exists.

There are several other methods of invoking scripts so that they might take the right version but they suffer from various problems. They are listed here because a program you're packaging may use them and you need to know about them if they break. If you mention them in README.fedora, please also add why they are dangerous to use.

  • pkg_resources.requires('MODULE[VERSIONINFO] '): Does not work with a default version (able to be imported via import MODULE). The setuptools author refuses to remove this limitation and refuses to document that it is a limitation. Therefore you may run across scripts that use this method and need to patch them to use one of the above, supported methods instead.
  • __requires__='MODULE[VERSIONINFO] ': This works but the setuptools author feels that it is only a workaround and will not support it. It works presently but could stop in a future version of setuptools. Some upstreams use this method and may need to be fixed if the setuptools author ever changes the interface.

Egg "Features" to avoid

Important.png
Old page
This page is very very old. The information on this page is not accurate. Don't package eggs. See https://src.fedoraproject.org/rpms/pyproject-rpm-macros for packaging wheels instead.

Egg metadata provide some features that are to be avoided as part of the packaging process for Fedora. Some of these may provide benefit to our users but should not be used when creating system packages.

  • Do not let easy_install download and install packages from the net to add to the build root. This will fail on the build system as well as being bad packaging. Packages which are downloaded are probably missing from your BuildRequires.

Links