From Fedora Project Wiki
(Draft revision to Packaging:SourceURL)
 
m (Remove $ and replace PROJECT with %{name} per FPC)
 
(46 intermediate revisions by 2 users not shown)
Line 16: Line 16:
{{admon/tip|Smallest Compressed Archive|If the upstream source archive is available in multiple compressed formats that our tools can decompress it's best to use the one that is smallest in size. This ensures the smallest source rpm to save space on the mirrors and downloads of source RPM packages.}}
{{admon/tip|Smallest Compressed Archive|If the upstream source archive is available in multiple compressed formats that our tools can decompress it's best to use the one that is smallest in size. This ensures the smallest source rpm to save space on the mirrors and downloads of source RPM packages.}}


There are several cases where upstream is not providing the source to you in an upstream tarball.  In these cases you must document how to generate the tarball used in the rpm either through a spec file comment or a script included as a separate SourceX:.
There are several cases where upstream is not providing the source to you in an upstream tarball.  In these cases you must document how to generate the tarball used in the rpm either through a Spec file comment or a script included as a separate SourceX:.


Here are some specific examples:
Here are some specific examples:
Line 34: Line 34:


When pulling from revision control, please remember to use a Name-version-release compatible with the [[Packaging/NamingGuidelines#PackageVersion|Version]] and
When pulling from revision control, please remember to use a Name-version-release compatible with the [[Packaging/NamingGuidelines#PackageVersion|Version]] and
[[Packaging/NamingGuidelines#PackageRelease|Release]] Guidelines.  In particular, check the section on [[Packaging/NamingGuidelines#SnapshotPackages|Naming Snapshots]] .
[[Packaging/NamingGuidelines#PackageRelease|Release]] Guidelines.  In particular, check the section on [[Packaging/NamingGuidelines#SnapshotPackages|Naming Snapshots]].


{{Anchor|ProhibitedCode}}
{{Anchor|ProhibitedCode}}
Line 42: Line 42:
Some upstream packages include patents or trademarks that we are not allowed to ship even as source code.  In these cases you have to modify the source tarball to remove this code before you even upload it to the build system.  Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:
Some upstream packages include patents or trademarks that we are not allowed to ship even as source code.  In these cases you have to modify the source tarball to remove this code before you even upload it to the build system.  Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:


From the spec:
From the Spec:
<pre>
<pre>
Source0: libfoo-1.0-nopatents.tar.gz
Source0: libfoo-1.0-nopatents.tar.gz
Line 79: Line 79:
Please note that the correct url is <code>downloads.sourceforge.net</code>, and '''NOT''' <code>download.sourceforge.net</code>.
Please note that the correct url is <code>downloads.sourceforge.net</code>, and '''NOT''' <code>download.sourceforge.net</code>.


== Github ==
{{Anchor|GitHostingServices}}


As many upstreams use github for their source control, it is worth covering how to handle that source in a Fedora Package.
== Git Hosting Services ==


Github provides a mechanism to create tarballs on demand, either from a specific commit revision, or from a specific tag. If the upstream does not create tarballs for releases, you can use this mechanism to produce them. If the upstream '''does''' create tarballs you should use them as tarballs provide an easier trail for people auditing the packages.
If the upstream '''does''' create tarballs you should use them as tarballs provide an easier trail for people auditing the packages.


For a number of reasons (immutability, availability, uniqueness), you must use the full commit revision hash when referring to the sources.
Git web-based hosting services provide a mechanism to create tarballs on demand, either from a specific commit revision, or from a specific tag.  If the upstream does not create tarballs for releases, you can use this mechanism to produce them.


The full 40-character hash can be copied from the github web interface at ​https://github.com/$OWNER/$PROJECT/tags or by cloning the repository and using
The full 40-character hash and associated Git Tag may be obtained by issuing the following git command:
<code>git rev-parse $TAG</code>


In this example, $TAG is the tag for the source revision we are interested in, $OWNER must be replaced with the github username for the project's owner, and
<span class="plainlinks">[//git-scm.com/docs/git-ls-remote git ls-remote]</span> <nowiki>https://HOSTING-SERVICE/OWNER/%{name}.git</nowiki>
$PROJECT must be replaced with the github identifier for the project.


Once the commit hash is known, you can define it in your spec file as follows:
<pre>
HOSTING-SERVICE:  name of the service, i.e. "github.com", "bitbucket.org", "gitlab.com", etc.
OWNER:            username for the repository owner
</pre>
 
You may also obtain the 40-character hash and associated Git Tag via the web-interface of the HOSTING-SERVICE, or by cloning the repository and issuing the <span class="plainlinks">[//git-scm.com/docs/git-show-ref git show-ref]</span> command.
 
Once the commit hash and Git Tag are known, you can define them in your Spec file as follows:


<pre>
<pre>
%global commit c5a4525bfa3bd9997834d0603c40093e50e3fd19
%global commit0 40-CHARACTER-HASH-VALUE
%global shortcommit %(c=%{commit}; echo ${c:0:7})
%global gittag0 GIT-TAG
%global shortcommit0 %(c=%{commit}; echo ${c:0:7})    [GitHub]
%global shortcommit0 %(c=%{commit}; echo ${c:0:11})  [Bitbucket]
%global shortcommit0 %(c=%{commit}; echo ${c:0:7})   [GitLab]
</pre> 
                                                       
Note that the numeric identifier after commit, gittag and shortcommit matches the associated identifier of the Source Tag.
 
=== Commit Revision ===
 
For the source tarball, you can use the following syntax:
<pre>
Source0:  https://github.com/OWNER/%{name}/archive/%{commit0}.tar.gz#/%{name}-%{shortcommit0}.tar.gz
Source0:  https://bitbucket.org/OWNER/%{name}/get/%{commit0}.tar.gz#/%{name}-%{shortcommit0}.tar.gz
Source0:  https://gitlab.com/OWNER/%{name}/repository/archive.tar.gz?ref=%{commit0}#/%{name}-%{shortcommit0}.tar.gz
...
 
%prep
%setup -qn %{name}-%{commit0}              [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0}  [BitBucket]
%setup -qn %{name}.git                    [GitLab]
</pre>
</pre>


For the source tarball, you should use this syntax:
If the release corresponds to a Git Tag with a sane numeric version, you must use that version to populate the Version Tag in the Spec file. If it does not, look at the source code to see if a version is indicated there, and use that value. If no numeric version is indicated in the code, you may set Version to 0, and treat the package as a "pre-release" package (and make use of the %{shortcommit0} macro). See [[Packaging:NamingGuidelines#Pre-Release_packages|Pre-Release packages]] for details.
Alternately, if you are using a specific revision that is either a pre-release revision or a post-release revision, you must follow the "snapshot" guidelines. They are documented here: [[Packaging:NamingGuidelines#Snapshot_packages|Snapshot packages]]. You can substitute %{shortcommit0} for %{checkout} in that section.
 
=== Git Tags ===
 
<span class="plainlinks">[//git-scm.com/docs/git-tag Git Tags]</span> represent a particular code point that upstream deems important; and are typically used to mark release points.
 
Bitbucket uses the %{shortcommit0} identifier as part of the archive directory structure; regardless of whether you use Git Tag or Commit Revision to retrieve it.  This is shown in the %prep section example.
 
For the source tarball, you can use the following syntax:
<pre>
<pre>
Source0:       https://github.com/$OWNER/$PROJECT/archive/%{commit}/$PROJECT-%{commit}.tar.gz
Source0: https://github.com/OWNER/%{name}/archive/GIT-TAG.tar.gz#/%{name}-%{version}.tar.gz
 
Source0:  https://bitbucket.org/OWNER/%{name}/get/GIT-TAG.tar.gz#/%{name}-%{version}.tar.gz
Source0:  https://gitlab.com/OWNER/%{name}/repository/archive.tar.gz?ref=GIT-TAG#/%{name}-%{version}.tar.gz
...
...


%prep
%prep
%setup -qn %{name}-%{commit}
%setup -qn %{name}-%{version}               [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0}    [BitBucket]
%setup -qn %{name}.git                      [GitLab]
</pre>
</pre>


Remember, in this syntax, $OWNER must be replaced with the github username for the project's owner, and $PROJECT must be replaced with the github identifier for the project.
=== Git Submodules ===
 
Some projects may use the <span class="plainlinks">[//git-scm.com/docs/git-submodule Git submodule]</span> capability.  If so, please note that the submodule will not be included in the tarball.  Instead, it will appear in the tarball as an empty directory.  In this situation, you will need to manually incorporate the submodule into the project.


If the release corresponds to a github Tag with a sane numeric version, you must use that version to populate the Version field in the spec file. If it does not, look at the source code to see if a version is indicated there, and use that value. If no numeric version is indicated in the code, you may set Version to 0, and treat the package as a "pre-release" package (and make use of the %{shortcommit} macro). See [[Packaging:NamingGuidelines#Pre-Release_packages]] for details.
The simplest method is to incorporate the submodule within the %prep section of the Spec file.  In this instance you will follow the preceding instructions to obtain the tarball for the submodule project. Instead of Source0, commit0, gittag0 and shortcommit0; you will use Source1, commit1, gittag1 and shortcommit1.
An example %prep section (assuming the use of Commit Revision) would be:


Alternately, if you are using a specific revision from github that is either a pre-release revision or a post-release revision, you must follow the "snapshot" guidelines. They are documented here: [[Packaging:NamingGuidelines#Snapshot_packages]]. You can substitute %{shortcommit} for %{checkout} in that section.
<pre>
%prep
%setup -qn %{name}-%{commit0} -a 1              [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0} -a 1    [BitBucket]
%setup -qn %{name}.git -a 1                      [GitLab]
rmdir SUBMODULE                                  [Remove SUBMODULE]
mv -f %{name}-%{commit1} SUBMODULE              [Move SUBMODULE TO %{name}]
</pre> 
If for some reason that isn't feasible, an alternative is to rebuild the tarball prior to it's inclusion in the Spec file. This involves cloning the project, issuing the Git commands to incorporate the submodule, then building a tarball of the newly created project directory structure.  You will then need to comment as described in the [[#RevisionControl|Using Revision Control]] section.  The following is an example specific to Git submodules:
 
<pre>
# The source of this package was pulled from upstreams's vcs.
# Use the following command to generate the tar ball:
# git clone http://HOSTING-SERVICE/OWNER/%{name}.git
# cd %{name}
# git submodule init
# git submodule update
# cd ..
# tar cvjf %{name}-%{version}.tar.gz %{name}/
</pre>


Keep in mind that github tarballs are generated on-demand, so their modification dates will vary and cause checksum tests to fail. Reviewers will need to use diff -r to verify the tarballs.
Keep in mind that the version specified reflects that of the project and '''NOT''' the submodule.


{{Anchor|VersionMacro}}
{{Anchor|VersionMacro}}
Line 122: Line 183:
== Using %{version} ==
== Using %{version} ==


Using %{version} in the SourceX: makes it easier for you to bump the version of a package, because most of the time you do not need to edit SourceX: when editing the specfile for the new package.
Using %{version} in the SourceX: makes it easier for you to bump the version of a package, because most of the time you do not need to edit SourceX: when editing the Spec file for the new package.


== Troublesome URLs ==
== Troublesome URLs ==
Line 134: Line 195:
rpm will then use %{name}-%{version}.tar.gz as the tarball name.  If you use <code>spectool -g foo.spec</code> to download the tarball, it will rename the tarball for you.
rpm will then use %{name}-%{version}.tar.gz as the tarball name.  If you use <code>spectool -g foo.spec</code> to download the tarball, it will rename the tarball for you.


Sometimes this does not work because the upstream cgi tries to parse the fragment or because you need to login or fill in a form to access the tarball.  In these cases, you have to put just the tarball's filename into the Source: field. To make clear where you got the tarball, you should leave notes in comments above the Source: line to explain the situation to reviewers and future packagers.  For example:
Sometimes this does not work because the upstream cgi tries to parse the fragment or because you need to login or fill in a form to access the tarball.  In these cases, you have to put just the tarball's filename into the Source: Tag. To make clear where you got the tarball, you should leave notes in comments above the Source: line to explain the situation to reviewers and future packagers.  For example:


<pre>
<pre>

Latest revision as of 17:43, 2 July 2015

Referencing Source

One of the design goals of rpm is to cleanly separate upstream source from vendor modifications. For the Fedora packager, this means that sources used to build a package should be the vanilla sources available from upstream. To help reviewers and QA scripts verify this, the packager needs to indicate where a reviewer can find the source that was used to make the rpm.

The most common case is where upstream distributes source as a tar.gz, tar.bz2 or zip archive that we can download from an upstream website. In these cases you must use a full URL to the package in the SourceX: line. For example:

Source0: http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz

Source0: http://ftp.gnome.org/pub/GNOME/sources/gnome-common/2.12/gnome-common-2.12.0.tar.bz2
Idea.png
Smallest Compressed Archive
If the upstream source archive is available in multiple compressed formats that our tools can decompress it's best to use the one that is smallest in size. This ensures the smallest source rpm to save space on the mirrors and downloads of source RPM packages.

There are several cases where upstream is not providing the source to you in an upstream tarball. In these cases you must document how to generate the tarball used in the rpm either through a Spec file comment or a script included as a separate SourceX:.

Here are some specific examples:

Using Revision Control

In some cases you may want to pull sources from upstream's revision control system because there have been many changes since the last release and you think that a tarball that you generate from there will more accurately show how the package relates to upstream's development. Here's how you can use a comment to show where the source came from:

# The source for this package was pulled from upstream's vcs.  Use the
# following commands to generate the tarball:
#  svn export -r 250 http://www.example.com/svn/foo/trunk foo-20070221
#  tar -cJvf foo-20070221.tar.xz foo-20070221
Source0: foo-20070221.tar.xz

When pulling from revision control, please remember to use a Name-version-release compatible with the Version and Release Guidelines. In particular, check the section on Naming Snapshots.

When Upstream uses Prohibited Code

Some upstream packages include patents or trademarks that we are not allowed to ship even as source code. In these cases you have to modify the source tarball to remove this code before you even upload it to the build system. Here's an example of using a script to document how you went from the upstream tarball to the one included in the package:

From the Spec:

Source0: libfoo-1.0-nopatents.tar.gz
# libfoo contains patented code that we cannot ship.  Therefore we use
# this script to remove the patented code before shipping it.
# Download the upstream tarball and invoke this script while in the
# tarball's directory:
# ./generate-tarball.sh 1.0
Source1: generate-tarball.sh

generate-tarball.sh:

#!/bin/sh

VERSION=$1

tar -xzvf libfoo-$VERSION.tar.gz
rm libfoo-$VERSION/src/patentedcodec.c
sed -i -e 's/patentedcodec.c//' libfoo-$VERSION/src/Makefile

tar -czvf libfoo-$VERSION-nopatents.tar.gz libfoo-$VERSION

Sourceforge.net

For packages hosted on sourceforge, use

Source0: http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz

changing ".tar.gz" to whatever matches the upstream distribution. Note that we are using downloads.sourceforge.net instead of an arbitrarily chosen mirror. You may use the package name/package version instead of the %{name} and %{version} macros, of course.

Please note that the correct url is downloads.sourceforge.net, and NOT download.sourceforge.net.

Git Hosting Services

If the upstream does create tarballs you should use them as tarballs provide an easier trail for people auditing the packages.

Git web-based hosting services provide a mechanism to create tarballs on demand, either from a specific commit revision, or from a specific tag. If the upstream does not create tarballs for releases, you can use this mechanism to produce them.

The full 40-character hash and associated Git Tag may be obtained by issuing the following git command:

git ls-remote https://HOSTING-SERVICE/OWNER/%{name}.git

HOSTING-SERVICE:  name of the service, i.e. "github.com", "bitbucket.org", "gitlab.com", etc.
OWNER:            username for the repository owner

You may also obtain the 40-character hash and associated Git Tag via the web-interface of the HOSTING-SERVICE, or by cloning the repository and issuing the git show-ref command.

Once the commit hash and Git Tag are known, you can define them in your Spec file as follows:

%global commit0 40-CHARACTER-HASH-VALUE
%global gittag0 GIT-TAG
%global shortcommit0 %(c=%{commit}; echo ${c:0:7})    [GitHub]
%global shortcommit0 %(c=%{commit}; echo ${c:0:11})   [Bitbucket]
%global shortcommit0 %(c=%{commit}; echo ${c:0:7})    [GitLab]

Note that the numeric identifier after commit, gittag and shortcommit matches the associated identifier of the Source Tag.

Commit Revision

For the source tarball, you can use the following syntax:

Source0:  https://github.com/OWNER/%{name}/archive/%{commit0}.tar.gz#/%{name}-%{shortcommit0}.tar.gz
Source0:  https://bitbucket.org/OWNER/%{name}/get/%{commit0}.tar.gz#/%{name}-%{shortcommit0}.tar.gz
Source0:  https://gitlab.com/OWNER/%{name}/repository/archive.tar.gz?ref=%{commit0}#/%{name}-%{shortcommit0}.tar.gz
...

%prep
%setup -qn %{name}-%{commit0}              [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0}   [BitBucket]
%setup -qn %{name}.git                     [GitLab]

If the release corresponds to a Git Tag with a sane numeric version, you must use that version to populate the Version Tag in the Spec file. If it does not, look at the source code to see if a version is indicated there, and use that value. If no numeric version is indicated in the code, you may set Version to 0, and treat the package as a "pre-release" package (and make use of the %{shortcommit0} macro). See Pre-Release packages for details.

Alternately, if you are using a specific revision that is either a pre-release revision or a post-release revision, you must follow the "snapshot" guidelines. They are documented here: Snapshot packages. You can substitute %{shortcommit0} for %{checkout} in that section.

Git Tags

Git Tags represent a particular code point that upstream deems important; and are typically used to mark release points.

Bitbucket uses the %{shortcommit0} identifier as part of the archive directory structure; regardless of whether you use Git Tag or Commit Revision to retrieve it. This is shown in the %prep section example.

For the source tarball, you can use the following syntax:

Source0:  https://github.com/OWNER/%{name}/archive/GIT-TAG.tar.gz#/%{name}-%{version}.tar.gz
Source0:  https://bitbucket.org/OWNER/%{name}/get/GIT-TAG.tar.gz#/%{name}-%{version}.tar.gz
Source0:  https://gitlab.com/OWNER/%{name}/repository/archive.tar.gz?ref=GIT-TAG#/%{name}-%{version}.tar.gz
...

%prep
%setup -qn %{name}-%{version}               [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0}    [BitBucket]
%setup -qn %{name}.git                      [GitLab]

Git Submodules

Some projects may use the Git submodule capability. If so, please note that the submodule will not be included in the tarball. Instead, it will appear in the tarball as an empty directory. In this situation, you will need to manually incorporate the submodule into the project.

The simplest method is to incorporate the submodule within the %prep section of the Spec file. In this instance you will follow the preceding instructions to obtain the tarball for the submodule project. Instead of Source0, commit0, gittag0 and shortcommit0; you will use Source1, commit1, gittag1 and shortcommit1. An example %prep section (assuming the use of Commit Revision) would be:

%prep
%setup -qn %{name}-%{commit0} -a 1               [GitHub]
%setup -qn OWNER-%{name}-%{shortcommit0} -a 1    [BitBucket]
%setup -qn %{name}.git -a 1                      [GitLab]
rmdir SUBMODULE                                  [Remove SUBMODULE]
mv -f %{name}-%{commit1} SUBMODULE               [Move SUBMODULE TO %{name}]

If for some reason that isn't feasible, an alternative is to rebuild the tarball prior to it's inclusion in the Spec file. This involves cloning the project, issuing the Git commands to incorporate the submodule, then building a tarball of the newly created project directory structure. You will then need to comment as described in the Using Revision Control section. The following is an example specific to Git submodules:

# The source of this package was pulled from upstreams's vcs.
# Use the following command to generate the tar ball:
# git clone http://HOSTING-SERVICE/OWNER/%{name}.git
# cd %{name}
# git submodule init
# git submodule update
# cd ..
# tar cvjf %{name}-%{version}.tar.gz %{name}/

Keep in mind that the version specified reflects that of the project and NOT the submodule.

Using %{version}

Using %{version} in the SourceX: makes it easier for you to bump the version of a package, because most of the time you do not need to edit SourceX: when editing the Spec file for the new package.

Troublesome URLs

When upstream has URLs for the download that do not end with the tarball name rpm will be unable to parse the tarball out of the source URL. One workaround for many cases is to construct a URL where the tarball is listed in a "URL fragment":

Source0: http://example.com/foo/1.0/download.cgi#/%{name}-%{version}.tar.gz

rpm will then use %{name}-%{version}.tar.gz as the tarball name. If you use spectool -g foo.spec to download the tarball, it will rename the tarball for you.

Sometimes this does not work because the upstream cgi tries to parse the fragment or because you need to login or fill in a form to access the tarball. In these cases, you have to put just the tarball's filename into the Source: Tag. To make clear where you got the tarball, you should leave notes in comments above the Source: line to explain the situation to reviewers and future packagers. For example:

 # Mysql has a mirror redirector for its downloads
 # You can get this tarball by following a link from:
 # http://dev.mysql.com/downloads/mysql/5.1.html
 Source0: mysql-5.1.31.tar.gz