From Fedora Project Wiki

(Created page with "= Rawhide synchronization for the GNU Binutils = This page documents the process by which the rawhide binutils package is synced with the upstream GNU Binutils project. The aim of this process is to make syncing simple and easy so that it can be performed on a regular basis. Due to the fact that the binutils is an important component in the toolchain used to build many of the packages in rawhide however, it is important to make sure that any attempt at syncing does no...")
 
Tag: Replaced
Line 1: Line 1:
= Rawhide synchronization for the GNU Binutils =
= Rawhide synchronization for the GNU Binutils =


This page documents the process by which the rawhide binutils package is synced with the upstream GNU Binutils project.  The aim of this process is to make syncing simple and easy so that it can be performed on a regular basis.  Due to the fact that the binutils is an important component in the toolchain used to build many of the packages in rawhide however, it is important to make sure that any attempt at syncing does not break anything.
*THIS PAGE IS UNDER DEVELOPMENT AND SHOULD NOT BE CONSIDERED READY FOR USE*


This process is based upon a similar process for the GNU C Library ([[GlibcRawhideSync]]), but tweak to fit the binutils package.
This page documents the process by which the rawhide binutils package is synced with the upstream GNU Binutils project.  The aim of this process is to make syncing simple and easy so that it can be performed on a regular basis.  Due to the fact that the binutils is an important component in the toolchain used to build many of the packages in rawhide however, it is important to extensively test a new binutils before putting it into the buildroot.  This does make the syncing process more time consuming and complicated.
 
This process is based upon a similar process for the GNU C Library ([[GlibcRawhideSync]]), but tweaked to fit the binutils package.


== TL;DR ==
== TL;DR ==
Line 9: Line 11:
Make sure you have authenticated and meet the pre-requisites (see notes below).
Make sure you have authenticated and meet the pre-requisites (see notes below).


If you do this every week, here's the cut and paste snippets you need, minus the bits that change every week...
Follow these steps:
 
<pre>
kinit
 
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
 
cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
cd $HOME/fedsrc/glibc; git pull --rebase
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec
 
fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit
 
fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`
 
rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)
 
cd $HOME/fedsrc/glibc; git push
fedpkg build
 
https://bodhi.fedoraproject.org/updates/?search=glibc
</pre>
 
== Synchronization Process ==
 
{{Admon/tip | Fedora CI/CD | Fedora Rawhide must pass Fedora CI/CD, in particular `baseos-qe.koji-build.scratch-build.validation` before you can consider the synchronization process complete.}}
 
'''1.''' Setup/Fetch a clean upstream master repository for glibc. This can be an existing directory also which you switch to master branch and ensure it has been rebased e.g. '''git pull --rebase''' cleanly to head.
<pre>
mkdir -p $HOME/src
cd $HOME/src
git clone git://sourceware.org/git/glibc.git glibc-pristine
</pre>
 
'''2.''' Setup/Fetch a clean Fedora rawhide glibc repository.  If this is not your first time syncing, you may need to remove the glibc-patches subdirectory that the previous sync created.
<pre>
cd $HOME/fedsrc
fedpkg clone glibc
</pre>
 
'''3.''' Run '''fedpkg sources''' to downloads the .tar.xy file needed for creating the glibc-patches repository (next step).
<pre>
cd $HOME/fedsrc/glibc
fedpkg sources
</pre>
 
'''4.''' Convert the Fedora rawhide glibc dist-git repository to a glibc-patches repository (git repo with Fedora patches applied as commits).
<pre>
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-patches-to-git.py
</pre>
 
'''5.''' Synchronize from upstream master to Fedora rawhide and review changes.
NOTE: Pick the appropriate set of instructions based on where we are in the release.
 
No release transitions occurring:
<pre>
cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose
git diff
Confirm that glibcsrcdir, glibcversion, and Release have the correct NEVRA.
</pre>
 
Transitioning from stable release -> master:
<pre>
cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose --import-git-branch master
git diff
Manually change the glibcsrcdir, glibcversion, and Release to match the new NEVRA.
</pre>
 
If a merge conflict occurs, review the section on '''Dealing with Merge Conflicts''' then return to the next step.
 
'''6.''' Manually document note-worthy changes in the %changelog. This step is probably the most complex. You need to look at all the changes since the last sync (the hash is recorded in the %changelog) in the upstream repo and see if there is anything note-worthy to talk about e.g. '''git diff HASH1^..HASH2'''. You will use this text in your commit message also.
<pre>
* Wed Nov 07 2018 Florian Weimer <fweimer@redhat.com> - 2.28.9000-14
- Auto-sync with upstream branch master,
  commit 1df872fd74f730bcae3df201a229195445d2e18a:
- libanl: Fix crash if first helper thread creation failed (#1646381)
</pre>
 
A useful command to summarize commits is <b>git log</b> like this:
<pre>
cd $HOME/src/glibc-pristine
git log --oneline e0cb7b6131..0ddb7ea842 | sed 's/^[^ ]* /- /'
</pre>
The two hashes to use are based on the first ten digits of the last sync's git commit and the current sync's git commit. You want the log listing to show all of the new commits that have been included in the current sync.  In the above example, we're updating from <tt>e0cb7b6131</tt> (the commit of the last sync) to <tt>0ddb7ea842</tt> (the commit of the current sync). Verify the boundary commits make sense.
 
'''7.''' Add new files (still in the Fedora rawhide directory from the last step) and commit
<pre>
cd $HOME/fedsrc/glibc
fedpkg new-sources glibc-*.tar.xz
git add glibc.spec
git commit
</pre>
An appropriate commit message would be:
<pre>
    Auto-sync with upstream branch master
   
    Upstream commit: 1df872fd74f730bcae3df201a229195445d2e18a
   
    - libanl: Fix crash if first helper thread creation failed (#1646381)
</pre>
Following the spec file %changelog example from above.
 
'''8.''' Test a scratch build, and wait for it to complete.
<pre>
fedpkg srpm
fedpkg scratch-build --srpm ./glibc-XXX.src.rpm
</pre>
 
'''9.''' Verify scratch build results by downloading logs and looking for any unexpected failures.  A list of known failures is kept at the end of this page, and should be updated as needed.
<pre>
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=$TASKID
</pre>
 
'''10.''' Push the commit and kick off a Rawhide build if the logs look good (you are not done yet!)
<pre>
git push
fedpkg build
</pre>
 
'''11.''' Fedora CI/CD must run and you must verify that `baseos-qe.koji-build.scratch-build.validation` succeeded. If Fedora CI/CD does not succeed please reach out to [mailto:mcermak@redhat.com Martin Cermak (BaseOS QE)]. To verify the status of CI/CD go to [https://bodhi.fedoraproject.org Bodhi] after your build successfully completes, select '''Search...''', type in '''glibc''', hit enter and pick the newest Rawhide build errata that was automatically created and select the '''Automated Tests''' tab. If `baseos-qe.koji-build.scratch-build.validation` is green then the test has passed and you are complete with the sync.
 
== Authentication ==
 
'''1.''' Kerberos init with Fedora realm.
 
'''2.''' Make sure your ssh agent has your Fedora ssh key for pagure.io.
 
== Pre-requisites ==
 
'''1.''' You have a Fedora account with an SSH key, and have setup your SSH key on pagure.io by logging in.
 
'''2.''' Install '''fedpkg''', '''python3-pygit2''', '''python3-rpm''' and '''git-merge-changelog'''
 
'''3.''' Configure your '''~/.gitconfig''' to have entries for '''name''', and a merge driver for GNU Changelogs.
<pre>
[user]
name = My Name
 
[merge "merge-changelog"]
        name = GNU-style ChangeLog Merge driver
        driver = /usr/bin/git-merge-changelog
[core]
attributesfile = ~/.gitattributes
</pre>
 
'''4.''' Configure your '''~/.gitattributes'''
<pre>
ChangeLog      merge=merge-changelog
</pre>
 
'''5.''' Clone '''glibc-maintainer-scripts''' into '''$GLIBC_MS''' for the git synchronization scripts.
<pre>
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
mkdir -p $HOME/fedsrc
cd $HOME/fedsrc
git clone https://pagure.io/glibc-maintainer-scripts.git
</pre>
 
'''6.''' Clone '''UpstreamToolchainBuildScripts''' from the Fedora Toolchain team for the build log fetching scripts.
<pre>
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
cd $HOME/fedsrc
git clone https://pagure.io/FedoraToolchainTeam/UpstreamToolchainBuildScripts.git
</pre>
 
Note: if you plan on editing either of the two git repos you just installed, use ssh://git@pagure.io instead of https://pagure.io
 
== Differences for Release Syncs ==
 
For syncing a glibc release branch into rawhide, you need to use a command line this, and manually reset the Release number the first time:
 
<pre>
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch [Release branch you are using e.g. release/2.31/master] --import-git [Path to your updated checked out glibc tree e.g. $HOME/src/glibc-2.31] --verbose
</pre>
 
For Fedora release branches, the process is basically the same, except you have to use the Bodhi system to push your build to a release.
 
* In the above instructions, use the release branch of glibc (typically like release/2.28/master) and the release branch of Fedora (typically like f29)
 
* Go to https://bodhi.fedoraproject.org/
 
* Click the ? icon in the upper right, to search for "glibc"
 
* Note previous updates for wordings and severities
 
* Select "Create" in the upper right and create a new update
 
* Type "glibc" in the packages field.  If there's a popup, select the appropriate entry.
 
* type "glibc" in the candidate builds field, select the relevent build from the popup.
 
* Click in the "Related bugs" field.  If your BZ doesn't show up, type in the number.
 
* Fill in the Update Notes field and final details.
 
* Submit!  The system will advertise the update and request karma.  When the update has enough karma, it's automatically pushed out.
 
* This is an example of a Fedora 39 sync to release/2.38/master. It assumes you have done at least one sync previously, have completed the pre-requisites, and cloned the source files.
 
<pre>
kinit
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
 
cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
* Checkout the version you are syncing to.
git checkout release/2.38/master
cd $HOME/fedsrc/glibc; git pull --rebase
* Checkout the version of Fedora that you are syncing.
git checkout f39
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
* Specify --import-git-branch for the correct release.
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch release/2.38/master --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec
 
fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit
 
fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`
 
rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)
 
cd $HOME/fedsrc/glibc; git push
fedpkg build
 
* Create the bodhi update. You can use a previous update as an example:
https://bodhi.fedoraproject.org/updates/?search=glibc
</pre>
 
== Dealing with Merge Conflicts ==
 
* Dealing with merge conflicts due to backport+rebase:
 
Sometimes, a fix is backported from upstream into rawhide, and subsequently a rebase is scheduled. During this rebase, the backport and its corresponding patch need to be dropped.
 
One way to achieve this is to manually edit the spec file prior to running '''glibc-patches-to-git''' so that the merge conflict is avoided in the first place.
 
Another way is to let the merge conflict occur, then run '''git rebase --skip''' in the '''glibc-patches''' directory/repository to skip the application of the current patch (i.e. the one backported, which is causing the merge conflict) and continue the rebase (this rebase is an internal detail of how '''glibc-sync-upstream''' is implemented and was started by the script before erroring out due to the conflict).
 
Then we continue the process by executing '''glibc-git-to-patches''' in '''$HOME/fedsrc/glibc''', which now skips the patch when creating '''glibc-patches'''.
 
Finally, we run '''glibc-patches-to-git --branch master''', to bring update dist-git with the recently refreshed '''glibc-patches'''.
 
* Dealing with Fedora patches that no longer apply cleanly due to changes in rawhide:
 
Another merge conflict scenario is when a Fedora patch does not apply cleanly.  You may see a message
like this:
 
CONFLICT (content): Merge conflict in nss/nsswitch.conf
error: Failed to merge in the changes.
 
In this case, the patch is needed for Fedora but an upstream change has caused the patch to be out of sync.  You need to fix the patch in the glibc-patches directory.  Using nss/nsswitch.conf as an example:
 
<pre>
cd $HOME/fedsrc/glibc/glibc-patches
edit nss/nsswitch.conf to fix the merge conflict
git add nss/nsswitch.conf
git rebase --continue
 
cd $HOME/fedsrc/glibc
</pre>
 
If you see that there is a glibc.spec and glibc.spec.new
 
<pre>
mv glibc.spec glibc.spec.old
mv glibc.spec.new glibc.spec
</pre>
 
Now complete the git-to-patches process, regenerating the patch that failed previously:
 
<pre>
$GLIBC_MS/glibc-git-to-patches.py -v
</pre>
 
You should see a message similar to this:
 
info: regenerating patch 'glibc-fedora-nsswitch.patch' because patch failed
 
<pre>
git add glibc-fedora-nsswitch.patch
</pre>
 
== Occasions that require special handling ==
 
* The addition of a new locale upstream:
 
When a new locale is added upstream, the change corresponds to a new line in `localedata/SUPPORTED` and `glibc.spec` needs to be adjusted to tackle this. The spec file includes a lua script containing a list of supported locales that is compared at build time to the list upstream. This is below the definition of `%package locale-source`. Grep for `^local locales =` to locate the list and add a line corresponding to the new locale there before building.
 
== Known Acceptable Build Failures ==
 
For the purposes of "looking for any unexpected failures" (above), this is the current list of build failures that have been deemed "not to stop a sync".  After you complete the scratch build and download the logs, compare the test failures in the logs with the results below.  For each failure not listed here, analyze it, determine if it's safe to continue syncing, and add that failure below.  For each failure listed below that no longer appears in the test results, remove it from below.  I.e. most syncs will require minor updates in this section.
 
=== All Hosts ===
 
<pre>
There is a known misc/tst-ttyname failure that needs waiving (https://bugzilla.redhat.com/show_bug.cgi?id=2210335) since we do not currently believe this is a glibc regression - per codonell.
</pre>
 
=== aarch64 ===
 
<pre>
Rawhide (1 fail):
FAIL: misc/tst-mount
 
FAIL: elf/tst-debug1
=====FAIL: elf/tst-debug1.out=====
Didn't expect signal from child: got `Bus error'
=====FAIL: elf/tst-debug1.test-result=====
FAIL: elf/tst-debug1
original exit status 1
=====FAIL: dlfcn/tst-dlinfo-phdr=====
see upstream bz 23293
</pre>
 
=== armv7hl ===
 
<pre>
Rawhide (1 fail): FAIL: debug/tst-longjmp_chk2
=====FAIL: debug/tst-longjmp_chk2.out=====
not on alternate stack
in signal handler
on alternate stack
Didn't expect signal from child: got `Aborted'
=====FAIL: debug/tst-longjmp_chk2.test-result=====
FAIL: debug/tst-longjmp_chk2
original exit status 1
 
FAIL: misc/tst-sysvshm-linux
</pre>
 
The tst-sysvshm-linux failure is related to:
https://sourceware.org/bugzilla/show_bug.cgi?id=26862:
 
=== i686 ===
 
<pre>
Rawhide (1 fail):
FAIL: misc/tst-mount
</pre>
 
This seems to be transient, unable to reproduce manually:
<pre>
=====FAIL: io/tst-stat-time64.out=====
tst-stat-time64.c:88: numeric comparison failure
  left: 8 (0x8); from: stx.stx_blocks
  right: 16 (0x10); from: st.st_blocks
error: 1 test failures
</pre>
 
<pre>
 
FAIL: math/test-double-erfc
FAIL: math/test-double-tgamma
FAIL: math/test-float32x-erfc
FAIL: math/test-float32x-tgamma
FAIL: math/test-float64-erfc
FAIL: math/test-float64-tgamma
FAIL: math/test-float-hypot
FAIL: math/test-float32-hypot
 
These relate to this bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=20305
FAIL: signal/tst-minsigstksz-1
FAIL: signal/tst-minsigstksz-2
</pre>
 
=== ppc64le ===
 
<pre>
 
FAIL: math/test-float-nearbyint
FAIL: math/test-float-rint
FAIL: math/test-float32-nearbyint
FAIL: math/test-float32-rint
 
Rawhide (2 fails)
FAIL: math/test-ibm128-llround
FAIL: math/test-ibm128-y1
 
Previous failures - no longer seeing these:
FAIL: misc/tst-mount
 
FAIL: misc/tst-sigcontext-get_pc
Sporadic failure:
FAIL: time/tst-cpuclock1
</pre>
 
=== s390x ===
 
<pre>
Rawhide (1 fail):
FAIL: misc/tst-mount
 
Sporadic failure:
FAIL: rt/tst-cpuclock2
</pre>
 
=== x86_64 ===
 
<pre>
Rawhide (1 fail):
FAIL: misc/tst-mount
 
=====FAIL: nptl/tst-stack4.out=====
=====FAIL: nptl/tst-stack4.test-result=====
FAIL: nptl/tst-stack4
original exit status 127
</pre>
 
 
== Known CI Failures ==
 
Please list known CI failures here. CI failures must be coordinated with BaseOS QE.
 
=== fedora-ci.koji-build.rpmdeplint.functional ===
<pre>
glibc-devel-2.36.9000-21.fc38.aarch64 - testout.log
 
glibc-devel-2.36.9000-21.fc38.aarch64 provides /usr/include/bits/sigaction.h which is also provided by glibc-headers-s390-2.36.9000-21.fc38.noarch


The above is a known-failure because not all headers can be installed at the same time. The CI test is attempting to install every package we can install and that's not a valid test of anything.
  1. Make sure that your clones of the upstream GNU Binutils and the Fedora rawhide binutils are up to date and cleanIf there any problems or uncommitted changes, resolve them before continuing.
It is not valid to install the aarch64 headers and s390x headers at the same time. The test doesn't know this. Nobody would every need to do thisBut it attempts all noarch package installs.
</pre>


=== baseos-qe.koji-build.scratch-build.validation ===
  <pre>
<pre>
  cd <upstream-binutils> ; git pull ; git status
===i686 kernel scratch build mock-output.log ===
  cd <rawhide-binutils>  ; git pull ; git status
kernel-6.2.0-0.rc3.20230111git7dd4b804e080.26.fc38.i686
  </pre>
built with glibc-devel-2.36.9000-21.fc38.i686


No matching package to install: 'bpftool'
  2. Create a source tarball from the upstream sources:
...
Not all dependencies satisfied
Error: Some packages could not be found.


known failure - not a glibc issue
  <pre>
</pre>
  cd <upstream-binutils> ; ./src-release -x binutils
  </pre>


<pre>
3. Copy the tarball into the fedora sources:
=== aarch64 lua scratch build build.log ===
lua-5.4.4-7.fc38.src.rpm, aarch64
built with glibc-devel-2.36.9000-21.fc38.aarch64


<pre>
  <pre>
testing 'format %a %A'
  cp binutils.tar.xz <rawhide-binutils>
>>> closing state <<<
  </pre>
/builddir/build/BUILDROOT/lua-5.4.4-7.fc38.aarch64//usr/bin/lua: strings.lua:342: assertion failed!
stack traceback:
[C]: in function 'assert'
strings.lua:342: in main chunk
[C]: in function 'dofile'
all.lua:165: in main chunk
[C]: in ?
.error: Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
    Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
RPM build errors:
Child return code was: 1


lua test suite fails assertion - not a glibc issue
  4.
</pre>

Revision as of 15:29, 20 February 2024

Rawhide synchronization for the GNU Binutils

  • THIS PAGE IS UNDER DEVELOPMENT AND SHOULD NOT BE CONSIDERED READY FOR USE*

This page documents the process by which the rawhide binutils package is synced with the upstream GNU Binutils project. The aim of this process is to make syncing simple and easy so that it can be performed on a regular basis. Due to the fact that the binutils is an important component in the toolchain used to build many of the packages in rawhide however, it is important to extensively test a new binutils before putting it into the buildroot. This does make the syncing process more time consuming and complicated.

This process is based upon a similar process for the GNU C Library (GlibcRawhideSync), but tweaked to fit the binutils package.

TL;DR

Make sure you have authenticated and meet the pre-requisites (see notes below).

Follow these steps:

 1. Make sure that your clones of the upstream GNU Binutils and the Fedora rawhide binutils are up to date and clean.  If there any problems or uncommitted changes, resolve them before continuing.
  cd <upstream-binutils> ; git pull ; git status
  cd <rawhide-binutils>  ; git pull ; git status
  
 2. Create a source tarball from the upstream sources:
  cd <upstream-binutils> ; ./src-release -x binutils 
  
3. Copy the tarball into the fedora sources:
  cp binutils.tar.xz <rawhide-binutils>
  
 4.