From Fedora Project Wiki


Build JDKs once, repack everywhere

Summary

This is the last step in https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs effort. JDKs in fedora are already static, and we repack portable tarballs into RPMs. Currently, the portable tarball is built for each Fedora and EPEL version. Goal here is to build each JDK (8,11,17,21,latest (20)) only once, in oldest live Fedora repack in all live Fedoras. If jdk is buitl in epel, it will be built in oldest possible epel and repacked in newer live epels.

Owner


Current status

Detailed Description

As described in https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs ; during last year, packaging of JDKs had changed dramatically. As described in the same wiki page and in individual sub changes and devel threads, the primary reason for this is to lower maintenance and still keep Fedora Java friendly.

  • In the first system wide change, we have changed the JDKs to build properly as standalone, portable JDK - the way JDK is supposed to be built. I repeat, we spent ten years by patching JDK to become properly dynamic against system libs, and all patches went upstream, but this has become a fight which can not be won.
  • As a second step we introduced portable RPMs, which do not have any system integration, only build JDK and pack the final tarball in RPM for Fedora use.
  • In third step - without any noise, just verified with fesco - https://pagure.io/fesco/issue/2907 - we stopped building JDK in fully integrated RPMs. Instead of this, normal RPMs BUildRequire portable RPMs and just unpack it, and repack it.

Now last step is ahead - to build portable LTS JDKs 8,11,17 and 21 in oldest live Fedora, and repack everywhere. java-latest-openjdk, which contains latests STS JDK - currently 20, soon briefly 21 and a bit after 22... If we would built java-latest-openjdk in oldest live EPEL - epel8 now, we have verified, that such repacked JDKs works fine, however repack from epel seem to not be acceptable, thus ajva-latest-openjdk will be built twice - one in oldest live fedora, and once in oldest live epel. Build forme oldest possible epel will be repacked to that one or newer epels, and build from oldest live fedroa to all fedoras.

theoretical tagging solution

fN-openjdk tags requested and created via: https://pagure.io/releng/issue/11830 

 0. if possible, request fN-openjdk protected permanent tags
 1. use tag from 0 or request side tags for all releases
 2. build the java-xy-openjdk-portable in the side tag for the oldest thing
 3. tag the result of (2) to all side tags from (1)
 4. waitrepo them
 5. build the java-xy-openjdk pkgs by repacking java-xy-openjdk-portable pkgs in all the side tags from (1)
 6. it may be needed to untag the result of (2) from all the side tags from (1)
 7. ship bodhi updates of java-xy-openjdk-portable from side tags
    (and delete the side tags)
  
  Where xy stands for 1.8.0, 11, 17 and latest.

The build from (2) will be eventually garbage collected. To prevent that, it might be re-tagged regularly. This is where releng might be able to help by creating a long lived tag to tag this into for preserving.

Include the config in dist-git repos, so fedpkg knows the target tag without user input


including portable srpms in release (improving of steps 2+6)

To include portable rpms in all live Fedoras is currently not possible. Best solution would be simply make and bodhi update of one portable rpm to all live fedoras. Bodhi is currenlty not capable to do so, issue was raised: https://github.com/fedora-infra/bodhi/issues/5387 investigating possibility to deliver single build as update to several releases.

"..It's not possible ATM, it would require a heavy rewrite of the code, starting from the database structure (every build is now related to a single release)..." Maybe on long run..."

On long run, if bodhi will allow this, that will be way to go. On short run, there are following options: 

a) ask releng to tag the portables directly
  - this needs manual approach of rare humans, thus no go unless strictly enforced by unpredicted conditions
  - this walks around whole testing repos. For portables tarballs, as nothing should depend on them, and are tested indirectly after repack, this should be technically ok, but is heavily discouraged in principle.
b) build portable for all OSes, but do not ship them (don't do bodhi update)
  - this would probably work for all frontiers, only the real repacked JDK will be different
  - pros is, that we will be sure that portables builds on live fedoras
  - cons is, that the portable JDK will not be available by dnf install anyway
c) build portable for all OSes, including bodhi update
  - pros is, that we will be sure that portables builds on live fedoras
  - another pros is that the portable JDK will be available by dnf install anyway
  - there may be clash during the build  which will cause to repack wrong (newer, non certified) portables
d) include SRPM_REBUILD.readme in srpm and generated PORTABLES_INSTALL.readme in RPMs, which will ideally at least contain:
  - instruction why you need portables
  - instruction how to find the portables
  - from SRPM_REBUILD.readme pointing to PORTABLES_INSTALL.readme
  - generated link to the  koji, allowing to download the SRPM
  - generated link to the  koji, allowing to download the binaries
  - generated instruction how to dnf install used portables

I would currently vote for d). If there will be complains about broken SRPM rebuild, or need to install portables without hacking, then fall-back a, b or c via Change Proposal. Once Bodhi allows single build to be tagged to several release, I will move to that.

Feedback

Benefit to Fedora

Java maintainers will finally have some free time... No kidding - maintenance and *certification* of so much supported JDKs on so much Fedora versions is brutal. By building once, and repack, we will regain cycles to continue support Fedora with all LTS and one STS JDK.

If we fail to build once and repack everywhere, Java maintainers will most likely need to lower the number of JDKs in fedora to system one only.

Scope

  • Proposal owners: Technically all JDKs (except 8, where some more tuning is needed, and EPEL for java-latest) are prepared, as they have a portable version, and RPMs just repack it. Except tuning up the JDK8 and EPEL for latest, scope owners are done.
  • Other developers: There will be needed significant support from RCM and maybe senior Fedora leadership to help to finish the build in oldest and enable to repack everywhere
  • Release engineering: #11438 There will be needed significant support from RCM, where I'm actually unsure what they will have to do to enable this. The mas rebuild will not be needed.
  • Policies and guidelines: AFAIK none (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: All supported JDKs will remain in Fedora in highest possible quality with full QA and certification, and its packagers will not lose their minds. Note that QA will still run on all live Fedoras, not only on the builder one.

Upgrade/compatibility impact

The change should be completely transparent to any user.

How To Test

sudo dnf update/install "java*" will install expected set of working packages.

SRPM rebuild of both portables (which were built once) and of any rpms (from this freshly rebuild portbales) have to remain possible

User Experience

The change should be absolutely transparent to any user.

Dependencies

To finish this we will need heavy support from RCM, and maybe others. Although there are precedents with such pacakge, they all bites. From SW point of view, the dependece chain is normal RPMs build requires portable RPMs and thats all.

Contingency Plan

  • Contingency mechanism: Even if It should be straight forward to revert back to building per OS, it may be impossible for current maintainers to save time for it. If this change is approved, we will be building 4-5 (jdk8,11,17,sts and 21) builds for all fedoras. If this change is not finished in time, we may need to orphan some of the JDKs. In better case, we will be able to keep living one LTS as system JDK, and java-latest-openjdk as future system JDK. That is 2*(3-5) builds (rawhide, (forked,), latest live, oldest live (oldest not yet dropped)). In worst case, we may be able to maintain only java-latest-openjdk. On long run changing it to rolling system JDK, which are the expected 3-5 builds.
  • Contingency deadline: N/A
  • Blocks release? No. The change can be introduced even on the fly to live distributions.

Documentation

N/A (not a System Wide Change)

Release Notes

Packagers and comaintainers tutorial

Releasing openjdks in fedora with this feature on follows https://fedoraproject.org/wiki/Changes/BuildJdkOncePackEverywhere#theoretical_tagging_solution ; here are actual steps for java-latest-openjdk(-portable), which are directly reusable for 1.8.0, 11, 17, 21...:

full version

portables

  1. apply all patches to rawhide and do usual fedpkg build
  2. once done, merge rawhide to latest fedora , then latest to latest-1 and so on to the oldest live
  3. in each of them fedpkg build to ensure jdk is buildable
    1. you can repeat this few times, and only proceed to next once you have release ready
  4. bump release in rawhide and yet again merge to N,N-1...oldest live. fedpkg build for rawhide
    1. do not build others!
    2. you can do shortcut:
      1. bump rpmrelease in rawhide
      2. build in rawhide
      3. build from rawhide into oldest live sidetag
      4. merge to N and N-1 later with other changes maybe
      5. eg the commit exmaple from below, is actually commit which belongs to rawhide
  5. ensure you have all necessary sidetags
    1. original set created as: https://pagure.io/releng/issue/11830 and https://pagure.io/releng/issue/11848
    2. if rawhide forks, request another side tag
  6. go to the oldest live fedora branch, and get hash of oldest commit, and use its relevant sidetag. Eg for java-latest-openjdk with latest live fedora of f38 and latest commit in f38 branch of a71185e830652280780c2d1c5c184feff3f21d2e :
    1. koji build f38-openjdk git+https://src.fedoraproject.org/rpms/java-latest-openjdk-portable.git#a71185e830652280780c2d1c5c184feff3f21d2e
    2. if the nvr already exists, build will properly fail
    3. above hash was https://koji.fedoraproject.org/koji/taskinfo?taskID=110562956
    4. resulting to java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38
    5. insted of koji build mytag terrible-hashes you can use fedpkg build --target=mytag
      1. eg: fedpkg build --target=f38-openjdk
      2. this is prefered path if you merged the release bump everywhere first
      3. it works also cross-branch
  7. tag the build to all your live sidetags
    1. koji tag f39-openjdk java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38
      1. should return zero
    2. for sake of completeness waitrepo it
      1. koji wait-repo f38-openjdk --build=java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38
        1. Successfully waited 0:00 for java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38 to appear in the f38-openjdk repo
      2. koji wait-repo f39-openjdk --build=java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38
        1. Successfully waited 0:12 for java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38 to appear in the f39-openjdk repo

rpms

  1. keep tuning and building repacking rpms in rawhide
    1. feel free to build also branches, but the override dance is usually not worhty
  2. merge rawhide to all live branches. If you were building non-rawhides, bump rpmrelease
    1. be aware! If the system jdk changed, double check, that the corresponding fedoras have proper system jdk!
  3. git push the live fedoras
  4. build to proper tags from relevant branches (mainly because of the system jdk threat)
    1. f38 > koji build f38-openjdk git+https://src.fedoraproject.org/rpms/java-latest-openjdk.git#57a69322e257875057895b9634cd75f6e6c62808
      1. https://koji.fedoraproject.org/koji/taskinfo?taskID=110594866
      2. java-latest-openjdk-21.0.1.0.12-4.rolling.fc38
      3. or f38 > fedpkg build --target=f38-openjdk
    2. f39 > koji build f39-openjdk git+https://src.fedoraproject.org/rpms/java-latest-openjdk.git#40e24677c5eb7e44c408ab1456dc51f52a93c2c1
      1. https://koji.fedoraproject.org/koji/taskinfo?taskID=110594820
      2. java-latest-openjdk-21.0.1.0.12-4.rolling.fc39
      3. or f39 > fedpkg build --target=f39-openjdk
    3. yet again, you ahve to select the latest (or other proper) tag. In this case it really should be per branch
  5. ship bodhi updates of java-latest-openjdk-portable from side tags
    1. koji tag fX-updates-candidate of your freshly built builds from fX-openjdk
    2. koji tag f(X-1)-updates-candidate of your freshly built builds from f(X-1)-openjdk
    3. ... for all live fedoras
    4. then do a bodhi updtae via gui/cli as usually
    5. eg for above builds:
      1. koji tag f38-updates-candidate java-latest-openjdk-21.0.1.0.12-4.rolling.fc38
      2. koji tag f39-updates-candidate java-latest-openjdk-21.0.1.0.12-4.rolling.fc39
      3. in this example I created updates ini gui
        1. https://bodhi.fedoraproject.org/updates/FEDORA-2023-f0ca8f1f17
        2. https://bodhi.fedoraproject.org/updates/FEDORA-2023-441c606b07
      4. I have experientially tagged java-latest-openjdk-portable-21.0.1.0.12-4.rolling.fc38 to f38-updates-candidate and f39-updates-candidate and f40-updates-candidate and it passed.
        1. As f40 was rawhide, it tagged itself properly in and will not be garbage collected
          1. please always do it
        2. unluckily it can not be shipped as update, but is nice marker that this one is the one repacked everywhere

shortcut

portables

  1. apply all patches to rawhide and do usual fedpkg build
  2. once done, merge rawhide to latest fedora , then latest to latest-1 and so on to the oldest live
  3. in each of them fedpkg build to ensure jdk is buildable
    1. you can repeat this few times, and only proceed to next once you have release ready
  4. bump release in rawhide and yet again merge to N,N-1...oldest live.
    1. do not build them
  5. assuming all branches are identical - should be, portables always should be
  6. simple fedpkg build in latest live fedora
    1. eg: git checkout f38 && git merge f39 && fedpkg build
  7. tag it to all all our fX-openjdk live sidetags. eg:
    1. koji tag f38-openjdk java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38
    2. koji tag f39-openjdk java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38
    3. koji tag f40-openjdk java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38
  8. waitrepo them all
    1. koji wait-repo f..-openjdk --build=java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38
      1. Successfully waited 1:32 for java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38 to appear in the f40-openjdk repo
    2. ...

rpms

  1. keep tuning and building repacking rpms in rawhide
    1. feel free to build also branches, but the override dance is usually not worhty
  2. merge rawhide to all live branches. If you were building non-rawhides, bump rpmrelease
    1. be aware! If the system jdk changed, double check, that the corresponding fedoras have proper system jdk!
  3. git push the live fedoras
  4. build to proper tags from relevant branches (mainly because of the system jdk threat)
    1. git checkout rawhide && fedpkg build --target=f40-openjdk (optional)
    2. git checkout f39 && git merge rawhide && git push && fedpkg build --target=f39-openjdk
    3. git checkout f38 && git merge f39 && git push && fedpkg build --target=f38-openjdk
    4. ...
  5. then do a bodhi updtae via gui/cli as usually
    1. eg for above builds:
      1. koji tag f38-updates-candidate java-1.8.0-openjdk-1.8.0.392.b08-7.fc38
      2. koji tag f39-updates-candidate java-1.8.0-openjdk-1.8.0.392.b08-7.fc39
      3. koji tag f40-updates-candidate java-1.8.0-openjdk-1.8.0.392.b08-7.fc40
      4. I have experientially tagged java-1.8.0-openjdk-portable-1.8.0.392.b08-5.fc38 also to f39-updates-candidate and f40-updates-candidate (it wsa in f38 due to shortcut) and it passed.
        1. it is interesting, that it propagated itself to the regular F40 rawhide build, so it is not going to be garbage collected
          1. please always do it
        2. => thus this shortuct seems to be recommended way to go

epel and STSs

For rolling package of java-lates-openjdk(-portable) which si packed for epels, both above are applicable. Shortened shortcut:

portables

  1. ensure you have all necessary sidetags - https://pagure.io/releng/issue/11848
  2. merge rawhide to newest epel
  3. in java-lates-openjdk-portable merge newst epel to epel-1 ... down to lastest live epel
    1. epel7 have lack of aarch64, so it is no longer used
    2. scratch build for each epel where you merge
  4. fedpkg build oldest epel
  5. tag the build from oldest epel to all your sidetags
    1. koji tag el8-openjdk java-latest-openjdk-portable-21.0.1.0.12-4.rolling.el8 (epel8 was oldest live rhel in time of writing this)
    2. koji tag el9-openjdk java-latest-openjdk-portable-21.0.1.0.12-4.rolling.el8
  6. waitrepo them all

rpms

  1. adjust java-lates-openjdk rpms in all epels as neessary
    1. usually simple merge rawhide to epelN then epelN to epelN-1.. down to bottom do not work, the integrations usually differs
  2. in corresponding branches fedpkg build to proper targets:
    1. in epel9 branch: fedpkg build --target=el9-openjdk
    2. in epel8 branch: fedpkg build --target=el8-openjdk
  3. tag the rpmbuilds to updates
    1. koji tag epel8-testing-candidate java-latest-openjdk-21.0.1.0.12-4.rolling.el8
    2. koji tag epel9-testing-candidate java-latest-openjdk-21.0.1.0.12-4.rolling.el9
  4. do gui/cli updates
    1. fedpkg update from proper branch verified to work

shortuct of shortcuts

  1. You can git commit git push git merge and fedpkg build as usually in all branches.
  2. Once you are done koji untag epelX-testing-candidate NVR and koji untag fX-updates-candidate NVR so they do not mess with future tagged build
  3. koji tag elX-openjdk NV.oldestR the desired build(s) to all el*-openjdk tags
  4. koji tag fX-openjdk NV.oldestR the desired build(s) to all f*-openjdk tags
  5. wait repo them
  6. do rpms and updates as in previous steps
  7. koji tag epelX-testing-candidate NV.oldestR and koji untag fX-updates-candidate NV.oldestR to all and especially rawhide.
    1. do not do updates of this portable (well of no portabales)
    2. update will happen automagically in rawhide, and the portables will not be garbage collected
    3. the tag will properly serve for SRPM rebuild
    4. from time to time between CPUs, it is worthy to do a portable fedora updates (once build per fedora) so the portables are at least semifresh
  8. note - if you keep wondering why we simply do not tag oldest rpms to all live fedoras:
    1. bodhi is unable to do update of one pkg to multiple OSes!
    2. and als integration changes and system jdk changes
Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/BuildJdkOncePackEverywhere&oldid=698928"