From Fedora Project Wiki
Line 193: Line 193:
|
|


* Separation makes automation a lot easier since adding to the changelog is just pre-pending some lines, and does not require any knowledge of rpm syntax.
* Separation makes automation a lot easier since adding to the changelog is just pre-pending some lines to a detached file
* Separation removes a source of noise from spec history
* Separation removes a source of noise from spec history
* Separation enables further automation such as the one proposed in
* Separation enables further automation such as the one proposed in

Revision as of 10:31, 5 July 2020

Patches in Forge macros - Auto macros - Detached rpm changelogs

Summary

redhat-rpm-config will be updated to add patching support to forge macros, a plug-able framework to register macros to execute in specific sections, and rpm changelogs in detached files.

Owner

  • Email: <nicolas.mailhot at laposte.net>

Current status

  • FESCo issue: <will be assigned by the Wrangler>
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

This is a system-wide change because all packages build with redhat-rpm-config, but it only concerns packages that opted to use this part of redhat-rpm-config (users of forge, fonts and go macros).

It was driven first, by the need to make the underlying macro infrastructure robust enough to package Go modules, and second, by an unfortunate rpm 4.15 regression that proved it was foolish to depend on rpmbuild to parse Tags in anything except canonical order.

Fully automated packaging

A framework was added so macro subsystems can register execution blocks in specific parts or the spec file. Execution blocks are orchestrated (using KISS rules) so for example the forge part of %prep is executed before the go parts that depend on forge archives being unpacked and patched, and macros that want to create srpm headers are executed before macros that want to create subpackage headers.

Such a framework is a requirement to control the generation order within the spec file and make sure rpm maintainers are not cross with you.

The macro execution plan is printed while it is being executed so the packager is not left wondering what the hell rpmbuild is doing (especially when syntax mistakes result in rpmbuild aborting with unhelpful error messages, or when rpmbuild breaks additive processing by executing the preamble multiple times without clearing state between executions).

rpmspec -P allows checking the generated spec and markers were added to this spec to show the execution block that generated a particular part of the spec (in all rpm sections that allow comments).

That means a spec with no special custom processing is reduced to a set of %global control variables that activate specific execution blocks, and absent custom processing needs, everything bellow those control variables is short and unchanging boilerplate.

A packager that needs custom processing can add custom code above or bellow the various %auto_foo calls, and check with rpmspec -P that the result does what he wants it to do. For obvious reliability reasons injecting custom code in the middle of an %auto_foo sequence is not allowed.

%global source_name …
%global source_release …
%global source_post_release …
    
%global forge_url0 …
%global forge_commit0 …
    
%global forge_url1 …
%global forge_tag1 …
    
%global go_module33 …
%global go_description33 …
    
%global font_family22 …
%global font_conf22 …
    
%auto_init
%auto_pkg
    
%sourcelist
%auto_sources
    
%patchlist
%auto_patches
    
%prep
%auto_prep
    
%generate_buildrequires
%auto_generate_buildrequires
    
%build
%auto_build
    
%install
%auto_install
    
%check
%auto_check
    
%auto_files
    
%changelog
%auto_changelog

Forge

  • the forge macros now process patches, including when the spec contains multiple forge sources, in a natural way
  • all dependencies on source/patch numbering were eradicated, you can write a whole multi-source/multi-patch spec without worrying about source or patch numbers
%forge_urlX …
%forge_tagX …
%forge_patchlistX %{expand:
foo.patch
bar.patch
whatever.patch
}

%forge_urlY …
%forge_commitY …
%forge_patchlistY %{expand:
another.patch
}
  • zero suffix is no longer special (freeing the packager from Source/Source0/%setup numbering problems), you can declare forge blocks starting at 42 if that‘s your preference
  • explicit %forgemeta, %forgesource, %forgesetup, etc calls are no longer needed, the corresponding execution blocks are registered in the %auto_call framework and will be executed by %auto_init %auto_sources, %auto_patches and %auto_prep as needed, without requiring any forge-specific boilerplate.
  • thus, forge macros use is now streamlined and (absent patching) reduced to a list of two-line declarations, suffixed with a number (X and y in the following example)
%forge_urlX …
%forge_commitX …
    
%forge_urlY …
%forge_tagY …

%auto_init

%sourcelist
%auto_sources

%patchlist
%auto_patches

%prep
%auto_prep

Automated SRPM header creation

The automation framework was used to implement a buildsys subsystem, charged with creating SRPM headers as needed, and controlling the SRPM evr. A future evolution of this subsystem is expected to make packages autobump without external help.

https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

Detached changelogs

The buildsys subsystem was used to implement detached changelogs in a reliable way

A future evolution of this subsystem is expected to make changelogs autobump without external help.

Generic -doc creation

The automation framework was used to implement a doc subsystem, charged with -doc subpackage creation, because creating documentation subpackages by hand gets annoying after the nth upstream that wants you do distribute heavy PDF files.

Huge refactoring and fleshing out of the fedora lua library

Writing high-level features like the above required defining a library of lua routines like an expand that expands fully, an unset that actually undefines, a read that does not think an empty variable is the same thing as an unset variable, a fedora.echo() wrapper around rpm.expand("%{echo:%{expand:" .. text .. "}}"), etc. Those are now available for others to make use of should they want to.

Designing APIs is hard and I make no claim that the result is perfect, just that is it more convenient and less tricky that the raw low level calls rpm exposes in its own rpm lua library.

Drawbacks

Nothing is free, and a higher level of automation required using rigid naming for control variables. Because software is a lot less tolerant of fuzzy naming than human beings.

So, all forge control variables are renamed, fonts control variables have been renamed too, and go control variables will need renaming (in that last case, that’s not a problem because moving to go modules requires reworking variables anyway, so it will be done as part of the module effort in F34).

To ease the transition a compatibility layer was added to forge macros so old variables and new variables are aliased both ways (this will eventually go away because it’s quite a lot of compatibility code to maintain). Mixing syntaxes (old and new) is not supported, you need to convert your spec file to new forge variables or not at all (if not at all, do not try to use new features like patching).

Feedback

Third party bumpspec tools will break This is dealt with in the follow-up change:
Is %autosetup included in the auto_ framework?
  • %autosetup is not part of the new framework, it antedates it.
  • All the new %auto_call macros are named %auto_something.
Why is a separate "rpm-changelog.txt" file changelog better than current changelog inside .spec?
  • Separation makes automation a lot easier since adding to the changelog is just pre-pending some lines to a detached file
  • Separation removes a source of noise from spec history
  • Separation enables further automation such as the one proposed in

https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

  • The change automates the rest of the spec from top to bottom, why should spec automation stop at the %changelog line?
Why is a separate "rpm-changelog.txt" file with manually maintained changelog better than current manually maintained changelog inside .spec?

See next question.

How about using git commit log for changelog instead?

This is a low level rpm change that does not depend on any specific SCM infrastructure, git included. It works directly at the rpm level. So it does not depend on the existence of git and git commits.

However, a git infra can make use of the now detached changelog to feed commit info to the rpm build process.

How will the changelog be maintained?

The changelog will be maintained any way you wish to maintain it, it’s just a plain text file in package sources.

An infrastructure that uses git, can feed git commit events to the detached changelog file, using dumb or elaborate git commit hooks, and any other method it wants to implement.

Upstream rpm intends to solve the same problem another way

I can’t comment or compare my approach with something that does not exist today except as wishful thinking. This is production code that works now in real-world packages, and builds upon 13 years of progressive automation of Fonts, Forge and Go Fedora packages, applying lessons learnt the hard way during this period.

When the forge macros this change builds upon were proposed and merged 3 years ago, people also opposed that upstream would do better some other way, and no one has seen a sliver of actual upstream implementation to this day.

If upstream eventually adds something useful in this problem space, this code will make use of it. The change makes use of very recent upstream additions like %sourcelist, it’s not as if it was blind to upstream improvements (it would be so nice if the reverse was true).

Conversely, if upstream’s implementation is too limited for practical application, it will be ignored (like the code ignores source_num/patch_num to compute source and patch suffixes).

This is too complicated and can certainly be done some other simpler way

Dealing with the real world with its warts is complicated. This is a real-world implementation not a paper design, and it has the scars to prove it.

Wishing something could be done simpler is not sufficient to produce a simpler implementation.

Some of the code in there workarounds pathologic rpm behaviour, so it *could* be simplified if the things it workarounds were fixed.

You will eat brainz

Yes I will eat brainz and do more evil things besides.


Benefit to Fedora

Spec files that do more with less manual expensive to maintain spec code.

Without this productivity win, complex efforts like converting Fedora Go packages to Go modules, or draining the Font packages swamp given that legacy formats are no longer supported by apps, are not possible with the current level of Fedora manpower.

Scope

  • Other developers: the way current forge macros call forge macros will need a little patching once the change lands. For other packagers, there should be no change except a warning in rpm build logs to switch to the new syntax before the compatibility layer is removed.
  • Policies and guidelines: Forge guidelines will need some rework (mostly simplification, because the new syntax is both more powerful and more regular). For the average packager, the new syntax is the same old syntax with little naming adjustments.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

This is a pure build tooling update, it changes how things are built not what is built.

How To Test

A redhat-rpm-config packages with the changes and some example packages are available in

The code itself is visible in

User Experience

N/A Packager experience change only

Dependencies

The change depends on a redhat-rpm-config merge by redhat-rpm-config maintainers

Contingency Plan

There is no contingency plan because the redhat-rpm-config merge will happen or not. If it does not happen, i18n, fonts and Go Changes that are/were envisioned for F33 or F34 will be postponed indefinitely.

Documentation

There is as much documentation as the average redhat-rpm-config change (ie comments in the macro files themselves)

Release Notes

N/A Packager productivity change only