From Fedora Project Wiki

< Changes

Revision as of 10:11, 9 March 2020 by Pmatilai (talk | contribs)

Sqlite RpmDB

Summary

Change format of the RPM database from Berkeley DB to a new Sqlite format.

Owner

Current status

  • Targeted release: Fedora 33
  • Last updated: 2020-03-09
  • FESCo issue: <will be assigned by the Wrangler>
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

The current rpm database implementation is based on Berkeley DB 5.x, a version which is unmaintained upstream for several years now. Berkeley DB 6.x is license incompatible so moving to that is not an option. In addition, the existing rpmdb implementation is notoriously unreliable as it's not transactional and has no other means to detect inconsistencies either.

Changing to a more sustainable database implementation is long overdue. We propose to change the default rpmdb format to the new sqlite based implementation. Support for current BDB format will be retained in Fedora 33, and phased out to read-only support in Fedora 34.

Benefit to Fedora

  • A far more robust rpm database implementation
  • Getting rid of Berkeley DB dependency in one of the core components

Scope

  • Proposal owners:
    • Once RPM 4.16 lands and passes initial shakedown, change the default rpmdb configuration to sqlite
    • Address any bugs and issues found by wider testing base
  • Other developers:
    • Test for hidden Berkeley DB dependencies in other software, address them as found and needed
  • Policies and guidelines: N/A (not a System Wide Change)
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Upgrading

  • Ability to upgrade is not affected
  • After upgrade completes, manual action (rpmdb --rebuilddb) will probably be needed to convert to sqlite. Alternatively user can change configuration to stay on BDB.

Compatibility

  • Container/chroot use-cases will be affected: older rpm versions will be unable to query/manipulate the rpmdb from outside the chroot
  • Koji/COPR may need to override the database format (back to) BDB for the time being

How To Test

  • Rpmdb gets thoroughly exercised as a matter of normal system operation, performing installs, updates, package builds etc
  • Of specific interest here is torture testing: forcibly killing rpm in various stages of execution - database should stay consistent and operational (other system state is out of scope)
  • Test database conversions from one backend to another (rpmdb --rebuilddb --define "_db_backend <backend>")

User Experience

  • In normal operation, users should see little or no change
  • Behavior in error situations is much more robust: forcibly killed transaction no longer causes database inconsistency or corruption

Dependencies

  • This change depends on RPM 4.16, support for sqlite rpmdb is not present in older versions
  • RPM will grow a new dependency on sqlite-libs
  • Technically the rpmdb format is an internal implementation detail of RPM only accessible through the librpm API, but some software is making assumptions both about the format and/or in particular, file naming. These are being tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1766120
  • Upgrade tooling could/should perform rpmdb rebuild at end, this would be a good thing to do regardless of this change

Contingency Plan

  • Contingency mechanism:
    • Revert the default database back to Berkeley DB backend in the package. Running 'rpmdb --rebuilddb' on hosts is currently required to actually convert the database, but means to automate conversion in specific conditions is being discussed upstream.
    • The rpm-team does not expect problems with the database backend itself, but reverting may be needed due to infrastructure or other tooling not being ready, primarily due to inability to access the database from older releases.
  • Contingency deadline: Beta freeze
  • Blocks release? Yes

Documentation

N/A (not a System Wide Change)

Release Notes

  • After upgrading from an older release, rpm operations will issue warnings about database backend configuration not matching what's on disk. Users should run 'rpmdb --rebuilddb' at earliest opportunity, or change configuration to stay on Berkeley DB backend (eg 'echo %_db_backend bdb > /etc/rpm/macros.db')
  • The details are subject to change, the database rebuild may be done by upgrade tooling
Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/Sqlite_Rpmdb&oldid=567903"