Devshell API

Developement

The following document explains the key components in Fedora Devshell, and how to extend it for your own nefarious purposes. The Modules section addresses the kinds of actions Fedora Devshell can do, and how they are implemented. The Utility functions highlights some of the key components used as building blocks. The Coding Guidelines are critically important, as they must be followed to in order to submit patches. The Submission Guidelines explain how to get your patches in.

Modules, Interfaces and Base Classes

Directory

Directory is a Base Class for pretty much any class needing persistence on the filesystem. It manages storing metadata attributes in a simple INI-style file under .devshell in each directory. It establishes the link between some directory on the file system and the operations available from the command line, inside the source code, and from any IDE. The initializer takes one optional parameter, a path to the directory on the file system, and it loads up the Directory from there. If not given, Directory uses the current working directory.

It exposes these methods:

• is_sysdir_dir(dir) - given some directory, it determines if this is a Directory managed by Fedora Devshell
• load_dir(dir) - a hook to be run when loading some directory that has already been created. do not call this directly, although feel free to override it in subclasses. make sure to call the parent method
• make_dir(dir) - a hook to be run when creating a Directory out of some new directory. do not call this directly, although feel free to override it in subclasses. make sure to call the parent method
• close() - a hook to save persisted information when Fedora Devshell exits. safe to call directly, the results should be idempotent. feel free to override it in subclasses, make sure to call the parent method
• rename(new_name) - a hook that renames the directory, assuming it's been changed on the file system to match already. feel free to override it in subclasses, make sure to call the parent method
• move(self, new_loc) - moves the directory to a new location, and calls rename
• copy(self, new_loc): - copies the directory to a new location and calls rename, returns the new Directory object

And properties:

• name - the name of the directory on the file system
• dir - the absolute path to the directory on the file system
• parent - the absolute path to the parent of the directory on the file system

MetaDirectory

This metaclass registers all directory objects with DirFactory

DirFactory

A subclass of Directory that will return the correct subclass of Directory for a given directory on the filesystem, rather than the base class.

Package

Package is an overarching manager for packages that will eventually become RPMs. There should be a one-to-one correspondence with Package and a source RPM. There will also theoretically be a one-to-one correspondence between Package and entries in the Fedora CVS repository. It manages a spec file, a list of sources, patches for the actively chosen PackageSources and other administrative details.

It exposes the following methods:

• add_spec(spec_file) - given some spec file, set the current spec file to be this one.
• get_srpm_name(profile) - given a profile, determine the name of the source RPM
• ver(profile=None) - given a possible profile, determine the upstream version from the current spec file
• copy_source(source_dir) - given some PackageSource, copy it into this Package and add it as a source
• move_source(source_dir) - given some PackageSource, move it into this Package and add it as a source
• add_source(source_dir) - given some PackageSource directory, assuming it is already inside the Package directory, add it as a source
• rem_source(source) - remove a PackageSource from the source list, without deleting it from the file system
• del_source(source) - delete a PackageSource from the file system and remove it from this Package
• fetch_sourceballs(profile=None) - with an optional profile for version information, fetch all the source tarballs from the PackageSources belonging to this package. To do so, it creates symlinks to save space

The following properties:

• spec_file - the name of the current spec file
• pkg_name - the canonical name of the package in the Fedora CVS repository
• sources - a list of the PackageSources
• sourceballen - a list of all the source tarballs that have been fetched at some point

Possible Package Types

Packages can be managed in all sorts of ways. Fedora currently uses CVS to manage the workflow. Other people might want to use git. In the future, we will need modules to manage these different workflows

BuildSystem

BuildSystem is a Python class of type MetaBuildSystem. MetaBuildSystems are registerd with a BuildSystemFactory, which can load any BuildSystem loaded in the run time space. It's initializer takes one argument, a PackageSource either on the file system or a Python object. BuildSystem is an interface on top of common build systems such as autotools, cmake, and cabal.

It exposes five methods.

• configure(target, *args) - runs the configure stage of the build system - can be a noop. target points to where to install it on the system, *args are passed to the underlying package source to work on a specific branch.
• build(*args) - runs the build stage - should not be a noop. *args are passed to the underlying package soruce to work on a specific branch.
• install(*args) - runs the install stage - cannot be a noop. *args are passed to the underlying package soruce to work on a specific branch.
• install_source(target, *args) - configures, builds, and installs some source. target points to where to install it on the system, *args are passed to the underlying package source to work on a specific branch.
• gen_spec() - creates a spec file specific to this build system

Cabal

Cabal is an implementation of a BuildSystem. It works on top of The Haskell Cabal, which is the build tool used for building Haskell packages. It uses the program cabal2spec provided in Fedora for generating the spec file.

Possible BuildSystem Types

Currently we only support Cabal. Plugging in other build systems should be easily feasible, possible options include Autotools , Ant, Python's setuptools, Ruby Gems, CMake, and many other build systems. We may want to look into handling multiple build targets for cross compiling.

PackageSource

PackageSource is a Python class of type MetaDirectory. MetaDirectorys are registered with a DirFactory, which can load any directory from the filesystem as the properly subclassed Directory in Fedora Devshell. PackageSource is a single source, be it a tarball or revision control that will go into an RPM Package.

It handles the management of patches between the original upstream source and the downstream sourcetree, generating tarballs and patches, Each implementation manages the storage and generation of these details.

It exposes these methods.

• setup_sourceball(ver=) - sets up a source tarball if needed out of the revision control. ver is the upstream version
• setup_sourceball_w_patches(ver=) - not sure of the behavior of this one yet, it's a placeholder
• src_dir(*args) - a contextmanager that will run a block of code inside a specfic branch or checkout
• src(*args) - a context manager that will run a block of code with a certain branch or checkout set as the active one.
• set_current_src() - a hook to be implemented by the subclass, notifies the subclass when the active source has changed and runs any arbitrary code.
• set_cur_to(*args) - where *args is processed. This parses *args, and sets the source to some source directory accordingly. Must be overridden in a subclass
• branch(*args) - returns the relative path to where a named branch is stored, based on *args.
• branch_dir(*args) - returns the absolute patch of branch

It exposes these properties

• pkg_src - a relative path where internal files to devshell are stored inside the package source
• pkg_src_dir - the absolute pathname to pkg_src
• branches - a relative path to where branches are stored inside their package source
• branches_dir - the absolute pathname to branches
• sourceball - a relative pathname to the current source tarball equivalent to this package source
• sourceball_loc - the absolute pathname to sourceball
• source - a relative pathname to the currently active sourcetree
• source_dir - the absolute pathname to source
• buildsystem - the type of buildsystem that this sourcetree uses
• builder - a reference to a BuildSystem that can build this package

SourceBall

A SourceBall is an implementation of PackageSource. It maintains both an actively edited work tree, and a copy of the original source tree, along with the original sourcetarball for comparison. It can only generate a single stack of patches, on top its one branch, 'orig'.

• *args - can either be 'head' or 'orig'.

RevisionControl

Subclasses PackageSource for all implementations that use RevisionControl. This is a very boring class.

Darcs

The only implementation of RevisionControl. It wraps around the Darcs Distributed Version Control System. It maintains multiple branches based on parameters passed directly to darcs. It does not yet do patch management.

• *args - can be one of several possibilities.
  • beings with 'new' <new-branch-name> - creates a new branch by that name
  • 'head' - the top level active source tree
  • 'branch' <branch-name> - switches to a named branch
  • anything else - passes these parameters to darcs and creates an unnamed patch after the hash of the arguments

Examples

new foo branch bar - forks bar as foo
new foo --tag 0.8 - forks tag 0.8 as foo
--tag 0.8 - switches to tag 0.8
--to-match 'hash 4593894085394890' - switches to some patch identified by some hash

Possible PackageSource Types

Currently, Darcs and SourceBall are the only supported types. Darcs does branching fundamentally different from Git, Bazaar, and Mercurial. Work will need to go into having the paradigms sit together nicely. Note that we aren't creating an all in one solution, but just using the important bits of each VCS to generate patches to go into RPMs. Ultimately, RPMs must be packaged with Tarballs, the use of RevisionControl is only to ease and aid the process.

Fetcher

Provides an upstream source for multiple packages. Exposes:

• latest_version(pkg) - returns the latest version of some package by name
• url(pkg, ver) - given a package and version, return a URL where source can be found, namely source control and/or a tarball
• latest_url(pkg) - shortcut function, given a package, return a URL to the latest release

Hackage

An implementation of Fetcher for downloading packages from HackageDB - The Haskell Package Repository

Possible Fetcher Types

There are many sources for packages, including Sourceforge, Launchpad, Fedora Hosted, CPan, Python Cheese Shop, and many many others. Implementing Fetchers is easy, so lets see some more.

Port

A Port is analogous to a BSD or Gentoo Port. It combines the above components into a single Module that has an intellegent overview of the whole situation. It can be used to download, setup, configure, install, and package any type of package, so long there is a port to handle it.

It currently exposes:

• close_later(directory) - given some Directory object, mark it for closing later. Ports may come across some Directory object it does not own, that it will pass on to the outside world. This marks it for deletion once the system is done running. This ensures that all changes to the Directory are persisted.

NOTE: This will very likely turn into a utility function later on.

HaskellPort

This will download any package that is hosted on HackageDB or at darcs.haskell.org, test build it, install it, and generate spec files for it. It combines Hackage, Cabal, Darcs, and SourceBall.

Possible Port Types

Theoretically, there could be one port for every package that is being managed. Many do have in fact their own idiosyncronacies that do need to be managed. Despite that, many components in packages are standardized out of other common systems. One goal of ports is to be able to use as much generic code as possible. For example, a KDE port might use a KDEFetcher, CMakeBuilder, SVNSource, and SourceBall. We may want to look into other ways to declaratively define Ports, and even include them as a repository somewhere.

Build

A wrapper for rpmbuild. It subclasses Directory, and takes an rpmdev-tree (such as ~/rpmbuild, or /usr/src/redhat) as the directory. It can then perform common rpmbuild operations on Packages.

It exposes:

• setup_source(package) - for some Package, import the spec file, tarballs, and patches into the rpmdev-tree using symlinks to save space and time
• build_quick_rpm(package, profile=None) - given some package, and some optional profile, build a quick rpm using the current system as the buildroot. NB: This package should never be shared publically, it's only good for testing
• build_source_rpm(package, profile=None) - given some package and some optional profile, build an SRPM
• rpmbuild(param, package, profile=None) - runs rpmbuild with some parameter, on some package with some optional profile, worker function for Build
• fetch_rpms(target_dir) - fetches all the RPMs from the rpmdev-tree to some target directory, by moving them. should be run after every invocation of rpmbuild
• fetch_build(package) - given some package, fetches the expanded, possibly built source tree from the rpmdev-tree, and deposits it in the package. this is for doing error debugging, test, and verification

Mock

Mock is a simple subclass of a Module, it is merely a worker module with no state of its own. It uses a Profile for all the stateful bits. It does one thing simplemindedly, it builds a source RPM in mock given some profile and rpmdev-tree for doing temporary work. The initializer takes a Profile and Build

It exposes:

• build_rpm(package) - given some package, use the Build and Profile to build it cleanly in Mock

Possible Mock Types

Mock is a bit specific purpose. We may want to define a better clean room type builder. There are many technologies for distributing and cleanly building packages, including Mock, Koji, and many other frameworks. We may want to create a generic RPMBuilder, that Mock is an implementation of. For example, via a single switch, we could send a package to Mock on a local system, do a scratch build in Koji, or inject it into a VM that can be sent across a grid using Condor.

Profile

Profile is a profile to be used in Mock. It manages seperate repos of packages built in mock before to be used as a base, version information, vis-a-vis which version of Fedora to use, and other important information about the build profile. It uses its own directory store mock configurations and results from mock.

It exposes the methods:

• configure_from_system(branch) - given some branch (the same as in Fedora CVS), make a new profile based on the local system mock configurations (in /etc/mock) in the directory.

And properties:

• dist - eg .fc7 or .olpc2 used in rpmmacro %dist
• distvar - eg: fedora or rhel
• distval - eg: 10 or 3 (string form)
• koji_target - eg: dist-f11
• dist_defines - a list of parameters for an rpm aware function to redefine certain macros to match the profile
• mock_cfg - the name of the mock config/profile to be used to compile packages for this profile
• mock_cfg_dir - directory where profile wide mock settings are kept
• result_dir - where to store results from mock

Coding Guidelines

The following basic rules must be applied. Exceptions can be granted for any reason, provided there's a good reason.

• Basic coding standards will follow PEP 8, except where noted below.
• Code should be as big as your head. If you have a small head or a large monitor, lower your font resolution ;).
• Code should be terse, and haiku like. It should explain a single concept or action clearly, without being overrun by scaffolding.
• Scaffolding should be broken down into utility functions, all exposed in the API. They should be documented clearly.
• Where logical, use context managers. This will isolate the alogrithm from the build up and break down.
• High nesting of if/then statements is not allowed
• High nesting of context managers and for loops are allowed, although extreme nesting is discouraged.
• Read only attributes from obj.cfg should be exposed as properties as such:

@property
def spec_file(self):
    '''returns the name of the spec file as it should be accordingto
    fedora guidelines, good for double checking'''
    return self.pkg_name + '.spec'

@property
def pkg_name(self):
    '''canonical name of the package in a source repository'''
    return self.cfg['pkg_name']

Submission Guidelines

All patches are welcome. The best way to submit patches initially is via email or mailing lists, or sending a pull request from another git repo. In order to get write access to the public repo, there is a one drink minimum. To demonstrate understanding of the Coding Guidelines, the submitter must have at least one other patch approved before gaining write access. This will ensure that the coder has had ample time to learn the simple guidelines.