It is still under construction and content may change. Do not rely on the information on this page.
Proposal author: --Ncoghlan (talk) 21:49, 21 August 2015 (UTC)
This page aims to provide a wider context for the draft subproposals discussed in:
- Env_and_Stacks/Projects/DeveloperPortal
- Env_and_Stacks/Projects/PackageReviewProcessRedesign
- Env_and_Stacks/Projects/UserLevelPackageManagement
- Env_and_Stacks/Projects/LanguageSpecificRepositories
The overarching concept behind this proposal is as follows:
To ensure Fedora (and its derivatives) provide a superb platform for constructing a software application from open source components, regardless of the open source programming language used to develop those components, or the open source packaging system used by the developers to publish them
To achieve this, it is proposed to consciously split the software assembly process into the following phases:
- Upstream development & publication
- Upstream release tracking
- Downstream filtering
- Downstream integration
For each of these phases, we aim to identify clear responsibilities and expectations for the Environments & Stacks working group and individual language SIGs in the provision of relevant software components to Fedora users. We also identify any online services specifically relevant to that phase.
Upstream development & publication
Environments & Stacks responsibilities
For this phase, the primary responsibility of the Environments & Stacks working group would be to set appropriate expectations for the individual language SIGs.
Environments & Stacks would also be directly responsible for any language agnostic tools which Fedora may choose to provide to developers for use in this phase of development (e.g. Nix, Conda, Conary)
Individual language SIG responsibilities
For upstream development and publication, it would be the responsibility of individual language SIGs to provide the following for at least Fedora, and potentially for EPEL (if SIG members are interested in that):
- regularly updated language runtimes
- popular development frameworks for their ecosystem (e.g. pygtk, pyqt, Django, Flask, Pyramid, Jupyter for Python)
- ecosystem specific tools for software testing (e.g. tox, nose, pytest for Python)
- ecosystem specific tools for static analysis (e.g. flake8, pylint, mypy for Python)
- ecosystem specific tools for local development environment management (e.g. vex for Python)
- tools for consuming software directly from upstream software repositories (e.g. Maven Central, pypi.python.org, rubygems.org, npm.org)
- tools for publishing software to upstream software repositories
- if available, tools for maintaining local repositories of software in upstream formats (e.g. Nexus for Java, devpi for Python)
In addition to providing and maintaining these kinds of software components, individual language SIGs would be expected to oversee the following activities:
- publication and maintenance of DevAssistant assistants for creation and modification of projects using the language on dapi.devassistant.org
- collaborating with softwarecollections.org and the CentOS SCL SIG on providing regularly updated language runtimes in that format
- collaborating with the Project Atomic community on Source-to-image builders to create container applications using the relevant language runtimes
- providing content for the Developer Portal ensuring that developers using their language on Fedora are readily able to:
- publish software to upstream software repositories
- publish software using COPR
- publish Docker container images to the upstream DockerHub
- publish Vagrant machine images to the upstream Atlas service
- providing content for the Fedora Release Notes that is tailored to the interests of their language community
Optionally, individual language SIGs may also choose to provide alternate runtimes that seem particularly interesting to their communities (such as the PyPy interpreter for Python), and tools for choosing a "default" runtime on a per-user basis (without affecting operating system components written in that language)
Relevant online services
For this phase of development, the only relevant Fedora specific online service is the Developer Portal. Other Fedora services would be involved only as part of the package maintenance process.
Upstream release tracking
The key goal of this step would be to get upstream software components readily available through COPR in a way which requires minimal ongoing effort on the part of package maintainers.
Environments & Stacks responsibilities
For this phase, the primary responsibilities of the Environments & Stacks working group would be to:
- set appropriate expectations for the individual language SIGs
- drive changes to the package review process and related tooling to facilitate this model
- in particular, moving reviews out of Bugzilla to a dedicated review tool
- provide tooling that makes it trivial for folks that have signed the Fedora Contributor Agreement to:
- select an upstream project
- register it for tracking on release-monitoring.org
- assert that it complies with Fedora's licensing policies
- (optionally) create a COPR repo for it with an automatically generated spec file, and link that repo to release-monitoring.org
- provide tooling that automatically updates a linked COPR repo whenever the upstream project issues a new release
Individual language SIG responsibilities
For this phase, the primary responsibilities of the individual language SIGs would be to:
- ensure that anitya (the software powering release-monitoring.org) can track updates from relevant upstream repositories
- ensure that tools are available to automate generation of SRPM files from upstream package definitions and sources (the results may or may not be compliant with Fedora packaging policies)
- ensure that source-to-image builder images are available to create container images directly from a source repo and an appropriate language runtime image
Relevant online services
Achieving the goals laid out for this phase would involve three key online services:
- release-monitoring.org (powered by Anitya)
- the Fedora Review Service (powered by Fresque, not yet proposed for deployment)
- the COPR build service
Anitya itself shouldn't need any changes, but COPR would need updates to track the links to Anitya monitored packages, and a new supporting service to trigger automated updates and rebuilds when upstream issues a new release.
The Fedora Review Service isn't a live project yet - Fresque exists, but I'm not aware of any currently active effort to get it into production. E&S would need to drive the process of getting that ready for use and deployed.
Downstream filtering
The key goal of this step would be to provide self-service filtered subsets of the COPR smorgasboard that meet additional criteria beyond "a Fedora Contributor decided to publish a COPR for them"
Environments & Stacks responsibilities
For this phase, the primary responsibilities of the Environments & Stacks working group would be to:
- drive the addition of a repo tagging and filtering system to COPR
- manage the definition of a set of "standard tags" identifying various attributes about a repo
- work with Fedora QA to enable Taskotron managed CI tasks to set appropriate tags to indicate when repo updates have passed their CI testing
- work with the Fresque developers to enable Fresque to set appropriate tags to indicate when a repo has passed through additional review gates (e.g. packaging policy compliance)
- create tooling to make it easy to define custom repositories based on COPR filtering criteria
Individual language SIG responsibilities
For this phase, the primary responsibilities of the individual language SIGs would be to provide guidance to developers on how to effectively use COPR and Taskotron for automated software CI.
Relevant online services
Achieving the goals laid out for this phase would involve four key online services:
- the Fedora Review service
- the COPR build service
- the Taskotron CI service
- a new filtered repo service called RepoFunnel
The impact on the first three was described in the E&S subsection above. RepoFunnel would be a new service that provided the ability to select a filtered subset of COPR repositories, create a corresponding downstream Pulp repo for them, and automate the process of keeping that repo updated when the query results changed. (I've started working on building a proof of concept for this part, since it's the key to making the whole idea practical)
Downstream integration
This is the point where longstanding Fedora processes take over.
Environments & Stacks responsibilities
At this point, E&S hands over the reins to FESCo, Release Engineering and the Base WG to define policies and procedures. E&S involvement covers ensuring that the handover from RepoFunnel to dist-git is smooth, and preferably automated via Fresque.
Individual language SIG responsibilities
This proposal doesn't change anything here - language SIGs work with FESCo to define appropriate packaging policies, maintain RPM macro packages, etc.
Relevant online services
This proposal doesn't change anything here - dist-git, koji, bodhi, Taskotron, etc