Infrastructure/Factory2/Focus/Greenwave

What is a Focus Document?

The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.

Background on Greenwave

If ResultsDB allows us to store individual results and WaiverDB allows us to override individual results, we still have a need for a service to look at results in aggregate and provide a way to decide yes or no to different questions, with respect to those results and waivers. This generalizes the “business” decisions that need to be made at various gating and automation checkpoints.

The name comes from this this very interesting wikipedia page.

Formerly known as PolicyEngine.

See also:

• Infrastructure/Factory2/Focus/ResultsDB
• Infrastructure/Factory2/Focus/WaiverDB
• Factory 2.0 talk at DevConf 2017
• Greenwave source code on pagure.io

About the name

This service was originally going to be called "Good Enough", as in "is this artifact good enough to progress to the next stage in the pipeline?" but because the name can't have a question mark in it, people tend to read it in the sense of "meh this is good enough". So the name was changed to PolicyEngine.

However, the name PolicyEngine conveyed some misconceptions about the scope of the tool and its place in the ecosystem. The word "engine" implies that it controls the workflow (in the sense of a "workflow engine" like JBPM). And the word "policy" places emphasis on the idea of defining complex policies as the main goal of the tool. But in fact, the service is:

• answering yes/no questions (or making decisions)
• about artifacts (RPM packages, source tarballs, …)
• at certain gating points in our pipeline
• based on test results
• according to some policy

It's not driving any of the processes, and it's not storing and applying arbitrary policies about anything.

So the team has decided to call the project Greenwave. In road traffic design, a green wave refers to timing of traffic lights so that a car can follow a "wave" of green signals without stopping as it travels along a route. In the domain of software delivery we are working on here, the name is sufficiently abstract not to give any misconceptions about the project scope. But there is a nice connection, if you picture software artifacts flowing efficiently through a series of green signals. (Because every piece of software we deliver is already perfect before it enters our pipeline, of course...) Credit to Filip Valder for originally suggesting the name.

Example policies

The Errata Tool today enforces a number of rules about test results for advisories. Equivalent policies will be represented in Greenwave:

• In order to transition to REL_PREP state, an advisory for the rhel-7 product version:
  • must have either:
    • pass result for dist.tps; or
      • some result for dist.tps with a waiver from a user in the qa group
  • and must have either:
    • pass result for dist.rpmdiff.file_list; or
    • some result for dist.rpmdiff.file_list with a waiver from a user in the devel group and a waiver from a user in the qa group
  • and …

Bodhi does not currently enforce any rules about test results, but some proposals have been floated, for example:

• In order to be pushed stable, an update for the fedora-26 product version:
  • must have either:
    • pass result for dist.depcheck; or
    • some result for dist.depcheck with a waiver
  • and …

New tools currently under development are also planning to use Greenwave to enforce quality. For example Freshmaker could use a policy like this:

• In order to be tagged into the buildroot and trigger a rebuild of dependent packages, a new build of glibc:
  • must have either:
    • pass result for x.y.z.glibc.sanitytest; or
    • some result for x.y.z.glibc.sanitytest with a waiver
  • and …

Why not X?

Jenkins Pipeline is a plugin for defining CI/CD pipelines, which can consist of arbitrarily many stages with support for parallel execution and arbitrary conditional logic. It may seem like a natural fit for managing a pipeline of build stages, like we are discussing here. But Greenwave needs to fit into the existing pipeline, which consists of many diverse tools maintained by diverse parts of the organisation. Using Jenkins Pipeline plugin would require that a single Jenkins job is in control of the entire pipeline. Instead, Greenwave is much narrower in scope: a service just to provide transition rules between stages, to be used incrementally by all the existing tools in the pipeline.

Greenwave may also seem superficially similar to a "business process workflow engine" such as JBPM (Maitai internally). But again, just like Jenkins, "workflow engines" like JBPM are designed to orchestrate arbitrary workflows by triggering actions in other systems and waiting for checkpoints or user inputs. Greenwave is essentially passive. It provides yes/no answers to questions posed by other active systems.

OpenStack Zuul is another tool for "gating", but it is centred around promoting git commits and triggering tests in OpenStack.

Required functionality

• Product version and component share the same meaning as in PDC, and use the identifiers from PDC.

• Each policy is applied to some decision context, identified by a free-form string label.

  New decision contexts are to be named through coordination between policy author and calling application. For example, a policy could mention the bodhi_update_push_stable decision context, and Bodhi would query Greenwave using that same string. Once conventions are established, the decision context labels can be documented or validated in more detail.

• Policies can be applied globally, per product version, or per product version and component.

  Various shortcuts should be supported to make policy authoring easier, for example listing multiple component names or matching on a regular expression (details TBD during implementation).

  Greenwave will enforce the union of all applicable policies for a given product–component–decision context combination.

  This allows, for example, the Fedora community to enforce minimum quality requirements at the distro level, while individual package owners can opt in to stricter requirements for their packages.

• For the minimum viable product (MVP), policies will be configured in one or more YAML files which are part of the application deployment. (Fedora will necessarily have different policies than the Red Hat internal deployment.) Exact structure TBD during implementation.

• Greenwave will listen for new results from ResultsDB and waivers from WaiverDB. When a new result or waiver is received, Greenwave will check all applicable policies for that item, and if the new result/waiver causes the decision to change it will send its own announcement on the message bus about the newly satisfied/unsatisfied policy.

  This will allow other automation (for example Freshmaker) to automatically take action as soon as tests have finished and policy is satisfied.

• Message bus functionality must support both fedmsg for Fedora and the internal "unified message bus".

Proposed API

 GET /decision

with the following query string parameters:

• product_version: as above
• decision_context: as above
• subject: items about which the caller is requesting a decision, for example list of build NVRs (but note that the meaning and type of this parameter will differ depending on decision context so this is TBD during implementation)

Returns a decision as a JSON object with the following keys:

• policies_satisified: overall boolean result, true only if all applicable policies are satisfied
• summary: human-readable internationalized summary of the outcome, for example "all required tests are passing" or "2 of 32 required tests failed" or "1 of 15 required test results is absent"
• applicable_policies: array of policy ids which were considered in evaluating this query
• unsatisfied_requirements: machine-readable details of what is missing in order to satisfy the applicable policies – structure TBD during implementation based on needs of calling applications, but this must be detailed enough so that the caller can tell exactly what action they need to take to satisfy the policies

Deferred functionality

The following features were considered, but are deferred from this initial version of Greenwave.

• User-defined policies

  Ultimately we want the policies enforced by Greenwave to be "self-service". That is, a product manager for OpenShift should be able to directly update the policy for their product to say that TPS is not required, or a BaseOS quality engineer should be able to configure the policy so that glibc builds must pass some set of sanity tests before Freshmaker triggers a rebuild of the whole world.

  However in the initial design described above, policies are expressed in a YAML configuration file deployed with the application. If a change to a policy is needed, it will have to be updated in git and deployed as a configuration change.

  This is intended as a short term measure, to help us release quickly. In the future, once Greenwave has been running "in the wild" and we have more confidence in our design, we can develop a database representation for the policies, plus corresponding API and user interface, to replace the YAML configuration file.

• Policies which consult other data sources, make numerical comparisons, contain arbitrary logical operators, or other complexities

  In this initial iteration, a policy can only specify a set of required test cases from ResultsDB. Policies which make decisions based on other facts (for example, data coming from the Bayesian project) are out of scope for the first phase. In future we could either find a way to coerce that data into ResultsDB, or else expand the policy mechanism beyond just test cases.

• More complex hierarchies of policy

  In this initial iteration described above, there can be a hierarchy of policies at three levels: global, product version, and product version–component. In future it may be necessary to add more levels to the hierarchy (for example per product policies like fedora, per product release policies like rhel-7.2-z, etc) but for now we want to keep things simple until we know what structure will work best.

Communication with ResultDB and WaiverDB

• A user or a system submit waivers to waiverDB.
• When a user (or system) queries Greenwave for a decision, it does so in terms of a subject. This is a list of dicts that each represent queries to ResultsDB. A query subject to Greenwave might look like [{item: 'python-requests-1.2.3-1.el7', type: 'koji_build'}].
• Greenwave takes each entry in the subject list and queries ResultsDB for all matching results. This gives Greenwave a list of results.
• Greenwave then takes each of those results and queries WaiverDB for any matching waivers.