It is still under construction and content may change. Do not rely on the information on this page.
This document summarizes alternative approaches for storing and presenting automated test results used by different automation solutions.
Beaker
rpmdiff
Background
rpmdiff is a collection of tools ...
- A test tool for static and comparative analysis of packages
- A test scheduler for pushing tests to test systems
- A web front-end for results reporting/waiving
Scheduling
Rpmdiff tests are initiated by ...
- ad-hoc "make" invocations by maintainers
- ad-hoc XML-RPC by maintainers (for a CLI tool)
- web UI for manual scheduling
- package updates - analogous to bodhi updates
There is more information recorded in the database around queuing and running tests on systems. I'm ignoring this since it's not something we're including in our discussion.
Execution
Not much to say here, rpmdiff compares two packages and reports feedback. There is a utility included with yum-utils called rpmdiff, these are similar, but not the same.
Results
Each test run produces a list of results. Each result consists of:
- severity - INFO, GOOD, BAD, VERIFY
- text - the message for the end-user
Reporting
All test results are sent to stdout. In addition, the results are saved to an xml data file which is sent back to the scheduler (via XML-RPC). The scheduler processes the input and ...
- Updates database so that results are visible from the web front-end
- Based on the results, emails failure information to maintainers
DB schema
Thankfully, the database doesn't appear very complex. The same database is used for storing scheduler and results information, which isn't something we'd need to worry about. So I'll focus just on the results ...
There are more tables, but the 4 key tables for test results appear to be:
- tests - just some metadata about the test, including a URL to a wiki page
- runs - stores job information (time, what is being testing, NVRs etc...). All information so that the same test could be initiated at a future date.
- results - all the test results produced by the run
- run (foreign key)
- test (foreign key) - basically a link back to the specific test
- test result - PASS, INFO, FAIL, WAIVE, INSPECT
- log - stdout associated with the test
- waivers - contains information for all result waivers
- person - some reference to who is waiving
- description - user supplied text describing waiver
- old_result - the test result prior to waiving
- datestamp