From Fedora Project Wiki
Purpose
As we're looking to work more on AutoQA, we wanted to write out what we see the requirements of Fedora test automation is going forward. To farther that goal, we wrote out what we feel the requirements and 'should have's would be for an ideal system.
If you are reading this and feel that something should either be added, removed or modified; please don't just edit the page - start a conversation on autoqa-devel@ so that we can have a better idea what you're thinking than what we can get from changed wiki page.
Definitions
For the sake of consistency:
- 'must' means that something is a requirement. maybe not on initial release but it has to at least seem possible without too much effort or too many dirty hacks
- 'should' means that it would be preferred but not absolutely required
- 'would be nice' means that it would be cool, but nothing to lose much sleep over.
Basic Requirements
- should be written mostly in Python
- must be package-able in fedora and EPEL repos
- mostly a licensing thing, other packaging issues could be overlooked if upstream is at least interested in taking patches to fix any issues.
- should have a friendly, responsive upstream
- must have an understandable codebase
- must be extendable without dirty hacks
Reporting
- must be able to coordinate with bodhi (1.0, 2.0)
- must be able to report some information via fedbus
- must support the ability to report to external systems
- must be clear about what test version, package-under-test version and fedora release correspond with the reports
- must be clear about the test system's state (package versions, installed packages etc.)
- should have some standardized reporting format
- based on something standard like XML, json, yaml etc.
Automation Framework
- must be able to support spawning VMs in Fedora infra's cloud
- or at least some other solution that supports rapid VM provisioning without the need to install from scratch for every test. installing from scratch every time is not an acceptable option.
- must be able to differentiate between fedora release numbers and package versions
- should be able to tell the difference between VMs and bare metal where appropriate
- would be nice to have the VM type used during tests as a variable when hooked up to a cloud-ish setup
- would be nice to support graphical installation testing
- would be nice to include support for grabbing new images from image builder when that's supported
Library
- should be written in mostly python
- should not make writing tests in languages other than python more difficult than it needs to be
- would be nice to support basic reporting options in other languages
Test Runner
- must support any language (within reason)
- must be able to pull in new/updated tests independently from runner or framework updates
- must support version-specific tests (ie variants for different fedora releases)
- should be runnable outside the framework for testing and development purposes
- would be nice to be able to support multiple libraries (beakerlib, application specific stuff, non-python support etc.)
Tests
- must be decoupled from the automation framework
- must be able to run outside the automation framework
- must report sane results (looking at you, depcheck)
Test Repository
- must support non-python languages
- must enable a review process for new tests before they are accepted
- must allow for test updates without admin or dev intervention
- should be in an existing package format (python egg, rpm etc.)