From Fedora Project Wiki

Revision as of 04:23, 15 February 2017 by Adamwill (talk | contribs) (point Docker install guide to upstream openQA docker directory (it's a bit out of date but we need to get it fixed upstream))

Fedora uses the openQA automated testing system as a significant part of the release validation testing process. On this page you can find more information about openQA, how Fedora uses it, and how to install your own instance of openQA so you can try it out and contribute to test development.

General information

openQA is an automated test system designed around operating system-level testing. Its key feature is that it interacts with the system under test much like a human would, by sending input using a virtual keyboard and mouse (or serial console input), and monitoring the screen, serial console and audio outputs for output. This makes it ideally suited to running operating system-level tests without making any modifications to the software being tested.

For much more information on openQA, see the upstream documentation.

How Fedora uses openQA

There are currently two official Fedora openQA deployments: production and staging.

At present, Fedora uses openQA for release validation type testing. We have developed a set of tests oriented around testing a complete Fedora distribution compose. These tests are run:

A subset of tests is run on every nightly two-week Atomic compose, and on each refresh of the semi-official post-release live respins.

The results of all openQA tests are forwarded to ResultsDB. When there is a release validation test event for a compose, openQA results are mapped to Wikitcms test cases and reported to the appropriate wiki pages: results you see from the user coconut are results derived from openQA testing.

An email summary of most of these runs is produced and mailed to test and devel by the check-compose tool.

The fedora_nightlies tool which produces the nightly compose finder page considers openQA results (as well as Autocloud results) in determining the test pass/fail status of the images it tracks.

Future plans include 'gating' Rawhide composes on openQA test results (i.e. not pushing a Rawhide compose into the mirror system if it fails the openQA tests), and using openQA to run tests on package updates and provide feedback to Bodhi.

Fedora openQA tests and tools

The Fedora openQA tests can be found in the os-autoinst-distri-fedora repository. The library, CLI and Fedmsg consumers that handle scheduling openQA jobs and forwarding results to ResultsDB and Wikitcms can be found in the fedora_openqa repository. The createhdds tool used to create disk image files required by some of the tests can be found in the createhdds repository.

Each of these repositories includes documentation relating to its specific role in the overall system. The fedora_openqa repository includes instructions on installing and using the scheduler and report forwarding features, and the os-autoinst-distri-fedora repository includes Fedora-specific instructions for test development and documentation of Fedora test conventions, functions and settings.

The official Fedora openQA deployments are managed via Ansible playbooks in the Fedora Infrastructure ansible repository. The roles can be found in roles/openqa, and we attempt to maintain them in such a way that you can use them to deploy openQA instances outside of Fedora infrastructure, though this is not tested very often and may bitrot from time to time.

Installing openQA

Running tests directly

To run the standard set of Fedora tests on a regular Fedora compose, you can use the fedora-openqa tool in the fedora_openqa repository. This will cause the server to download all testable image files from the compose and run all the tests for each image.

To run tests only for a specific image, including images not from regular composes, you can send a request to the openQA REST API using a client tool included in openQA. An example invocation can be found in the upstream documentation. Note that the settings required may vary depending on the compose and image being tested and the test being run; you may have to look into the behaviour of fedora_openqa to determine the necessary settings.

There are also several tips on using another script to take shortcuts during test development in the upstream test developer guide.

openQA and other automated test systems

People quite often wonder how Fedora came to start using openQA, and how use of openQA relates to other automated test projects, particularly Taskotron.

Taskotron (and its predecessor AutoQA) were initially envisioned as comprehensive automated test systems for Fedora, and their design goals do encompass much of the testing that is currently performed by openQA. However, around late 2014, we came to a point where Taskotron was not yet capable of implementing most of the release validation testing required for Fedora releases, but performing all of this testing manually was starting to overwhelm the QA team.

After SUSE's Richard Brown implemented basic openQA testing of Fedora as a SUSE 'hack week' project, Fedora QA decided to see if we could use openQA to automate at least some of the release validation tests immediately. We found we were able to do so quite quickly and that the project did substantially reduce the burden of manual testing.

We quite quickly progressed from a manually-deployed openQA instance running on openSUSE to the production and staging Ansible-ized deployments of openQA-on-Fedora we have now, and found it easy and valuable to extend openQA testing beyond the original plan to cover quite a large proportion of the release validation tests, and run on nightly composes as well as candidates.

We now conceive of Taskotron as being a set of modular components that can be used in automated testing and CI workflows, and no longer necessarily expect absolutely all Fedora-related automated testing to run through Taskotron's native test execution components. openQA result forwarding to ResultsDB can already be considered a form of openQA/Taskotron integration, as ResultsDB began as the 'Taskotron results store'; the move to consider it a more generic system for storing test results is part of the overall adjustment to thinking of Taskotron as a modular system.

We believe openQA will be used permanently for the types of testing for which it is particularly well-suited. There are some tests currently implemented in openQA not because they are particularly well-suited to its abilities but simply because it was available at the time, and these may be moved into more appropriate systems in future as they become available. There are currently no plans to re-implement the most unique and useful openQA features - mouse/keyboard input, screen matching, and so forth - as part of Taskotron development.

As mentioned above, openQA is already quite well integrated into the rest of the Fedora infrastructure through the use of Fedmsg and the forwarding of results into ResultsDB. We also envisage potentially using Taskotron triggers to schedule openQA jobs in the future, rather than using the current standalone scheduler code.