From Fedora Project Wiki

Revision as of 21:52, 27 May 2020 by Adamwill (talk | contribs) (fix link to richard brown's blog post)

Fedora uses the openQA automated testing system as a significant part of the release validation testing process, and for testing updates. On this page you can find more information about openQA, how Fedora uses it, and how to install your own instance of openQA so you can try it out and contribute to test development.

General information

openQA is an automated test system designed around operating system-level testing. Its key feature is that it interacts with the system under test much like a human would, by sending input using a virtual keyboard and mouse (or serial console input), and monitoring the screen, serial console and audio outputs for output. This makes it ideally suited to running operating system-level tests without making any modifications to the software being tested.

For much more information on openQA, see the upstream documentation.

How Fedora uses openQA

There are currently two official Fedora openQA deployments: production and staging.

At present, Fedora uses openQA for release validation type testing. We have developed a set of tests oriented around testing a complete Fedora distribution compose. These tests are run:

A subset of tests is run on the Cloud images from the nightly Cloud stable release composes, and on each refresh of the semi-official post-release live respins.

We also run a subset of the compose tests, plus some other tests, on some of the updates under test for Fedora stable and Branched releases. The tests are run on all critical path updates, and for some other specific 'whitelisted' packages (where the testing seems specifically relevant and valuable) - they are not run on all updates as we do not have sufficient testing resources for this. The results of these tests are visible on the Automated Tests tab of the update page, for updates which are tested. Updates can optionally be 'gated' on these results.

The results of all openQA tests are forwarded to ResultsDB. When there is a release validation test event for a compose, openQA results are mapped to Wikitcms test cases and reported to the appropriate wiki pages: results you see from the user coconut are results derived from openQA testing.

An email summary of most of the compose runs is produced and mailed to test and devel by the check-compose tool.

The fedora_nightlies tool which produces the nightly compose finder page considers openQA results in determining the test pass/fail status of the images it tracks.

Future plans include 'gating' Rawhide composes on openQA test results (i.e. not pushing a Rawhide compose into the mirror system if it fails the openQA tests).

Fedora openQA tests and tools

The Fedora openQA tests can be found in the os-autoinst-distri-fedora repository. The library, CLI and fedora-messaging consumers that handle scheduling openQA jobs and forwarding results to ResultsDB and Wikitcms can be found in the fedora_openqa repository. The createhdds tool used to create disk image files required by some of the tests can be found in the createhdds repository.

Each of these repositories includes documentation relating to its specific role in the overall system. The fedora_openqa repository includes instructions on installing and using the scheduler and report forwarding features, and the os-autoinst-distri-fedora repository includes Fedora-specific instructions for test development and documentation of Fedora test conventions, functions and settings.

The official Fedora openQA deployments are managed via Ansible playbooks in the Fedora Infrastructure ansible repository. The roles can be found in roles/openqa, and we attempt to maintain them in such a way that you can use them to deploy openQA instances outside of Fedora infrastructure, though this is not tested very often and may bitrot from time to time.

Installing openQA

Running tests directly

To run the standard set of Fedora tests on a regular Fedora compose, you can use the fedora-openqa tool in the fedora_openqa repository. This will cause the server to download all testable image files from the compose and run all the tests for each image.

To run tests only for a specific image, including images not from regular composes, you can send a request to the openQA REST API using a client tool included in openQA. An example invocation can be found in the upstream documentation. Note that the settings required may vary depending on the compose and image being tested and the test being run; you may have to look into the behaviour of fedora_openqa to determine the necessary settings.

There are also several tips on using another script to take shortcuts during test development in the upstream test developer guide.

openQA and other automated test systems

People quite often wonder how Fedora came to start using openQA, and how use of openQA relates to other automated test projects, particularly Taskotron and Fedora CI.

Taskotron (and its predecessor AutoQA) were initially envisioned as comprehensive automated test systems for Fedora, and their design goals did encompass much of the testing that is currently performed by openQA. However, around late 2014, we came to a point where Taskotron was not yet capable of implementing most of the release validation testing required for Fedora releases, but performing all of this testing manually was starting to overwhelm the QA team.

After SUSE's Richard Brown implemented basic openQA testing of Fedora as a SUSE 'hack week' project, Fedora QA decided to see if we could use openQA to automate at least some of the release validation tests immediately. We found we were able to do so quite quickly and that the project did substantially reduce the burden of manual testing.

We quite quickly progressed from a manually-deployed openQA instance running on openSUSE to the production and staging Ansible-ized deployments of openQA-on-Fedora we have now, and found it easy and valuable to extend openQA testing beyond the original plan to cover quite a large proportion of the release validation tests, and run on nightly composes as well as candidates; it was similarly fairly easy and valuable to extend the system to update testing.

Fedora CI was then conceived with broadly similar goals to Taskotron, and is expected to replace it later in 2020. Fedora CI is particularly aimed at enabling CI workflows for Fedora packagers. It is focusing on enabling integration of package-level tests with package pull request, build and update workflows. It is not necessarily expected that all automated testing related to Fedora will run through Fedora CI. Fedora CI works differently to openQA, and so each system is capable of some forms of testing that are not possible or would be difficult or inefficient in the other; openQA's strengths are particularly in installation and desktop testing, and other forms of testing where it is useful to have control over an entire virtual machine life cycle.

As part of the broader Fedora CI / Factory 2.0 effort we have built some systems and standards to support integration of multiple automated test systems. For instance, openQA, Fedora CI and Taskotron all submit results to Fedora ResultsDB, and we have configured Fedora's Greenwave to be able to gather and evaluate results from all three systems. All three systems also publish fedora-messaging messages, and work is ongoing to ensure openQA, Fedora CI and possibly other systems publish messages conforming to the CI Messages spec, which would enable other workflows built on fedora-messaging to use information from all of the systems.

We believe openQA will be used permanently for the types of testing for which it is particularly well-suited. There are some tests currently implemented in openQA not because they are particularly well-suited to its abilities but simply because it was available at the time, and these may be moved into more appropriate systems in future as they become available. There are currently no plans to re-implement the most unique and useful openQA features - mouse/keyboard input, screen matching, and so forth - as part of Fedora CI development.