From Fedora Project Wiki

m (add some categories)
(add requirements for test case pages)
Line 65: Line 65:
*: There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name ''Fedora '''RELEASE''' '''MILESTONE''' Test Results''
*: There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name ''Fedora '''RELEASE''' '''MILESTONE''' Test Results''
*: If there are nightly validation pages for the release, there is a category with the name ''Fedora '''RELEASE''' Nightly Test Results'', which is a member of the that release's category
*: If there are nightly validation pages for the release, there is a category with the name ''Fedora '''RELEASE''' Nightly Test Results'', which is a member of the that release's category
* Test case pages are in the ''QA'' namespace and their name begins with ''Testcase_'' (which, as noted above, is effectively equivalent to ''Testcase '')


=== Contents ===
=== Contents ===
Line 86: Line 87:


The result text for a given page ''may'' be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no ''test name'', or with the same ''test name'') but they appear in separate sections, they are considered two separate ''test instances''.
The result text for a given page ''may'' be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no ''test name'', or with the same ''test name'') but they appear in separate sections, they are considered two separate ''test instances''.
==== Test case pages ====
The contents of test case pages is not currently used by wikitcms the 'system'; it effectively simply expects that what they contain will be some sort of content from which a human being can generate a test result. The contents are 'parsed' only by the human testers.
In practice, it is strongly established that each page should contain a single test case using the [[Template:QA/Test_Case]] template, according to [[QA:SOP_test_case_creation]].


=== Auxiliary pages ===
=== Auxiliary pages ===

Revision as of 18:18, 22 January 2015

Wikitcms is a term used to refer to Fedora's use of this Mediawiki instance as an ad hoc test management system ('TCMS' stands for 'Test Case Management System', but that precise initialism is in fact quite rarely used). It is also the name of a Python module which provides an interface to the 'system'.

Background

The use of the Fedora wiki to track test results dates back to at least the page QA/FC6Test2TreeTesting - at that time, the Fedora wiki was a MoinMoin instance, not this Mediawiki instance. The rough form of a table with different tests as the rows and results as columns is visible even there. Since then, this basic format has been gradually elaborated.

Detailed history

Features

The following can be considered as features of the wikitcms 'system':

  • Storage and organization (as pages, in categories) of test cases
  • Advanced formatting and templating for test case content (via the mediawiki engine)
  • Result reporting without authentication
  • Capability to distinguish between test case and 'test instance'
  • Results sorted by environment (with arbitrary 'environments')
  • Tests for a given build grouped by 'test type' and section within test type

Implementation

Fundamentally, wikitcms consists of a set of conventions and expectations about the names, contents and categorization of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.

General caveat for things interfacing with wikitcms

The component pages of the 'system' can often be human edited, and so variation must be expected in the content within the requirements specified below. Two considerations apply to most data you may want to take from or enter into the system:

  • Mediawiki tends to tolerate and ignore leading and trailing whitespace: for instance, mediawiki displays {{result|fail|adamwill|123456}} and {{ result | fail|adamwill | 123456 }} identically, and things interfacing with the system should treat them as equivalent
  • In page names underscore (_) and space ( ) characters should be treated as identical and interchangeable: Foo_Bar and Foo Bar lead to the same page, and therefore any code handling page names should expect to cope with either format and consider them as identical

The wikitcms module follows these principles and handles most variations in the formatting of the data it can extract, so using it is recommended.

Naming conventions

  • Release validation result pages are in the Test Results namespace, and are named Fedora RELEASE MILESTONE COMPOSE TESTTYPE or Fedora RELEASE MILESTONE DATE TESTTYPE, where:
    RELEASE is a Fedora release version number
    MILESTONE is a valid Fedora milestone for the release, or a nightly compose type, currently Rawhide or Branched
    COMPOSE is a TC or RC compose identifier, e.g. TC1 or RC2 (nb: RC2.1 is a valid format here)
    DATE is a nightly compose date, in YYYYMMDD format
    TESTTYPE is one of the validation test types (see below)
  • Test types are defined by the existence of a template page (in the Template namespace) named TESTTYPE test matrix
  • Validation test category pages are named as follows:
    The top-level Category:Test_Results exists
    Each Fedora release has a Fedora RELEASE Test Results category that is a member of Category:Test_Results
    There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name Fedora RELEASE MILESTONE Test Results
    If there are nightly validation pages for the release, there is a category with the name Fedora RELEASE Nightly Test Results, which is a member of the that release's category
  • Test case pages are in the QA namespace and their name begins with Testcase_ (which, as noted above, is effectively equivalent to Testcase )

Contents

Result pages

The result pages contain a section heading, Test Matrix, which acts as a separator. The page wikitext beyond that separator is considered the result text. Results take the form of one or more wiki tables, but separating tests into different tables is only cosmetic.

Result tables must follow these rules:

  • They have a single header row, which appears as a single line in the wikitext. The format where each cell in the header row appears on a newline in the wikitext is not allowed for wikitcms purposes
  • Each row other than the header row - a result row - represents a single test instance (usually just a test case, but see elsewhere in this section for the way in which test names and page sections can produce multiple test instances for a single test case)
  • In contrast to the header row, result rows place each of their cells on a new line in the wikitext. The format where the entire row appears on a single line of wikitext is not allowed for wikitcms purposes
  • The first column indicates the level (usually milestone) of the test
  • One of the columns must be used for the purpose of identifying the test instance with which that row is associated
    For each row, the contents of that column must be formatted to begin with a link to the test case associated with the row, e.g. [[QA:Testcase_Mediakit_Checksums]]
    The link may be named, in which case the name portion is considered the test name, e.g. [[QA:Testcase_Boot_default_install|Workstation live]]. This is an example of a test instance which is not simply a single test case. It is considered a test instance with the test case QA:Testcase_Boot_default_install and the test name Workstation live. The same test case could appear in another row with a different test name, which would constitute another test instance
  • Each column after the identifier column is a result column: its column title represents a test environment, and its contents is one or more test results
  • Test results must use the Template:Result template, e.g. {{result|fail|adamwill|123456}} (to indicate a failure from the user adamwill with the related bug #123456)
    Comments may appear following the test result with which they are associated; all text between one test result and the next (or the end of the line) is considered a comment associated with that test result

The result text for a given page may be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no test name, or with the same test name) but they appear in separate sections, they are considered two separate test instances.

Test case pages

The contents of test case pages is not currently used by wikitcms the 'system'; it effectively simply expects that what they contain will be some sort of content from which a human being can generate a test result. The contents are 'parsed' only by the human testers.

In practice, it is strongly established that each page should contain a single test case using the Template:QA/Test_Case template, according to QA:SOP_test_case_creation.

Auxiliary pages

Auxiliary pages are those which are used in the process of generating the other pages in the system. They are not used as stores of results data directly. These pages are critical to the functioning of the system and care must be taken when editing them. Sensible simple edits to the text in the templates are usually fine, but please be careful about touching any of the critical wiki syntax. Please contact QA before making changes to any of these pages if you are at all unsure of the consequences.

Not all template pages and so on that are used by pages that form a part of the system are not really crucial to its operation; only those that would cause the system described here not to function correctly are considered auxiliary pages.

Result page generation system

The result page generation system consists of a set of nested templates. It is set up such that one complete result page can be generated using a single template, e.g. {{subst:Validation results|testtype=Installation|release=40|milestone=Alpha|compose=TC1}} to generate the Alpha TC1 Installation results page for Fedora 40. As long as it is invoked with correct parameters (i.e. a sensible Fedora release number, existing test type, valid milestone and compose or date), it should always generate result pages that comply with the requirements listed above. This style of invocation is the only intended interface to this mechanism; any other use of the component templates is not expected and may have unintended consequences. The wikitcms module, and the relval tool based upon it, generates result pages using this system.

The pages which make up the system are:

CurrentFedoraCompose page

Template:CurrentFedoraCompose defines the current compose: the TC, RC or nightly compose which is considered to be under active testing by the QA team at present. It can be used as a template with various parameters to display various elements of the current compose's identity, and it is also intended to be used as the authoritative source of that data by things interfacing with wikitcms as a 'system'.

The wikitcms module and relval use it in various ways, and its contents are usually expected to be updated by passing the --current parameter to relval when creating a new validation event. Hand editing the page is not recommended; its format is quite specific and mistakes may cause it to be processed incorrectly.

Things interfacing with wikitcms as a 'system' should expect each line in the page's wikitext that begins with a pipe symbol (|) to represent a different attribute of the 'current' compose's identification. The pipe symbol may be followed by any number of spaces (including none), then a string containing no spaces that identifies an attribute, then any number of spaces (including none), then an equals sign (=), then any number of spaces (including none), then a string which may contain spaces that specifies the value for that attribute. The value ends at the end of the line, or - if any number of spaces directly precedes the end of the line - at the last character before the first of those spaces. For example, the line:

|full   =22 Rawhide 20150107  

(which ends with two space characters) means the full attribute has the value 22 Rawhide 20150107.

The following attributes should always be present: full, release, milestone, compose, date. The value for one of compose or date should always be empty; compose being empty indicates that the current compose is a nightly build (in which case the value of milestone should be either Rawhide or Nightly), while date being empty indicates that the current compose is a TC or RC (in which case the value of milestone should be a valid release milestone). release should always be a valid Fedora release number. full combines all the non-empty attributes in a human-readable string defining the complete identification of the current compose.

Template:FedoraVersion and Template:FedoraVersionNumber

These two pages are used throughout the wiki and are not maintained as part of the wikitcms system, but they are used extensively by pages that form a part of wikitcms and so they are considered auxiliary pages for its purposes. Changes to their behaviour could cause unexpected results for wikitcms.

Supplementary pages

Supplementary pages are those that are provided only as conveniences for human interaction with the system, not for storage or organization of results data. Things that treat wikitcms as a 'system' should not usually rely on their existence, though in practice this assumption is usually safe.

Results summary page

For each validation test event (a TC, RC, or nominated nightly compose), there is a page named as above, but with Summary in place of the TESTTYPE. It is produced by transclusion of the result pages, which are set up (by the generation system) such that transcluding them shows only their result text. Thus the summary page combines the result text from all the result pages.

Current redirect pages

For each test type and also for the results summary page, a page exists in the Test Results namespace, named Current TESTTYPE Test (or Current Summary for the summary page). These pages should always redirect to the result page for the current compose, as defined by the Template:CurrentFedoraCompose page defined above. In the case of a conflict, Template:CurrentFedoraCompose should be treated as the authoritative definition of what is current, not the destinations of these redirect pages, though conflicts should happen rarely if at all so long as the relval utility is used - it updates both Template:CurrentFedoraCompose and the redirects, or neither.