From Fedora Project Wiki

< CI

(Improve examples)
(Moved to Pagure.)
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Requirements =
Fedora CI metadata specification moved to https://pagure.io/fedora-ci/metadata.
 
In order to use the [[Flexible_Metadata_Format|Flexible Metadata Format]] effectively for the CI testing we need to agree on the essential set of attributes to be used. For each attribute we need to standardize:
 
* '''Name''' ... unique, well chosen, possibly with a prefix
* '''Type''' ... expected content type: string, number, list, dictionary
* '''Purpose''' ... description of the attribute purpose
 
For names we should probably consider using namespace prefix (such as test-description, requirement-description) to prevent future collisions with other attributes. Each attribute definition should contain at least one apt example of the usage. Or better, a set of user stories to be covered.
 
= Attributes =
 
In this section there are attributes proposed so far. Material for discussion. Nothing final for now.
 
== Summary ==
 
In order to efficiently collaborate on test maintenance it's crucial to have a short summary of what the test does.
 
* Name ... summary, test-summary
* Type ... string (one line)
* Purpose ... concise summary of what the test does
 
User stories:
 
* As a developer reviewing 10 failed tests I would like to get quickly an idea of what my change broke.
 
Notes:
 
* Shall we recommend 50 characters or less? Like there is for [https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting commits]?
 
Example:
 
summary: Test wget recursive download options
 
== Description ==
 
For complex tests it makes sense to provide more detailed description to better clarify what is covered by the test.
 
* Name ... description, test-description
* Type ... string (multi line)
* Purpose ... detailed description of what the test does
 
User stories:
 
* As a tester I come to a test code I wrote 10 years ago (so I have absolutely no idea about it) and would like to quickly understand what it does.
* As a developer I review existing test coverage for my component a would like to get an overall idea what is covered without having to read the whole test code.
 
Example:
 
description: |
  This test checks all available wget options related to
  downloading files recursively. First a tree directory
  structure is created for testing. Then a file download
  is performed for different recursion depth specified by
  the "--level=depth" option.
 
== Tags ==
 
Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects.
 
* Name: tags
* Type: list
* Purpose: free-form tags for easy filtering
 
Notes:
 
* Tags should be case-sensitive. Or not?
 
User stories:
 
* A a developer/tester I would like to run only a subset of available tests.
 
Example:
 
tags: [Tier1, fast]
 
== Test ==
 
This is the [http://fmf.readthedocs.io/en/latest/features.html#leaves key content] attribute defining how the test is to be executed.
 
* Name: test
* Type: string
* Purpose: shell command which executes the test
 
User stories:
 
* As a developer/tester I want to easily execute all available tests with just one command.
* As a test writer I want to run a single test script in multiple ways (e.g. providing different parameters)
 
Example:
 
test: ./runtest.sh
 
== Path ==
 
As the object hierarchy does not need to copy the filesystem structure (e.g. [http://fmf.readthedocs.io/en/latest/features.html#virtual virtual] test cases) we need a way how to define where the test is located.
 
* Name: path
* Type: string
* Purpose: filesystem directory to be entered before executing the test
 
User stories:
 
* As a test writer I define two virtual test cases, both using the same script for executing.
 
Example:
 
path: wget/recursion
 
 
= Examples =
 
== BeakerLib Tests ==
 
Three beakerlib tests, each in it's own directory:
 
main.fmf
 
  test: ./runtest.sh
  /one:
      path: tests/one
  /two:
      path: tests/two
  /three:
      path: tests/three
 
fmf
 
  tests/one
  path: tests/one
  test: ./runtest.sh
 
  tests/two
  path: tests/two
  test: ./runtest.sh
 
  tests/three
  path: tests/three
  test: ./runtest.sh
 
== Three Scripts ==
 
Three different script residing in a single directory:
 
main.fmf
 
path: tests
/one:
    test: ./one
/two:
    test: ./two
/three:
    test: ./three
 
fmf
 
tests/one
path: tests
test: ./one
 
tests/two
path: tests
test: ./two
 
tests/three
path: tests
test: ./three
 
== Virtual Tests ==
 
Thre virtual test cases based on a single test script:
 
main.fmf
 
path: tests/virtual
/one:
    test: ./script --one
/two:
    test: ./script --two
/three:
    test: ./script --three
 
fmf
 
tests/one
path: tests
test: ./script --one
 
tests/two
path: tests
test: ./script --two
 
tests/three
path: tests
test: ./script --three

Latest revision as of 14:15, 18 October 2018

Fedora CI metadata specification moved to https://pagure.io/fedora-ci/metadata.