From Fedora Project Wiki

< CI

Revision as of 14:34, 12 April 2018 by Psss (talk | contribs) (The first five attributes)

Requirements

In order to use the Flexible Metadata Format effectively for the CI testing we need to agree on the essential set of attributes to be used. For each attribute we need to standardize:

  • Name ... unique, well chosen, possibly with a prefix
  • Type ... expected content type: string, number, list, dictionary
  • Purpose ... description of the attribute purpose

For names we should probably consider using namespace prefix (such as test-description, requirement-description) to prevent future collisions with other attributes. Each attribute definition should contain at least one apt example of the usage. Or better, a set of user stories to be covered.

Attributes

In this section there are attributes proposed so far. Material for discussion. Nothing final for now.

Summary

In order to efficiently collaborate on test maintenance it's crucial to have a short summary of what the test does.

  • Name ... summary, test-summary
  • Type ... string (one line)
  • Purpose ... concise summary of what the test does

User stories:

  • As a developer reviewing 10 failed tests I would like to get quickly an idea of what my change broke.

Notes:

  • Shall we recommend 50 characters or less? Like there is for commits?

Description

For complex tests it makes sense to provide more detailed description to better clarify what is covered by the test.

  • Name ... description, test-description
  • Type ... string (multi line)
  • Purpose ... detailed description of what the test does

User stories:

  • As a tester I come to a test code I wrote 10 years ago (so I have absolutely no idea about it) and would like to quickly understand what it does.
  • As a developer I review existing test coverage for my component a would like to get an overall idea what is covered without having to read the whole test code.

Tags

Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects.

  • Name: tags
  • Type: list
  • Purpose: free-form tags for easy filtering

Notes:

  • Tags should be case-sensitive. Or not?

User stories:

  • A a developer/tester I would like to run only a subset of available tests.

Test

This is the key content attribute defining how the test is to be executed.

  • Name: test
  • Type: string
  • Purpose: shell command which executes the test

User stories:

  • As a developer/tester I want to easily execute all available tests with just one command.
  • As a test writer I want to run a single test script in multiple ways (e.g. providing different parameters)

Path

As the object hierarchy does not need to copy the filesystem structure (e.g. virtual test cases) we need a way how to define where the test is located.

  • Name: path
  • Type: string
  • Purpose: filesystem directory to be entered before executing the test

User stories:

  • As a test writer I define two virtual test cases, both using the same script for executing.