ResultsDB: Difference between revisions

Revision as of 06:38, 26 April 2011

This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

What is ResultsDB

ResultsDB is a system designed to hold the results of tests executed by the AutoQA framework.

Motivation behind ResultsDB

From the beginning of AutoQA development, all test results were stored on the autoqa-results mailing list ^[1]. While this is not a bad system for storing test results on a small scale, searching through these results (ie. find all test results for foobar-1.3-fc15) is overly difficult and cumbersome.

The primary motivation behind ResultsDB is to provide a simple, easily accessible interface to test results through simple querying API. ResultsDB is also capable of storing test results but this is a secondary objective.

Since ResultsDB is only a system for storing test results, different 'frontend' systems can be created using ResultsDB as a data source. These frontends could be a simple tool to aggregate recent results for packages (potentially a table for a package/update's results within bodhi or koji) or something more complicated like a tool for gathering test-related metrics (historical fail/pass ratio of a specific test in a Fedora release, failure rate of critpath packages etc.).

Current State of Development

At the time of this writing, most of the first version of ResultsDB has been completed:

The underlying database schema has been designed.^[2]
The data input API to be used for submitting test results has been implemented.^[3]
A proof-of-concept front end to display test results has been implemented using TurboGears2.^[4]
A second proof-of-concept frontend to create test plans (ie. The Package Update Acceptance Test Plan ^[5]) has been started^[6].

API

Down with direct SQL
During discussions, it was decided against using a direct SQL-based API. Instead, the preference was for a set of specific filter-methods with monitorable (?) arguments.

At the time of this writing, ResultsDB has a well defined API for storing results but the API for retrieving results has yet to be designed. The current thought behind the retrieval API is that it could be better designed once a proper production dataset exists.

When the Fedora Message Bus is deployed, that would provide another mechanism to monitor for and broadcast events.

Database Schema

The primary goal when designing the database schema was versatility. The desire was to have the ability to store test data from yet-to-be-created tests in addition to meeting the needs of 'other activities' that may become relevant in the future.

Job Table

For the immediate future, there are no plans to utilize the Job table in ResultsDB. That table is in the schema to allow for the ability to group several testruns (e.g. rpmguard and conflicts for a specific package). At the time of this writing, this type of grouping is not compatibly with current scheduling methods but the grouping can be easily taken care of with various frontend implementations.

Testrun Table

The Testrun table corresponds with a single execution of a test. Any data not covered by attributes in Testrun (such as tested package/update, architecture, etc) will be stored in the TestrunData table as key-value pairs. The potential and expected keys for these key-value pairs will be defined by test metadata. It is possible to include information about 'always present' and 'sometimes present' keys ^[7].

TestrunPhase Table

Testruns can be divided into phases (e.g. Setup phase of the test, downloading packages from koji) which are stored in the TestrunPhase table. If the test developer splits the test into multiple phases, each phase has its own result in addition to the global result stored in the Testrun table. Once tests are split up into phases, it will be possible to use test metadata to determine the severity of failures (e.g. a failure during the setup phase is not as severe as a failure during the test phase).

Generic Testplan Frontend

The testplan frontends are designed to be simple applications with an MVC ^[8] architecture.

Model - The data from ResultsDB makes up the model
Controller - Metadata stored in Mediawiki pages ^[9] makes up the Controller.
View - Simple frontend implementations^[6] make up the view.

It is anticipated that multiple testplan frontends will be created for ResultsDB and this architecture should be flexible enough to support the implementation of several independent frontends.

Example

A frontend was created for the Package Update Acceptance Testplan (PUATP) ^[5] as an example. There are multiple 'levels' of acceptance, based on the results of different tests. Many AutoQA tests are run on most, if not all, packages and the same data can be presented in different contexts to allow for easier understanding of that data.

At the time of this writing, FedoraQA is using MediaWiki for a TCMS. A method to store and retrieve test metadata on wiki pages was proposed^[10] and is currently being used for the more complicated PUATP template^[9].

The frontend ^[6] parses metadata from the wiki page, queries ResultsDB for test results from a specific NVR/testcase combination, and renders the information into an easily readable format.

Mediawiki Testcase Metadata

The semantics of metadata for the PUATP testcase ^[9] is quite simple.

testcases (required): A dictionary that connects the testcase name(key) to the URL of the testcase metadata (value). Testcase names (i.e. rpmlint or initscripts) are the primary method of reference beyond this dictionary.
testcase_classes (required): A list defining any number of 'classes'. Any class in this list must be farther defined in the metadata for the same testcase.
mandatory, introspection, advisory (test classes, user defined): Each test class is a dictionary which describes which testcases are to be taken into account, which results will be accepted, and what result should the test class have, if either of the tests is out of the specified set.
- testcases: List of testcase names that fall under the specified class
- pass: List of testcase results that qualify as 'accepted'. If all possible test results are in this set, the overall result of the test class will always be "PASSED".
- on_fail: Overall test class result if any of the specified testcases return a result not on the 'pass' list

Links

[1] ttps://fedorahosted.org/pipermail/autoqa-results/

[2] ttps://fedoraproject.org/wiki/AutoQA_resultsdb_schema

[3] ttps://fedoraproject.org/wiki/AutoQA_resultsdb_API

[4] ttps://www.assembla.com/code/resultsdb/git/nodes?rev=devel

[PUATP-5] 5.0 ^5.1 https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan

[git-repo-6] 6.0 ^6.1 ^6.2 https://www.assembla.com/code/resultsdb_puatp/git/nodes?rev=master

[testcase_metadata-7] ttps://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Metadata

[8] ttp://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

[PUATP_metadata-9] 9.0 ^9.1 ^9.2 https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Acceptance_Test_Plan_Metadata

[metadata_test_page-10] ttps://fedoraproject.org/wiki/QA:Test_Plan_Metadata_Test_Page

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

@@ Line 3: / Line 3: @@
 == What is ResultsDB ==
-ResultsDB is a database, in which we'll store results of tests executed by the AutoQA framework.
+<!-- ResultsDB is more than a database, and more of an information storage system with clearly defined non-SQL APIs -->
+ResultsDB is a system designed to hold the results of tests executed by the AutoQA framework.
 === Motivation behind ResultsDB ===
+From the beginning of AutoQA development, all test results were stored on the autoqa-results mailing list <ref>https://fedorahosted.org/pipermail/autoqa-results/</ref>. While this is not a bad system for storing test results on a small scale, searching through these results (ie. find all test results for foobar-1.3-fc15) is overly difficult and cumbersome.
-At the moment, all the results are stored on the autoqa-results mailing list <ref>https://fedorahosted.org/pipermail/autoqa-results/</ref>. Although this is not really a bad system for storing the results, searching for some particular result (even as simple query as "All results for foobar-1.3-fc15") is not really comfortable. ResultsDB's main purpose is not to store the data, but to provide easy access to the results via simple querying API.
+The primary motivation behind ResultsDB is to provide a simple, easily accessible interface to test results through simple querying API. ResultsDB is also capable of storing test results but this is a secondary objective.
-This means, that using the ResultsDB as database backend, many 'frontends' can be created. Be it simple tools which will just aggregate most recent results for specified package (maybe a table inside koji/bodhi info pages), some statistical tool for gathering fail/pass ratio of *insert test name of your choice* for given Fedora release, etc.
+Since ResultsDB is only a system for storing test results, different 'frontend' systems can be created using ResultsDB as a data source. These frontends could be a simple tool to aggregate recent results for packages (potentially a table for a package/update's results within bodhi or koji) or something more complicated like a tool for gathering test-related metrics (historical fail/pass ratio of a specific test in a Fedora release, failure rate of critpath packages etc.).
-== Current state ==
+== Current State of Development ==
-From development point of view, there is quite well-defined database schema <ref>https://fedoraproject.org/wiki/AutoQA_resultsdb_schema</ref>, 'input' API <ref>https://fedoraproject.org/wiki/AutoQA_resultsdb_API</ref>, and proof-of-concept TurboGears2 application <ref>https://www.assembla.com/code/resultsdb/git/nodes?rev=devel</ref>, implementing these two.
+At the time of this writing, most of the first version of ResultsDB has been completed:
+* The underlying database schema has been designed.<ref>https://fedoraproject.org/wiki/AutoQA_resultsdb_schema</ref>
+* The data input API to be used for submitting test results has been implemented.<ref>https://fedoraproject.org/wiki/AutoQA_resultsdb_API</ref>
+* A proof-of-concept front end to display test results has been implemented using TurboGears2.<ref>https://www.assembla.com/code/resultsdb/git/nodes?rev=devel</ref>
+* A second proof-of-concept frontend to create test plans (ie. The Package Update Acceptance Test Plan <ref name='PUATP'>https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan</ref>) has been started<ref name='git-repo'>https://www.assembla.com/code/resultsdb_puatp/git/nodes?rev=master</ref>.
-I have also started to work on a proof-of-concept frontend <ref name='git-repo'>https://www.assembla.com/code/resultsdb_puatp/git/nodes?rev=master</ref>, which will allow us to create test plans (like Package Update Acceptance Testplan <ref name='PUATP'>https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan</ref>), using mediawiki pages to define the testplan's requirements <ref name='metadata_test_page'>https://fedoraproject.org/wiki/QA:Test_Plan_Metadata_Test_Page</ref> <ref name='PUATP_metadata'>https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Acceptance_Test_Plan_Metadata</ref> (this will be covered in more detail further in the text).
+<!-- may use this later
+using mediawiki to store test plan requirements
+(example ) has been started
-=== API ===
+-->
+== API ==
-We have quite well-defined 'input' API - i.e. the API for storing data into ResultsDB. The 'output' API needs to be designed. I have not gone to that particular area that much, because at the time being. My plan was to sneak ResultsDB reporting into the production to gain some reasonable dataset, and then adjust the output API based on the actual needs of our frontend(s).
+{{admon/note|Down with direct SQL|During discussions, it was decided against using a direct SQL-based API.  Instead, the preference was for a set of specific filter-methods with monitorable (?) arguments.}}
+At the time of this writing, ResultsDB has a well defined API for storing results but the API for retrieving results has yet to be designed. The current thought behind the retrieval API is that it could be better designed once a proper production dataset exists.
+When the Fedora Message Bus is deployed, that would provide another mechanism to monitor for and broadcast events.
+== Database Schema ==
-{{admon/note|Down with direct SQL|During discussions, it was decided against using a direct SQL-based API.  Instead, the preference was for a set of specific filter-methods with monitorable (?) arguments.}}
+The primary goal when designing the database schema was versatility. The desire was to have the ability to store test data from yet-to-be-created tests in addition to meeting the needs of 'other activities' that may become relevant in the future.
-Also, once a Fedora Message Bus is available, the bus could be used to emit and monitor for event announcements.
+==== Job Table ====
+For the immediate future, there are no plans to utilize the Job table in ResultsDB. That table is in the schema to allow for the ability to group several testruns (e.g. rpmguard and conflicts for a specific package). At the time of this writing, this type of grouping is not compatibly with current scheduling methods but the grouping can be easily taken care of with various frontend implementations.
-=== Thoughts behind the database schema ===
+==== Testrun Table ====
+The Testrun table corresponds with a single execution of a test. Any data not covered by attributes in Testrun (such as tested package/update, architecture, etc) will be stored in the TestrunData table as key-value pairs. The potential and expected keys for these key-value pairs will be defined by test metadata. It is possible to include information about 'always present' and 'sometimes present' keys <ref name='testcase_metadata'>https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Metadata</ref>.
-It's safe to ignore the 'Job' idea at the moment - the idea behind Jobs was that we wanted to be able to internally group several test-runs (e.g. rpmgguard & conflicts for some particular package) into the job. This option is a) IMHO not really feasible with the current way of scheduling tests b) easily substituted with the 'generic testplan frontend' (see below).
+==== TestrunPhase Table ====
+Testruns can be divided into phases (e.g. Setup phase of the test, downloading packages from koji) which are stored in the TestrunPhase table. If the test developer splits the test into multiple phases, each phase has its own result in addition to the global result stored in the Testrun table. Once tests are split up into phases, it will be possible to use test metadata to determine the severity of failures (e.g. a failure during the setup phase is not as severe as a failure during the test phase).
-Other than that - our primary goal was versatility - we wanted to be able to store results not only from the tests we did have at the moment, but to be able to meet the needs of 'almost anything'.
+== Generic Testplan Frontend ==
-'Testrun' table corresponds with a single execution of a test, all the data not covered by it's attributes (such as tested package/update, architecture, etc) are to be stored in the 'TestrunData' as key-value pairs. Metadata for each test should define a set of used keyval pairs (with possible information about 'always present' and 'sometimes present' keys) - e.g. <ref name='testcase_metadata'>https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Metadata</ref>.
+The testplan frontends are designed to be simple applications with an MVC <ref>http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller</ref> architecture.
+* '''Model''' - The data from ResultsDB makes up the model
+* '''Controller''' - Metadata stored in Mediawiki pages <ref name='PUATP_metadata'>https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Acceptance_Test_Plan_Metadata</ref> makes up the Controller.
+* '''View''' - Simple frontend implementations<ref name='git-repo'/> make up the view.
-Testruns can be devided into phases (e.g. Setup phase of the test, downloading packages from koji...) - hence the TestrunPhase table. If the test developer decides to split the test into multiple phases, each phase can have it's result. Also metadata can be used to specify how to treat failures/warnings/etc from each phase - warning in setup phase might be less important than warning in run-once, etc.
+It is anticipated that multiple testplan frontends will be created for ResultsDB and this architecture should be flexible enough to support the implementation of several independent frontends.
-=== Generic Testplan frontend ===
+==== Example ====
+A frontend was created for the Package Update Acceptance Testplan (PUATP) <ref name='PUATP'/> as an example. There are multiple 'levels' of acceptance, based on the results of different tests. Many AutoQA tests are run on most, if not all, packages and the same data can be presented in different contexts to allow for easier understanding of that data.
-The idea behind this tool is quite simple - Model-View-Controller <ref>http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller</ref> architecture. ResultsDB will provide data (model), metadata on Mediawiki <ref name='PUATP_metadata'/> would be Controller, and the simple-logic frontend <ref name='git-repo'/> is the View part. Given that we probably will want to have multiple test plans, this architecture might suit us quite well.
+At the time of this writing, FedoraQA is using MediaWiki for a TCMS. A method to store and retrieve test metadata on wiki pages was proposed<ref name='metadata_test_page'>https://fedoraproject.org/wiki/QA:Test_Plan_Metadata_Test_Page</ref> and is currently being used for the more complicated PUATP template<ref name='PUATP_metadata'/>.
-The <ref name='PUATP'/> is great example. There are multiple 'levels' of acceptance, based on the results of different tests. Our multi-purpose test are run on most (if not all) of the packages anyway, so why not just use the data, and present it in different contexts?
+The frontend <ref name='git-repo'/> parses metadata from the wiki page, queries ResultsDB for test results from a specific NVR/testcase combination, and renders the information into an easily readable format.
-Our TCMS is MediaWiki at the moment, so wwoods proposed quite elegant way of storing (and retrieving, of course) metadata on some wiki page <ref name='metadata_test_page'/>.  I just used that idea to store a 'more complicated' kind of template <ref name='PUATP_metadata'/>.
+==== Mediawiki Testcase Metadata ====
-The semantics of metadata on <ref name='PUATP_metadata'/> is quite simple. Let's take this particular
+The semantics of metadata for the PUATP testcase <ref name='PUATP_metadata'/> is quite simple.
-* <code>testcases</code> (required): A dictionary, which connects name of the testcase (key), and the url of testcase metadata (value). Only testcase names (i.e. rpmlint, initscripts, etc) are used from this point further.
+* <code>testcases</code> (required): A dictionary that connects the testcase name(key) to the URL of the testcase metadata (value). Testcase names (i.e. rpmlint or initscripts) are the primary method of reference beyond this dictionary.
-* <code>testcase_classes</code> (required): A list, which defines any number of 'classes'. These are to be user-defined further in the metadata.
+* <code>testcase_classes</code> (required): A list defining any number of 'classes'. Any class in this list must be farther defined in the metadata for the same testcase.
 * <code>mandatory, introspection, advisory</code> (test classes, user defined): Each test class is a dictionary which describes which testcases are to be taken into account, which results will be accepted, and what result should the test class have, if either of the tests is out of the specified set.
-** <code>testcases</code>: List of testcase names to be used in this testclass
+** <code>testcases</code>: List of testcase names that fall under the specified class
-** <code>pass</code>: List of 'accepted' results. If all of the test results are in this set, overall test class result is "PASSED".
+** <code>pass</code>: List of testcase results that qualify as 'accepted'. If all possible test results are in this set, the overall result of the test class will always be "PASSED".
-** <code>on_fail</code>: If any of the test results is not in the 'pass' list, the overall test class result is the value defined here.
+** <code>on_fail</code>: Overall test class result if any of the specified testcases return a result not on the 'pass' list
-The frontend <ref name='git-repo'/> then simply parses the metadata, queries ResultsDB for test results for given NVR/testcase combination, and displays it.
 == Links ==
 <references/>

Search