From Fedora Project Wiki

< QA

(Frantisek, this was a wrong place to put it, that was about test cases. Let's include in instructions in the wiki template instead.)
(15 intermediate revisions by 5 users not shown)
Line 3: Line 3:
== Get people on board ==
== Get people on board ==


Make sure all involved parties are aware of the Test Day. Usually, you would want the maintainer(s) of the affected component(s) to be aware of the Test Day and, ideally, present when it happens. If you know of experienced users of the component(s) who will likely be able both to provide useful feedback at the Test Day and bring in other users, it will help to bring them on board in advance, too. If the affected component has a triager - see the BugZappers team's [[BugZappers/Components_and_Triagers|list of components and triagers]] - bring them in too.
Make sure all involved parties are aware of the Test Day. Usually, you would want the maintainer(s) of the affected component(s) to be aware of the Test Day and, ideally, present when it happens. If you know of experienced users of the component(s) who will likely be able both to provide useful feedback at the Test Day and bring in other users, it will help to bring them on board in advance, too.


== Set a date ==
== Set a date ==
Line 17: Line 17:
== Create the Wiki page ==
== Create the Wiki page ==


This will be the central source of information regarding your Test Day: make sure it's complete, accurate and up to date at all times. Use this [[QA/Test_Days/Template|Test Day template]] as a base: copy and paste the source of that page and fill it in for your event. Your page should be given a name of the form Test_Day:(DATE)_(TOPIC) - e.g. Test_Day:2009-05-07_Virtualization . It should be in the Test Days category. You can add your event to the Test Day schedule for the appropriate release. Release Test Day schedules are linked to from the [[QA/Test_Days|main Test Days page]].
This will be the central source of information regarding your Test Day: make sure it's complete, accurate and up to date at all times. Use this [[QA/Test_Days/Template|Test Day template]] as a base: copy and paste the source of that page and fill it in for your event. Your page should be given a name of the form Test_Day:(DATE)_(TOPIC) - e.g. Test_Day:2014-09-25_Virtualization . It should be in the category for Test Days for the pending release - for instance, [[:Category:{{FedoraVersion|long|next}} Test Days]]. You can add your event to the Test Day schedule for the appropriate release. Release Test Day schedules are linked to from the [[QA/Test_Days|main Test Days page]].


As a goal, your Wiki page should provide enough information that someone could perform all the testing and provide their results without any other reference or help.
As a goal, your Wiki page should provide enough information that someone could perform all the testing and provide their results without any other reference or help.
Line 24: Line 24:


A test case is simply a precise set of instructions which cause a reliably repeatable operation to occur which allows you to determine some kind of information from the results of the operation. There is a [[Template:QA/Test_Case|template]] to be used for creating test cases: if you visit that page, you will see a link which explains how to use this template to create a test case. You can see example test cases in the [[:Category:Test_Cases|test case category page]]. When creating a test case, try to make sure the instructions are explicit enough that anyone who may want to participate in the test day can follow them, and entirely unambiguous so that the user cannot perform the test incorrectly. Don't cram multiple operations into a single test case: create separate test cases for each different operation you wish to test. If possible, try to create the Test Case in such a way that it will be re-usable in the future (ask yourself if someone running an edition of Fedora from five years after you write the test case would still be able to follow it), although this is not always possible.
A test case is simply a precise set of instructions which cause a reliably repeatable operation to occur which allows you to determine some kind of information from the results of the operation. There is a [[Template:QA/Test_Case|template]] to be used for creating test cases: if you visit that page, you will see a link which explains how to use this template to create a test case. You can see example test cases in the [[:Category:Test_Cases|test case category page]]. When creating a test case, try to make sure the instructions are explicit enough that anyone who may want to participate in the test day can follow them, and entirely unambiguous so that the user cannot perform the test incorrectly. Don't cram multiple operations into a single test case: create separate test cases for each different operation you wish to test. If possible, try to create the Test Case in such a way that it will be re-usable in the future (ask yourself if someone running an edition of Fedora from five years after you write the test case would still be able to follow it), although this is not always possible.
It's always a good idea to also include the [[QA:Testcase Exploratory Testing|Exploratory Testing]] test case, to motivate people to try different things and not just lock themselves into the prepared test cases.


Link to each of the test cases from the How to Test section of the Wiki page.
Link to each of the test cases from the How to Test section of the Wiki page.


=== Set up the TestDayApp ===
=== Set up the Test Day App ===


<!--This section is up for edit. If you have any changes you would like to suggest, go for it! -->
<!--This section is up for edit. If you have any changes you would like to suggest, go for it! -->


The results for test cases are stored and managed in the [http://testdays.qa.fedoraproject.org/testdays/all_events TestDayApp]. The TestDayApp takes metadata from your test day's metadata page and creates an easy to use interface for submitting test results. An explanation of the required metadata is on [http://fedoraproject.org/wiki/User:Jskladan/Sandbox:TestdayAppMetadataDescription Jskladan's page] while an example can be found [https://fedoraproject.org/wiki/Test_Day:2013-10-23_Graphics_Radeon_TestdayApp_Metadata here].  
Test Day results can be stored and managed in the [https://testdays.fedoraproject.org/events TestDayApp]. The app takes metadata from your test day's metadata page and creates an easy to use interface for submitting test results. There is [[QA:TestdayAppMetadataDescription|an explanation of the required metadata]]. An example can be found at [[Test_Day:2013-10-23_Graphics_Radeon_TestdayApp_Metadata]].


==== Step 1 ====
==== Step 1 ====


Create the test case matrix with your required test cases you documented earlier. Put this matrix on the wiki page for your test day. An example is below.
Create the required metadata for the test day. Go to the [https://testdays.fedoraproject.org/admin/add_event create/update page], enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and a new entry in the app will be created.


===== Example Test Cases =====
Should you update the metadata, once again visit [https://testdays.fedoraproject.org/admin/add_event the create/update page], enter the full URL for the metadata and click 'Submit'. As long as you keep the URLs for test cases the same, the results will be intact.
 
{|
! User
! [[QA:Testcase 1|Initial Setup]]
! [[QA:Testcase 2|Login Screen]]
! [[QA:Testcase 3|Activities]]
! [[QA:Testcase 4|Lock Screen]]
! References
|-
| [[User:SampleUser|Sample User]]
| {{result|none}} 
| {{result|pass}}
| {{result|warn}} <ref>Test pass, but also encountered {{bz|54321}}</ref>
| {{result|fail}} <ref>{{bz|12345}}</ref>
| <references/>
|-
|}


==== Step 2 ====
==== Step 2 ====


Create the required metadata for the test day. These test cases should match the matrix you created earlier - so you can paste in the results after the test day is complete.
Once the Test Day is completed you should export the results from the app. Go to the [https://testdays.fedoraproject.org/admin/export_event export page], enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and you'll be presented with the results of your test day in wiki format.


==== Step 3 ====
==== Step 3 ====


Once the test day is completed you should export the results from the TestDayApp. Go to the [http://testdays.qa.fedoraproject.org/testdays/admin/export_testday export page], enter the full url for the metadata of your test day and click 'Submit.' Your browser will reload and you'll be presented with the results of your test day in wiki format.
Copy the wiki-formatted results from step 2 and paste them into your Test Day wiki page. Use the "Show Preview" button to check your formatting.


==== Step 4 ====
== Create a calendar event ==


Update the test case matrix you created in step 1. Copy the wiki-formatted results from step 3 and paste them after the "Sample User." Use the "Show Preview" button to check your formatting.
For each test day we want to have a corresponding event in our [https://apps.fedoraproject.org/calendar/QA/ QA calendar]. Each event '''must have''' the string <code>Test Day</code> in its title, so that it shows up in our [https://apps.fedoraproject.org/calendar/list/QA/?subject=Test+Day test day schedule]. The description supports Markdown, but it's better to use plaintext, because while Fedocal shows Markdown formatting, most other calendar software (Google Calendar, Thunderbird, etc) don't. The description needs to contain the link to the test day page, and ideally also a paragraph or two explaining what this test day is about.


== Build a live CD ==
== Build a live image ==


If some or all of the test cases can be successfully performed from a live boot, it is highly recommended to provide a live CD image. Instructions on doing so can be found [[QA/Test_Days/Live_Image|here]]. Remember to customize the spin to include whatever application you want to test, and any utilities that are needed for testing it. Also make these available through a repository if they're not in rawhide, for people who want to test from their rawhide system rather than the live CD. Put the live CD image up in a fedorapeople space; contact the Infrastructure Team to extend your quota if necessary.
If some or all of the test cases can be successfully performed from a live boot, it is highly recommended to provide a live image. Instructions on doing so can be found [[QA/Test_Days/Live_Image|here]]. Remember to customize the spin to include whatever application you want to test, and any utilities that are needed for testing it. Also make these available through a repository if they're not in the [[Repositories|official repositories]], for people who want to test from their installed system rather than the live image. Put the live image image up in a [[Infrastructure/fedorapeople.org|fedorapeople.org]] space. Contact the [[Infrastructure]] team to extend your quota if necessary.


== Promote the Test Day ==
== Promote the Test Day ==


* Blog about it (and make sure your blog's on Planet Fedora). '''Multiple blog posts are good!''' Don't just have one person blog - have as many as possible, and you can blog more than once if you like (once two days ahead of the event and once on the day itself, for instance)
* Blog about it (and make sure your blog's on Planet Fedora). '''Multiple blog posts are good!''' Don't just have one person blog - have as many as possible, and you can blog more than once if you like (once two days ahead of the event and once on the day itself, for instance)
* Post to social media - Diaspora, Twitter, Google+, Facebook, the more the merrier - using the #fedora hashtag, and the Fedora and/or Fedora QA groups for the platforms that have them
* Send an announcement mail to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list
* Send an announcement mail to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list
* Contact the [[FWN/Beats/QualityAssurance|Fedora Weekly News QA beat]] author to get news about your event added to FWN. FWN is written over the weekend, so make sure to make contact before the weekend before your event
* Contact the Fedora Magazine team and ask them to post about the Test Day - link to your blog and/or social media posts about the event. You can use their [http://fedoramagazine.org/submit-an-idea-or-tip/ contact form], but it might be faster to contact the [[Marketing#Mailing_List|marketing mailing list]] or the {{fpchat|#fedora-mktg}} IRC channel
* If it affects a specific country, contact the Ambassador(s) for that country. If it affects a specific community, contact the places where that community gathers - IRC, forums, whatever
* If it affects a specific country, contact the Ambassador(s) for that country. If it affects a specific community, contact the places where that community gathers - IRC, forums, whatever
* If it's possibly of interest to a wider community than just Fedora users (make sure you have a live CD, for this one!), submit it to some general interest news sites. Don't be scared - news sites love content. They live off it. They have a very low bar for it. Try [http://www.osnews.com OS News], [http://www.linuxtoday.com Linux Today], [http://www.lwn.net LWN], and [http://www.lxer.com LXer] for everything, and [http://www.phoronix.com Phoronix] for anything that a home enthusiast community is likely to be interested in. You should also get a post in the [http://www.fedoraforum.org Fedora Forums] news section, but you'll need someone with posting privileges to do that
* If it's possibly of interest to a wider community than just Fedora users (make sure you have a live image, for this one!), submit it to some general interest news sites. Don't be scared - news sites love content. They live off it. They have a very low bar for it. Try [http://www.osnews.com OS News], [http://www.linuxtoday.com Linux Today], [http://www.lwn.net LWN], and [http://www.lxer.com LXer] for everything, and [http://www.phoronix.com Phoronix] for anything that a home enthusiast community is likely to be interested in. You should also get a post in the [http://www.fedoraforum.org Fedora Forums] news section, but you'll need someone with posting privileges to do that
* If you are a Red Hatter, send the announcement also to the internal QE mailing list.
* If you are a Red Hatter, send the announcement also to the internal QE mailing list.


Line 93: Line 79:
== Do a post-event review ==
== Do a post-event review ==


Doing a review of the Test Day once it's finished helps make it clear what the Test Day achieved, acts as a useful reference for issues discovered, and helps to promote the Test Day process to other users. A good way to write the review is to base it around a generated list of the bugs reported as a result of the Test Day. You can generate such a list by running this command:
Doing a review of the Test Day once it's finished helps make it clear what the Test Day achieved, acts as a useful reference for issues discovered, and helps to promote the Test Day process to other users.  
<pre>
 
curl "https://fedoraproject.org/wiki/Test_Day:Current" \
The [https://www.happyassassin.net/testdays testdays] app can help you with this, by generating some statistics about the Test Day and a list of the bugs reported. Follow the instructions on that page to enable the repository where it lives and install it. To get the information for a Test Day that took place on 2015-06-01, you would run: {{command|testdays stats --since 2015-05-31 --until 2015-06-02 --list-bugs}}
| grep -o "bugzilla\.redhat\.com.*[=\/][0-9]\{6\}" \
 
| grep -o "[0-9]\{6\}" | tr '\n' ',' | sed 's|,$||' \
You should do this after [[#Set up the TestDayApp|copying the results from the Test Day app back into the Wiki page]], or else it will yield no results. The command will produce a list of bugs referred in the page, and some statistics on the total number of testers, tests, and bugs. Note that the ''Fixed %'' it reports is most useful for a Test Day that took place some time ago, to see how many of the bugs reported were fixed - shortly after the Test Day takes place, that number is naturally likely to be very low. You might find it useful, for instance, if you run a regular Test Day on the same topic for multiple releases - perhaps when writing the report for the {{FedoraVersion|long|next}} Test Day, you could run the command for the {{FedoraVersion|long|current}} Test Day and report how many of the bugs from that event were fixed.
| xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc}" -b
 
</pre>
If you're not sure what a review email should look like, [http://www.redhat.com/archives/fedora-test-list/2009-August/msg00377.html here's an example]. You should send the report to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list, and perhaps to other mailing lists related to the topic of the Test Day.
The command will produce a list of bugs whose bug report URLs are in the Test Day page, with status information on each bug. (The second grep command is a safety valve because the first can give messy results when people mention multiple bug reports in a single line). You may have to weed out any 'false' references to pre-existing bugs which are mentioned on the page for some reason. This command requires the {{package|python-bugzilla}} package be installed. If you're not sure what a review email should look like, [http://www.redhat.com/archives/fedora-test-list/2009-August/msg00377.html here's an example]. You should send the report to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list, and perhaps to other mailing lists related to the topic of the Test Day.


=== Who attended? ===
=== Who attended? ===

Revision as of 11:08, 25 November 2020

This page explains how to go about arranging a Test Day event. If you have any questions or concerns about the Test Day process, please contact the QA group.

Get people on board

Make sure all involved parties are aware of the Test Day. Usually, you would want the maintainer(s) of the affected component(s) to be aware of the Test Day and, ideally, present when it happens. If you know of experienced users of the component(s) who will likely be able both to provide useful feedback at the Test Day and bring in other users, it will help to bring them on board in advance, too.

Set a date

Aim for a day when:

  • Most involved people (organizer, maintainers, experienced users, triagers, QA team) can be available
  • The feature will be close enough to complete for the testing to mean something, but outside of any deep related freeze periods
  • No other event which would be likely to draw away users with an interest in the Test Day is happening - including other Test Days!

Try to be flexible when it comes to the time the Test Day actually happens: consider timezone issues, especially if many users of the feature are in a particular geographical area. Ideally, aim to have developers or experienced users present throughout the entire period when it is the date of the Test Day somewhere in the world.

Create the Wiki page

This will be the central source of information regarding your Test Day: make sure it's complete, accurate and up to date at all times. Use this Test Day template as a base: copy and paste the source of that page and fill it in for your event. Your page should be given a name of the form Test_Day:(DATE)_(TOPIC) - e.g. Test_Day:2014-09-25_Virtualization . It should be in the category for Test Days for the pending release - for instance, Category:Fedora 40 Test Days. You can add your event to the Test Day schedule for the appropriate release. Release Test Day schedules are linked to from the main Test Days page.

As a goal, your Wiki page should provide enough information that someone could perform all the testing and provide their results without any other reference or help.

Create test cases

A test case is simply a precise set of instructions which cause a reliably repeatable operation to occur which allows you to determine some kind of information from the results of the operation. There is a template to be used for creating test cases: if you visit that page, you will see a link which explains how to use this template to create a test case. You can see example test cases in the test case category page. When creating a test case, try to make sure the instructions are explicit enough that anyone who may want to participate in the test day can follow them, and entirely unambiguous so that the user cannot perform the test incorrectly. Don't cram multiple operations into a single test case: create separate test cases for each different operation you wish to test. If possible, try to create the Test Case in such a way that it will be re-usable in the future (ask yourself if someone running an edition of Fedora from five years after you write the test case would still be able to follow it), although this is not always possible.

It's always a good idea to also include the Exploratory Testing test case, to motivate people to try different things and not just lock themselves into the prepared test cases.

Link to each of the test cases from the How to Test section of the Wiki page.

Set up the Test Day App

Test Day results can be stored and managed in the TestDayApp. The app takes metadata from your test day's metadata page and creates an easy to use interface for submitting test results. There is an explanation of the required metadata. An example can be found at Test_Day:2013-10-23_Graphics_Radeon_TestdayApp_Metadata.

Step 1

Create the required metadata for the test day. Go to the create/update page, enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and a new entry in the app will be created.

Should you update the metadata, once again visit the create/update page, enter the full URL for the metadata and click 'Submit'. As long as you keep the URLs for test cases the same, the results will be intact.

Step 2

Once the Test Day is completed you should export the results from the app. Go to the export page, enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and you'll be presented with the results of your test day in wiki format.

Step 3

Copy the wiki-formatted results from step 2 and paste them into your Test Day wiki page. Use the "Show Preview" button to check your formatting.

Create a calendar event

For each test day we want to have a corresponding event in our QA calendar. Each event must have the string Test Day in its title, so that it shows up in our test day schedule. The description supports Markdown, but it's better to use plaintext, because while Fedocal shows Markdown formatting, most other calendar software (Google Calendar, Thunderbird, etc) don't. The description needs to contain the link to the test day page, and ideally also a paragraph or two explaining what this test day is about.

Build a live image

If some or all of the test cases can be successfully performed from a live boot, it is highly recommended to provide a live image. Instructions on doing so can be found here. Remember to customize the spin to include whatever application you want to test, and any utilities that are needed for testing it. Also make these available through a repository if they're not in the official repositories, for people who want to test from their installed system rather than the live image. Put the live image image up in a fedorapeople.org space. Contact the Infrastructure team to extend your quota if necessary.

Promote the Test Day

  • Blog about it (and make sure your blog's on Planet Fedora). Multiple blog posts are good! Don't just have one person blog - have as many as possible, and you can blog more than once if you like (once two days ahead of the event and once on the day itself, for instance)
  • Post to social media - Diaspora, Twitter, Google+, Facebook, the more the merrier - using the #fedora hashtag, and the Fedora and/or Fedora QA groups for the platforms that have them
  • Send an announcement mail to the test-announce mailing list
  • Contact the Fedora Magazine team and ask them to post about the Test Day - link to your blog and/or social media posts about the event. You can use their contact form, but it might be faster to contact the marketing mailing list or the #fedora-mktg[?] IRC channel
  • If it affects a specific country, contact the Ambassador(s) for that country. If it affects a specific community, contact the places where that community gathers - IRC, forums, whatever
  • If it's possibly of interest to a wider community than just Fedora users (make sure you have a live image, for this one!), submit it to some general interest news sites. Don't be scared - news sites love content. They live off it. They have a very low bar for it. Try OS News, Linux Today, LWN, and LXer for everything, and Phoronix for anything that a home enthusiast community is likely to be interested in. You should also get a post in the Fedora Forums news section, but you'll need someone with posting privileges to do that
  • If you are a Red Hatter, send the announcement also to the internal QE mailing list.

It's probably best to announce the test day about a week in advance, and do a 'quick reminder' post the day before.

Pre-event reminder

Remind the involved parties - maintainers, experienced users, triagers, QA team - about the event shortly before it happens, to make sure they show up. And remember to show up yourself!

Update Test_Day:Current

Update the Test_Day:Current wiki redirect. The Test_Day:Current page is used to conveniently direct testers to the most current test day. To update the page, login to the wiki and click here.

Do a post-event review

Doing a review of the Test Day once it's finished helps make it clear what the Test Day achieved, acts as a useful reference for issues discovered, and helps to promote the Test Day process to other users.

The testdays app can help you with this, by generating some statistics about the Test Day and a list of the bugs reported. Follow the instructions on that page to enable the repository where it lives and install it. To get the information for a Test Day that took place on 2015-06-01, you would run: testdays stats --since 2015-05-31 --until 2015-06-02 --list-bugs

You should do this after copying the results from the Test Day app back into the Wiki page, or else it will yield no results. The command will produce a list of bugs referred in the page, and some statistics on the total number of testers, tests, and bugs. Note that the Fixed % it reports is most useful for a Test Day that took place some time ago, to see how many of the bugs reported were fixed - shortly after the Test Day takes place, that number is naturally likely to be very low. You might find it useful, for instance, if you run a regular Test Day on the same topic for multiple releases - perhaps when writing the report for the Fedora 40 Test Day, you could run the command for the Fedora 39 Test Day and report how many of the bugs from that event were fixed.

If you're not sure what a review email should look like, here's an example. You should send the report to the test-announce mailing list, and perhaps to other mailing lists related to the topic of the Test Day.

Who attended?

Test Days are promoted in the community and are community events. At some Test Days there is often a number of the developers/QA members attending, often Red Hat employees, so you might be interested in other members of the community who gave up some of their free time and contributed in testing.

For events with great number of participants, when it is difficult to weed out who is who (can't really say based on what testers write in the User: field unless they have a descriptive profile), you can use this program to generate the list of attendees and (optionally) check it up against the Red Hat roster (you will need python 2.7, check --help for more info)