From Fedora Project Wiki

< FWN‎ | Beats

(create fwn 234 qa beat)
(create fwn 235 qa beat)
Line 8: Line 8:
<references/>
<references/>


=== Proven testers ===
=== Instructions and process for moving bug reports upstream ===
 
The proven testers project was well underway, with most of the mentor requests being handled and many group members actively posting feedback. Aaron Farnes proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091860.html</ref> a substantial revision to the wiki page<ref>http://fedoraproject.org/wiki/User:Dafrito/Proven_tester</ref> which would incorporate information on joining the group (which was a separate page) and on the mentoring process (which was not yet documented), as well as rewriting the testing instructions. [[User:Jdulaney|John Dulaney]] liked it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091861.html</ref>, as did [[User:Jlaska|James Laska]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091879.html</ref>. [[User:Adamwill|Adam Williamson]] thought it was well laid out, but too wordy and too far abstracted to work as a clear set of instructions for prospective proven testers<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091887.html</ref>. Later, he posted<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091984.html</ref> an alternative draft<ref>http://fedoraproject.org/wiki/User:Adamwill/Draft_proventesters</ref> which made fewer changes from the existing page, adding in information on joining the group and on the mentoring process with minimal disruption to the existing content. Aaron liked it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091985.html</ref>.
 
[[User:MCloaked|Mike Cloaked]] generally liked Aaron's draft<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091938.html</ref>, but raised a question around kernel testing, and whether it was entirely safe for a single proven tester to 'approve' a kernel update when it was unlikely any single tester could come close to comprehensively testing a kernel. Adam thought<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091943.html</ref> that generally proven testers should approve updates which do not break critical path functions for them, but that it might make sense to develop a different policy for the kernel. Rick Stevens suggested requiring positive karma for each specific bug fix<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091954.html</ref>, but Adam explained that this was impossible under the current Bodhi system<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091956.html</ref>.


[[User:maxamillion|Adam Miller]] proposed two alternatives<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091881.html</ref> for sponsoring new proven testers - either having the whole group vote on proven tester candidates at weekly meetings, or allowing mentors to sponsor candidates when the mentor believes the candidate is ready. [[User:Adamwill|Adam Williamson]] was firmly in favour of the more liberal option<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091883.html</ref>, and with no disagreement, Adam Miller said he would go ahead and update the documentation to reflect this process<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091917.html</ref>. [[User:Jlaska|James Laska]] suggested holding the voting process in reserve in case a need to vet candidates more extensively transpired in future<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091918.html</ref>.
[[User:Ankursinha|Ankur Sinha]] began a discussion<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092018.html</ref>, <ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092036.html</ref> on providing instructions for users on the Wiki for reporting bugs to upstream bug trackers. [[User:Johannbg|Jóhann Guðmundsson]] felt that this was the wrong approach<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092040.html</ref> and that it should be the responsibility of package maintainers to submit reports upstream when appropriate. In general, though, the group felt that providing instructions for users would be helpful in at least some cases, and Ankur said he would work on a draft of such a page<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092091.html</ref>.


<references/>
<references/>


=== Musician's guide testing ===
=== Rawhide acceptance testing ===


[[User:Crantila|Christopher Antila]] wrote to let the group know<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091987.html</ref> that the Docs SIG had been working on a guide to creating music with Fedora<ref>http://fedoraproject.org/wiki/User:Crantila/FSC/Testing</ref>. He asked for help checking the consistency and language in the guide, and also for testers to try and follow the guide and provide feedback on whether it comprehensively covered the necessary information.
[[User:Jlaska|James Laska]] reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092095.html</ref> on the first automated Rawhide acceptance test plan<ref>http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan</ref> run for Fedora 14, and linked to the full results<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_14_Pre-Alpha_Rawhide_Acceptance_Test_1</ref>. The run exposed four bugs, which were significant enough that the test images could not be declared 'last known good'.


<references/>
<references/>


=== Installation validation test matrix update ===
=== Proven testers ===


In the Trac ticket<ref>http://fedorahosted.org/fedora-qa/ticket/95</ref>, [[User:Rhe|Rui He]] reported that she had made considerable progress on re-designing the installation validation test matrix<ref>http://fedoraproject.org/wiki/QA:Fedora_14_Install_Results_Template</ref>, including re-organizing the tests into categories and making the matrices for each category collapsible and re-orderable. [[User:Jlaska|James Laska]] thought the changes looked very good.
[[User:Adamwill|Adam Williamson]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092075.html</ref> that he had updated the Wiki proven testers material, making the main page<ref>http://fedoraproject.org/wiki/Proven_tester</ref> cover all aspects of the project (based on his previous draft mentioned in last week's issue) and turning the JoinProvenTesters page into a redirect to it. Later in the week he updated the instructions again<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092101.html</ref>.


<references/>
<references/>


=== Desktop validation testing expansion ===
=== AutoQA package acceptance testing ===


[[User:Adamwill|Adam Williamson]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092010.html</ref> a plan to expand desktop validation testing during the Fedora 14 cycle to the Xfce, LXDE and KDE desktops, as well as the default GNOME desktop. He explained that, on an experimental basis, the desktop validation test suite would have to be run against each desktop at each release validation test point, and any release criteria-breaking failure in any desktop would need to be fixed before the release could be made. He noted that he had contacted the leaders of the various desktop SIGs, and they were enthusiastic about the idea. [[User:Jdulaney|John Dulaney]] asked<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092011.html</ref> whether ISOs would be available for each desktop. Adam explained<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092012.html</ref> that each desktop can be installed from the DVD image, and that the automated nightly builds could also be used for testing.
During the QA weekly meeting of 2010-07-12<ref>http://fedoraproject.org/wiki/QA/Meetings/20100712</ref>, [[User:Wwoods|Will Woods]] explained that the AutoQA team had decided to consider automating the package update acceptance test plan<ref>http://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan</ref> as its highest priority. They felt this would provide a very visible and useful example of the potential of AutoQA as soon as possible. Automated package acceptance testing would prevent packages with certain types of serious problems, such as broken dependencies, from being accepted as updates. Will had identified clear areas of focus which are necessary to implement automated package acceptance testing, and these can be tracked on the AutoQA roadmap<ref>http://fedorahosted.org/autoqa/roadmap</ref>.


<references/>
<references/>


=== AutoQA ===
=== Localization testing ===


At the QA meeting, [[User:Wwoods|Will Woods]] reported that the AutoQA team was working on a helloworld test (a test test), which would exist to check that watchers and hooks - particularly the bodhi watcher and hook - work correctly. This is a prerequisite for the dependency check test, one of the major AutoQA priorities. [[User:Jskladan|Josef Skladanka]] said he had a test instance of the ResultsDB up and running on one of AutoQA's infrastructure machines, and had rewritten the initscripts and rpmlint tests to store their results in the database. He would continue to work on converting other tests. [[User:Kparal|Kamil Paral]] announced that he had patched autoqa to use autotest labels correctly, which allows us to configure the actual running of tests in several ways - ensuring they are run on particular machine configurations. He pointed to a mailing list post<ref>http://fedorahosted.org/pipermail/autoqa-devel/2010-June/000742.html</ref> with a more detailed explanation.
[[User:Igor|Igor Pires Soares]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092029.html</ref> that he would be working to update the Fedora 13 localization results template<ref>http://fedoraproject.org/wiki/QA:Fedora_13_l10n_Results_Template</ref> for Fedora 14 use at FUDCon Santiago, and asked for feedback and ideas on improving it. [[User:Jlaska|James Laska]] suggested<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092094.html</ref> a page to explain the general testing procedure, and asked where test results are recorded. Igor accepted the general procedure idea, and explained that the template was intended to allow local teams to record results in whatever way they felt most appropriate<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092096.html</ref>.


<references/>
<references/>


=== Triage metrics ===
=== Boot menu release criterion proposal ===


At the Bugzappers weekly meeting of 2010-07-06<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-07-06/bugzappers.2010-07-06-15.01.log.html</ref>, [[User:Jraber|Jeff Raber]] updated his progress with triage metrics. He had been updating the wiki page on his work<ref>http://fedoraproject.org/wiki/User_talk:Jraber</ref>, and working on a patch to python-bugzilla to allow querying bug history data, which is required for some of the planned metrics. [[User:Adamwill|Adam Williamson]] said he would try to take the metrics Jeff had already implemented and work them into a prototype weekly update email format to send to the list for feedback.
After a contentious first Fedora 14 blocker bug review meeting<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092147.html</ref>, [[User:Adamwill|Adam Williamson]] proposed two alternative ways to cover boot menu functionality in the release criteria<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092148.html</ref>. He suggested either having a basic test of essential functionality (the installer eventually boots without manual interaction) at the Alpha stage and a more advanced test (the graphical boot menu appears as intended) at Beta or Final stage, or simply having the more advanced test at Alpha stage. [[User:Jlaska|James Laska]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092152.html</ref> and [[User:Jkeating|Jesse Keating]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092150.html</ref> both came down in favour of simply requiring the menu to work correctly at Alpha stage.


<references/>
<references/>

Revision as of 01:46, 22 July 2010

QualityAssurance

In this section, we cover the activities of the QA team[1]. For more information on the work of the QA team and how you can get involved, see the Joining page[2].

Contributing Writer: Adam Williamson

  1. ↑ http://fedoraproject.org/wiki/QA
  2. ↑ http://fedoraproject.org/wiki/QA/Join

Instructions and process for moving bug reports upstream

Ankur Sinha began a discussion[1], [2] on providing instructions for users on the Wiki for reporting bugs to upstream bug trackers. Jóhann Guðmundsson felt that this was the wrong approach[3] and that it should be the responsibility of package maintainers to submit reports upstream when appropriate. In general, though, the group felt that providing instructions for users would be helpful in at least some cases, and Ankur said he would work on a draft of such a page[4].

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092018.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092036.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092040.html
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092091.html

Rawhide acceptance testing

James Laska reported[1] on the first automated Rawhide acceptance test plan[2] run for Fedora 14, and linked to the full results[3]. The run exposed four bugs, which were significant enough that the test images could not be declared 'last known good'.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092095.html
  2. ↑ http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan
  3. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_14_Pre-Alpha_Rawhide_Acceptance_Test_1

Proven testers

Adam Williamson announced[1] that he had updated the Wiki proven testers material, making the main page[2] cover all aspects of the project (based on his previous draft mentioned in last week's issue) and turning the JoinProvenTesters page into a redirect to it. Later in the week he updated the instructions again[3].

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092075.html
  2. ↑ http://fedoraproject.org/wiki/Proven_tester
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092101.html

AutoQA package acceptance testing

During the QA weekly meeting of 2010-07-12[1], Will Woods explained that the AutoQA team had decided to consider automating the package update acceptance test plan[2] as its highest priority. They felt this would provide a very visible and useful example of the potential of AutoQA as soon as possible. Automated package acceptance testing would prevent packages with certain types of serious problems, such as broken dependencies, from being accepted as updates. Will had identified clear areas of focus which are necessary to implement automated package acceptance testing, and these can be tracked on the AutoQA roadmap[3].

  1. ↑ http://fedoraproject.org/wiki/QA/Meetings/20100712
  2. ↑ http://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan
  3. ↑ http://fedorahosted.org/autoqa/roadmap

Localization testing

Igor Pires Soares announced[1] that he would be working to update the Fedora 13 localization results template[2] for Fedora 14 use at FUDCon Santiago, and asked for feedback and ideas on improving it. James Laska suggested[3] a page to explain the general testing procedure, and asked where test results are recorded. Igor accepted the general procedure idea, and explained that the template was intended to allow local teams to record results in whatever way they felt most appropriate[4].

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092029.html
  2. ↑ http://fedoraproject.org/wiki/QA:Fedora_13_l10n_Results_Template
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092094.html
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092096.html

Boot menu release criterion proposal

After a contentious first Fedora 14 blocker bug review meeting[1], Adam Williamson proposed two alternative ways to cover boot menu functionality in the release criteria[2]. He suggested either having a basic test of essential functionality (the installer eventually boots without manual interaction) at the Alpha stage and a more advanced test (the graphical boot menu appears as intended) at Beta or Final stage, or simply having the more advanced test at Alpha stage. James Laska[3] and Jesse Keating[4] both came down in favour of simply requiring the menu to work correctly at Alpha stage.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092147.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092148.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092152.html
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/092150.html
Retrieved from "https://fedoraproject.org/w/index.php?title=FWN/Beats/QualityAssurance&oldid=187641"