(create fwn 233 qa beat) |
(create fwn 234 qa beat) |
||
| Line 10: | Line 10: | ||
=== Proven testers === | === Proven testers === | ||
The proven testers project was well underway, with most of the mentor requests being handled and many group members actively posting feedback. Aaron Farnes proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091860.html</ref> a substantial revision to the wiki page<ref>http://fedoraproject.org/wiki/User:Dafrito/Proven_tester</ref> which would incorporate information on joining the group (which was a separate page) and on the mentoring process (which was not yet documented), as well as rewriting the testing instructions. [[User:Jdulaney|John Dulaney]] liked it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091861.html</ref>, as did [[User:Jlaska|James Laska]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091879.html</ref>. [[User:Adamwill|Adam Williamson]] thought it was well laid out, but too wordy and too far abstracted to work as a clear set of instructions for prospective proven testers<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091887.html</ref>. Later, he posted<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091984.html</ref> an alternative draft<ref>http://fedoraproject.org/wiki/User:Adamwill/Draft_proventesters</ref> which made fewer changes from the existing page, adding in information on joining the group and on the mentoring process with minimal disruption to the existing content. Aaron liked it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091985.html</ref>. | |||
[[User:MCloaked|Mike Cloaked]] generally liked Aaron's draft<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091938.html</ref>, but raised a question around kernel testing, and whether it was entirely safe for a single proven tester to 'approve' a kernel update when it was unlikely any single tester could come close to comprehensively testing a kernel. Adam thought<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091943.html</ref> that generally proven testers should approve updates which do not break critical path functions for them, but that it might make sense to develop a different policy for the kernel. Rick Stevens suggested requiring positive karma for each specific bug fix<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091954.html</ref>, but Adam explained that this was impossible under the current Bodhi system<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091956.html</ref>. | |||
[[User:maxamillion|Adam Miller]] proposed two alternatives<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091881.html</ref> for sponsoring new proven testers - either having the whole group vote on proven tester candidates at weekly meetings, or allowing mentors to sponsor candidates when the mentor believes the candidate is ready. [[User:Adamwill|Adam Williamson]] was firmly in favour of the more liberal option<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091883.html</ref>, and with no disagreement, Adam Miller said he would go ahead and update the documentation to reflect this process<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091917.html</ref>. [[User:Jlaska|James Laska]] suggested holding the voting process in reserve in case a need to vet candidates more extensively transpired in future<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091918.html</ref>. | |||
<references/> | |||
=== Musician's guide testing === | |||
[[User:Crantila|Christopher Antila]] wrote to let the group know<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/091987.html</ref> that the Docs SIG had been working on a guide to creating music with Fedora<ref>http://fedoraproject.org/wiki/User:Crantila/FSC/Testing</ref>. He asked for help checking the consistency and language in the guide, and also for testers to try and follow the guide and provide feedback on whether it comprehensively covered the necessary information. | |||
<references/> | <references/> | ||
=== | === Installation validation test matrix update === | ||
In the Trac ticket<ref>http://fedorahosted.org/fedora-qa/ticket/95</ref>, [[User:Rhe|Rui He]] reported that she had made considerable progress on re-designing the installation validation test matrix<ref>http://fedoraproject.org/wiki/QA:Fedora_14_Install_Results_Template</ref>, including re-organizing the tests into categories and making the matrices for each category collapsible and re-orderable. [[User:Jlaska|James Laska]] thought the changes looked very good. | |||
<references/> | <references/> | ||
=== | === Desktop validation testing expansion === | ||
[[User:Adamwill|Adam Williamson]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092010.html</ref> a plan to expand desktop validation testing during the Fedora 14 cycle to the Xfce, LXDE and KDE desktops, as well as the default GNOME desktop. He explained that, on an experimental basis, the desktop validation test suite would have to be run against each desktop at each release validation test point, and any release criteria-breaking failure in any desktop would need to be fixed before the release could be made. He noted that he had contacted the leaders of the various desktop SIGs, and they were enthusiastic about the idea. [[User:Jdulaney|John Dulaney]] asked<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092011.html</ref> whether ISOs would be available for each desktop. Adam explained<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092012.html</ref> that each desktop can be installed from the DVD image, and that the automated nightly builds could also be used for testing. | |||
<references/> | <references/> | ||
=== | === AutoQA === | ||
At the QA meeting, [[User:Wwoods|Will Woods]] reported that the AutoQA team was working on a helloworld test (a test test), which would exist to check that watchers and hooks - particularly the bodhi watcher and hook - work correctly. This is a prerequisite for the dependency check test, one of the major AutoQA priorities. [[User:Jskladan|Josef Skladanka]] said he had a test instance of the ResultsDB up and running on one of AutoQA's infrastructure machines, and had rewritten the initscripts and rpmlint tests to store their results in the database. He would continue to work on converting other tests. [[User:Kparal|Kamil Paral]] announced that he had patched autoqa to use autotest labels correctly, which allows us to configure the actual running of tests in several ways - ensuring they are run on particular machine configurations. He pointed to a mailing list post<ref>http://fedorahosted.org/pipermail/autoqa-devel/2010-June/000742.html</ref> with a more detailed explanation. | |||
<references/> | <references/> | ||
=== | === Triage metrics === | ||
At the Bugzappers weekly meeting of 2010-07-06<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-07-06/bugzappers.2010-07-06-15.01.log.html</ref>, [[User:Jraber|Jeff Raber]] updated his progress with triage metrics. He had been updating the wiki page on his work<ref>http://fedoraproject.org/wiki/User_talk:Jraber</ref>, and working on a patch to python-bugzilla to allow querying bug history data, which is required for some of the planned metrics. [[User:Adamwill|Adam Williamson]] said he would try to take the metrics Jeff had already implemented and work them into a prototype weekly update email format to send to the list for feedback. | |||
<references/> | <references/> | ||
Revision as of 22:30, 14 July 2010
QualityAssurance
In this section, we cover the activities of the QA team[1]. For more information on the work of the QA team and how you can get involved, see the Joining page[2].
Contributing Writer: Adam Williamson
Proven testers
The proven testers project was well underway, with most of the mentor requests being handled and many group members actively posting feedback. Aaron Farnes proposed[1] a substantial revision to the wiki page[2] which would incorporate information on joining the group (which was a separate page) and on the mentoring process (which was not yet documented), as well as rewriting the testing instructions. John Dulaney liked it[3], as did James Laska[4]. Adam Williamson thought it was well laid out, but too wordy and too far abstracted to work as a clear set of instructions for prospective proven testers[5]. Later, he posted[6] an alternative draft[7] which made fewer changes from the existing page, adding in information on joining the group and on the mentoring process with minimal disruption to the existing content. Aaron liked it[8].
Mike Cloaked generally liked Aaron's draft[9], but raised a question around kernel testing, and whether it was entirely safe for a single proven tester to 'approve' a kernel update when it was unlikely any single tester could come close to comprehensively testing a kernel. Adam thought[10] that generally proven testers should approve updates which do not break critical path functions for them, but that it might make sense to develop a different policy for the kernel. Rick Stevens suggested requiring positive karma for each specific bug fix[11], but Adam explained that this was impossible under the current Bodhi system[12].
Adam Miller proposed two alternatives[13] for sponsoring new proven testers - either having the whole group vote on proven tester candidates at weekly meetings, or allowing mentors to sponsor candidates when the mentor believes the candidate is ready. Adam Williamson was firmly in favour of the more liberal option[14], and with no disagreement, Adam Miller said he would go ahead and update the documentation to reflect this process[15]. James Laska suggested holding the voting process in reserve in case a need to vet candidates more extensively transpired in future[16].
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091860.html
- ↑ http://fedoraproject.org/wiki/User:Dafrito/Proven_tester
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091861.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091879.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091887.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091984.html
- ↑ http://fedoraproject.org/wiki/User:Adamwill/Draft_proventesters
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091985.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091938.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091943.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091954.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091956.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091881.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091883.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091917.html
- ↑ http://lists.fedoraproject.org/pipermail/test/2010-July/091918.html
Musician's guide testing
Christopher Antila wrote to let the group know[1] that the Docs SIG had been working on a guide to creating music with Fedora[2]. He asked for help checking the consistency and language in the guide, and also for testers to try and follow the guide and provide feedback on whether it comprehensively covered the necessary information.
Installation validation test matrix update
In the Trac ticket[1], Rui He reported that she had made considerable progress on re-designing the installation validation test matrix[2], including re-organizing the tests into categories and making the matrices for each category collapsible and re-orderable. James Laska thought the changes looked very good.
Desktop validation testing expansion
Adam Williamson announced[1] a plan to expand desktop validation testing during the Fedora 14 cycle to the Xfce, LXDE and KDE desktops, as well as the default GNOME desktop. He explained that, on an experimental basis, the desktop validation test suite would have to be run against each desktop at each release validation test point, and any release criteria-breaking failure in any desktop would need to be fixed before the release could be made. He noted that he had contacted the leaders of the various desktop SIGs, and they were enthusiastic about the idea. John Dulaney asked[2] whether ISOs would be available for each desktop. Adam explained[3] that each desktop can be installed from the DVD image, and that the automated nightly builds could also be used for testing.
AutoQA
At the QA meeting, Will Woods reported that the AutoQA team was working on a helloworld test (a test test), which would exist to check that watchers and hooks - particularly the bodhi watcher and hook - work correctly. This is a prerequisite for the dependency check test, one of the major AutoQA priorities. Josef Skladanka said he had a test instance of the ResultsDB up and running on one of AutoQA's infrastructure machines, and had rewritten the initscripts and rpmlint tests to store their results in the database. He would continue to work on converting other tests. Kamil Paral announced that he had patched autoqa to use autotest labels correctly, which allows us to configure the actual running of tests in several ways - ensuring they are run on particular machine configurations. He pointed to a mailing list post[1] with a more detailed explanation.
Triage metrics
At the Bugzappers weekly meeting of 2010-07-06[1], Jeff Raber updated his progress with triage metrics. He had been updating the wiki page on his work[2], and working on a patch to python-bugzilla to allow querying bug history data, which is required for some of the planned metrics. Adam Williamson said he would try to take the metrics Jeff had already implemented and work them into a prototype weekly update email format to send to the list for feedback.