From Fedora Project Wiki

< FWN‎ | Beats

(create fwn 236 qa beat)
(create fwn 237 qa beat)
Line 8: Line 8:
<references/>
<references/>


=== QA metrics ===
=== Localization testing ===
 
Discussion on the topic of localization testing continued, with [[User:Rhe|Rui He]] proposing to run a Test Day during the Fedora 14 cycle for localization testing, particularly of keyboard layouts<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092261.html</ref>. [[User:Igor|Igor Pires Soares]] thought this was a great idea and proposed some dates<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092274.html</ref>. Eventually, Rui and Igor agreed on 2010-09-09 as the date, and added the event to the schedule<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092304.html</ref>.
 
<references/>
 
=== Upstream bug reporting ===


[[User:Kparal|Kamil Paral]] asked for feedback<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092196.html</ref> on the topic of gathering statistics on QA activities, as part of a Fedora-wide 'Fedora Hall Of Fame' project<ref>http://fedoraproject.org/wiki/User:Kparal/Idea:Fedora_Hall_of_Fame</ref>. He asked what would be the most important tools to monitor and the most important characteristics to gather. [[User:Johannbg|Jóhann Guðmundsson]] pointed out some of the pitfalls that can come with trying to measure and reward voluntary contributions<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092198.html</ref>. [[User:Jlaska|James Laska]] pointed out that gathering statistics would be valuable in ways beyond trying to reward contributors<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092200.html</ref>. Jóhann clarified that he was only concerned with the 'Hall of Fame' concept and agreed that statistics could be useful in monitoring and improving overall team performance<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092205.html</ref>. Kamil thought that in practice the negative effects Jóhann feared would not be so pronounced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092214.html</ref>. Jóhann followed up with some concrete suggestions of areas and numbers that would be interesting<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092215.html</ref>.  
Discussion also continued on the topic of providing instructions for reporting bugs to upstream bug trackers. [[User:Ankursinha|Ankur Sinha]] proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092266.html</ref> a rough draft<ref>http://fedoraproject.org/wiki/User:Ankursinha/Reporting_Bugs</ref>. Jonathan Kamens felt it encouraged users to report bugs upstream instead of reporting them to Fedora, which he thought was a bad idea<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092269.html</ref>. [[User:Adamwill|Adam Williamson]] suggested focusing solely on the actual steps for reporting bugs upstream, rather than providing any potential reasons why someone might want to do so<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092271.html</ref>. [[User:Johannbg|Jóhann Guðmundsson]] suggested that the instructions should be incorporated into the pages for particular components in the Wiki, rather than being a separate page of their own<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092272.html</ref>.


<references/>
<references/>


=== Localization testing ===
=== Rawhide acceptance testing ===
 
[[User:Jlaska|James Laska]] reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092270.html</ref> on the third (and final) automated Rawhide acceptance test plan<ref>http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan</ref> run for Fedora 14, and linked to the full results<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_14_Pre-Alpha_Rawhide_Acceptance_Test_3</ref>. The run encountered only a single bug, and the test images were declared 'last known good'. However, James cautioned that the images had not included systemd or Python 2.7, which could cause significant changes for the test composes.
 
<references/>
 
=== SELinux use in testing ===


Following on from last week's coverage of [[User:Igor|Igor Pires Soares]]'s localization testing discussion<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092029.html</ref>, [[User:Noriko|Noriko Mizumoto]] suggested<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092193.html</ref> using the newly open-sourced Nitrate tool<ref>http://fedorahosted.org/nitrate/</ref> for tracking the tests. Igor asked<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092201.html</ref> if the tool was yet packaged for Fedora and deployed in Fedora Infrastructure. [[User:Jlaska|James Laska]] reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092203.html</ref> that it was not, but said he thought it would be a good candidate to be included. Later, Igor announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092219.html</ref> that he had made an initial set of updates to the template<ref>http://fedoraproject.org/wiki/QA:Fedora_14_l10n_Results_Template</ref> for Fedora 14.
[[User:Adamwill|Adam Williamson]] reminded the group that SELinux should always be enabled when testing Fedora, if possible<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092273.html</ref>. He explained that since SELinux being enabled is the default configuration of Fedora, it is best for testers to enable SELinux so problems related to it can be caught and fixed as soon as possible.


<references/>
<references/>


=== Boot menu release criterion proposal ===
=== Proven tester special testing procedures ===


After a contentious first Fedora 14 blocker bug review meeting<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092147.html</ref>, [[User:Adamwill|Adam Williamson]] proposed two alternative ways to cover boot menu functionality in the release criteria<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092148.html</ref>. He suggested either having a basic test of essential functionality (the installer eventually boots without manual interaction) at the Alpha stage and a more advanced test (the graphical boot menu appears as intended) at Beta or Final stage, or simply having the more advanced test at Alpha stage. [[User:Jlaska|James Laska]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092152.html</ref> and [[User:Jkeating|Jesse Keating]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092150.html</ref> both came down in favour of simply requiring the menu to work correctly at Alpha stage.
[[User:Adamwill|Adam Williamson]] recalled a previous proposal to have special testing procedures for the kernel for proven testers, and suggested the idea be put into place as a general concept and proposed some specific procedures for PackageKit, in light of a recent update which had broken update notification for some users<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092292.html</ref>. The general response to the proposal was positive. Bob Lightfoot suggested integration with fedora-easy-karma, so that it would display the special testing procedure, or a link to it, when requesting feedback on a package for which a special procedure existed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092313.html</ref>. Adam thought this was a good idea and promised to look into it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092316.html</ref>.


<references/>
<references/>


=== Bootloader menu release criterion ===
=== Automated bug checking tools release criteria ===


[[User:Adamwill|Adam Williamson]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092231.html</ref> that, following the list discussion, he had added the simpler version of the proposed release criterion mentioned in last week's issue to the Fedora 14 Alpha release criteria<ref>http://fedoraproject.org/wiki/Fedora_14_Alpha_Release_Criteria#Alpha_Release_Requirements</ref>. [[User:Rhe|Rui He]] added a validation test to check this<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092248.html</ref>.
[[User:Adamwill|Adam Williamson]] proposed adding release criteria to cover the functionality of default automated bug checking tools (such as abrt and sealert)<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092322.html</ref>. [[User:Jdulaney|John Dulaney]] wondered why making such tools submit reports at an early stage was a high priority. Adam clarified<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092326.html</ref> that the tools should usually always be able to submit reports, and when they cannot, it's an unexpected bug: no regular groundwork is required for the tools to be able to submit reports for new releases.


<references/>
<references/>


=== Lessons learned about update process ===
=== Proven testers working on Fedora 14 ===


[[User:Kevin|Kevin Fenzi]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092236.html</ref> that he had created a page to track lessons learned from problems in the update process<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092236.html</ref>, and asked the group to document any serious issues they had experienced involving updates to stable releases so that FESCo could benefit from the information in planning changes to the update processes and tooling.
[[User:Adamwill|Adam Williamson]] announced that, with the branching of Fedora 14, proven testers are now needed to test and approve critical path package updates for this release<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092324.html</ref>, just as with the stable Fedora 13 release. He explained that the procedure for proven testers to follow was just the same as that for the existing stable release.


<references/>
<references/>

Revision as of 01:17, 5 August 2010

QualityAssurance

In this section, we cover the activities of the QA team[1]. For more information on the work of the QA team and how you can get involved, see the Joining page[2].

Contributing Writer: Adam Williamson

Localization testing

Discussion on the topic of localization testing continued, with Rui He proposing to run a Test Day during the Fedora 14 cycle for localization testing, particularly of keyboard layouts[1]. Igor Pires Soares thought this was a great idea and proposed some dates[2]. Eventually, Rui and Igor agreed on 2010-09-09 as the date, and added the event to the schedule[3].

Upstream bug reporting

Discussion also continued on the topic of providing instructions for reporting bugs to upstream bug trackers. Ankur Sinha proposed[1] a rough draft[2]. Jonathan Kamens felt it encouraged users to report bugs upstream instead of reporting them to Fedora, which he thought was a bad idea[3]. Adam Williamson suggested focusing solely on the actual steps for reporting bugs upstream, rather than providing any potential reasons why someone might want to do so[4]. Jóhann Guðmundsson suggested that the instructions should be incorporated into the pages for particular components in the Wiki, rather than being a separate page of their own[5].

Rawhide acceptance testing

James Laska reported[1] on the third (and final) automated Rawhide acceptance test plan[2] run for Fedora 14, and linked to the full results[3]. The run encountered only a single bug, and the test images were declared 'last known good'. However, James cautioned that the images had not included systemd or Python 2.7, which could cause significant changes for the test composes.

SELinux use in testing

Adam Williamson reminded the group that SELinux should always be enabled when testing Fedora, if possible[1]. He explained that since SELinux being enabled is the default configuration of Fedora, it is best for testers to enable SELinux so problems related to it can be caught and fixed as soon as possible.

Proven tester special testing procedures

Adam Williamson recalled a previous proposal to have special testing procedures for the kernel for proven testers, and suggested the idea be put into place as a general concept and proposed some specific procedures for PackageKit, in light of a recent update which had broken update notification for some users[1]. The general response to the proposal was positive. Bob Lightfoot suggested integration with fedora-easy-karma, so that it would display the special testing procedure, or a link to it, when requesting feedback on a package for which a special procedure existed[2]. Adam thought this was a good idea and promised to look into it[3].

Automated bug checking tools release criteria

Adam Williamson proposed adding release criteria to cover the functionality of default automated bug checking tools (such as abrt and sealert)[1]. John Dulaney wondered why making such tools submit reports at an early stage was a high priority. Adam clarified[2] that the tools should usually always be able to submit reports, and when they cannot, it's an unexpected bug: no regular groundwork is required for the tools to be able to submit reports for new releases.

Proven testers working on Fedora 14

Adam Williamson announced that, with the branching of Fedora 14, proven testers are now needed to test and approve critical path package updates for this release[1], just as with the stable Fedora 13 release. He explained that the procedure for proven testers to follow was just the same as that for the existing stable release.