FWN/Beats/QualityAssurance: Difference between revisions

Revision as of 03:15, 15 August 2009

QualityAssurance

In this section, we cover the activities of the QA team^[1].

↑ http://fedoraproject.org/wiki/QA

Test Days

Last week's main track Test Day^[1] was on NetworkManager^[2]. There was a solid turnout of testers and developers, and several bugs were filed and fixed. A report on this Test Day is available^[3].

Last weeks' Fit and Finish Test Day^[4] was on peripherals. Several people turned out to help test, and a variety of different bugs with different types of peripheral were reported.

Next week's main track Test Day^[5] will be on ABRT changes for Fedora 12^[6]. ABRT is the Automatic Bug Reporting Tool which helps users file bug reports automatically when applications fail, and it has been extensively improved for Fedora 12. It's an easy component to test and it will help improve the quality of future Fedora releases, so please come along and help out! The Test Day will be held on Thursday 2009-08-20 in IRC #fedora-qa.

The Fit and Finish^[7] Test Day track will be holding its own Test Day^[8], on printing. This is a vital area for many users and has lots of potential quirks with different types of printer connected in different ways, so please come out and help make sure the printing user experience is as smooth as possible! Live images will be available before the Test Day. The Test Day will be held on Tuesday 2009-08-18 in IRC #fedora-fit-and-finish (note this is not the same channel where main track Test Days take place).

If you would like to propose a main track Test Day for the Fedora 12 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac^[9].

Weekly meetings

The QA group weekly meeting^[1] was held on 2009-08-10. The full log is available^[2]. James Laska asked for feedback on the quality of downloads of the Alpha test compose from the alt.fedoraproject.org server. Adam Williamson reported that his download had been fast and trouble-free. Kamil Paral's had been slower, but that was tracked down to bandwidth limitations on his end.

James Laska asked why Rawhide still contained anaconda 12.7, when later versions had been released and built. Jesse Keating stated that later versions of anaconda had been entirely broken and thus had not passed his critical path package checks. Adam Williamson asked why major regressions in anaconda seemed to be being introduced during an Alpha freeze. Jesse Keating explained that anaconda development was treated as an independent 'upstream project', like rpm, and so did not respect Fedora freezes. Adam suggested that, in that case, Fedora packaging of anaconda should not accept new upstream versions as a matter of course, especially during freezes, but cherry-pick appropriate fixes, due to the sensitivity of anaconda to changes and its position of fundamental importance in any Fedora release.

James Laska called for those who had filed or were monitoring critical bugs for the Alpha release to continue to work on verifying fixes for them and closing them where appropriate.

James Laska asked for a general overview of Rawhide's readiness for the Alpha release. The consensus was that anaconda was still not yet ready, but most other components were in decent shape. Adam Williamson noted that packages fixing the known major breakage in xorg-x11-server-1.6.99-25 had been tagged into Rawhide over the weekend. James also worried that many features on the Fedora 12 feature list did not seem to be complete in terms of development or have complete test plans yet, but no action was thought to be possible on this.

Will Woods reported on the progress of the AutoQA project. He had completed the automated installation tests, and refactored the pre-existing autoqa tests into the new autotest system. He also had some tests starting to send their results to a mailing list, and hoped to have this process available to the public soon.

The Bugzappers group weekly meeting^[3] was held on 2009-08-11. The full log is available^[4]. Adam Williamson admitted he had not found time to ask the desktop development team for their position on the new triage process, or check which Bugzilla changes generate an email by default.

The group wanted to take a final decision on the question of changing the process by which bugs are marked as triaged. After a long discussion, it was agreed to go ahead with a plan to switch to using the Triaged keyword rather than the ASSIGNED state, starting with bugs for Fedora 13. Adam Williamson agreed to send a wrap-up email to the mailing list.

Edward Kirk brought up the recent fedora-devel-list mail^[5] which had mentioned the need for triaging of XMMS bugs. However, several group members had looked over the list of bugs on XMMS that were still open or had been closed due to age, and found nothing that could be pursued.

Edward Kirk also worried that meetings were being planned only by himself and Richard June and were not being planned according to a defined policy. He intended to write a SOP for planning meetings, and encourage the use of the agenda item submission process to make sure no important issues were not making it to the meeting agenda.

The next QA weekly meeting will be held on 2009-08-17 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-08-18 at 1500 UTC in #fedora-meeting.

Fedora 12 Alpha delay

At the final go/no-go meeting on Monday 2009-08-10, it was decided with the unanimous agreement of release engineering and QA groups to slip Fedora 12 Alpha's release by one week due to several blocker bugs still outstanding, including several bugs which could cause installations to fail completely in very common circumstances. Jesse Keating announced the slip^[1] to the development mailing list.

↑ http://www.redhat.com/archives/fedora-devel-list/2009-August/msg00634.html

Fedora 12 Alpha blocker bug reviews

James Laska reminded the group^[1] that several bugs blocking the Alpha release (as of Sunday 2009-08-09) were in MODIFIED state and required further testing. Later, he sent a follow-up^[2] with updated status on several of the bugs listed.

[1] ttp://fedoraproject.org/wiki/QA

[2] ttp://fedoraproject.org/wiki/Test_Day:2009-08-13_NetworkManager

[3] ttp://fedoraproject.org/wiki/Tools/NetworkManager

[4] ttp://www.redhat.com/archives/fedora-test-list/2009-August/msg00377.html

[5] ttp://fedoraproject.org/wiki/Test_Day:2009-08-11_Fit_and_Finish:Peripherals

[6] ttp://fedoraproject.org/wiki/Test_Day:2009-08-20

[7] ttp://fedoraproject.org/wiki/Features/ABRTF12

[8] ttp://fedoraproject.org/wiki/Fit_and_Finish

[9] ttps://fedoraproject.org/wiki/Test_Day:2009-08-18_Fit_and_Finish:Printing

[10] ttp://fedorahosted.org/fedora-qa/

[11] ttp://fedoraproject.org/wiki/QA/Meetings

[12] ttp://fedoraproject.org/wiki/QA/Meetings/20090810

[13] ttp://fedoraproject.org/wiki/BugZappers/Meetings

[14] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-08-11/fedora-meeting.2009-08-11-15.03.log.html

[15] ttp://www.redhat.com/archives/fedora-devel-list/2009-August/msg00490.html

[16] ttp://www.redhat.com/archives/fedora-devel-list/2009-August/msg00634.html

[17] ttp://www.redhat.com/archives/fedora-test-list/2009-August/msg00241.html

[18] ttp://www.redhat.com/archives/fedora-test-list/2009-August/msg00312.html

[1]

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[1]

[2]

[3]

[4]

[5]

[1]

[1]

[2]

@@ Line 10: / Line 10: @@
 === Test Days ===
-There was no Test Day last week. Next week's main track Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-13_NetworkManager</ref> will be on NetworkManager<ref>http://fedoraproject.org/wiki/Tools/NetworkManager</ref>. NetworkManager will see some significant changes and improvements for Fedora 12, so we want to test all its capabilities. Testing will include regular and wireless ethernet connections, IPv6, cellular data modems, and cellphone tethering both via USB and Bluetooth. A testing live image will be available on the Test Day page prior to the event. Just about everyone needs to network with something, so please come along and make sure your networking needs will be met in Fedora 12! The Test Day will be held on 2009-08-13 in IRC #fedora-qa.
+Last week's main track Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-13_NetworkManager</ref> was on NetworkManager<ref>http://fedoraproject.org/wiki/Tools/NetworkManager</ref>. There was a solid turnout of testers and developers, and several bugs were filed and fixed. A report on this Test Day is available<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00377.html</ref>.
-The Fit and Finish<ref>http://fedoraproject.org/wiki/Fit_and_Finish</ref> Test Day track will be holding its third event<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-11_Fit_and_Finish:Peripherals</ref>, on peripherals. This is a broad-based day where we want to look at experiences involving any kind of peripheral - an external device you can attach to your computer (via a cable or even Bluetooth). Some test cases for the more common types of peripheral are already up on the page, and live images will be available before the Test Day. Please come along to contribute your experiences and suggestion improvements on how Fedora deals with peripherals! The Test Day will be held on 2009-08-11 in IRC #fedora-fit-and-finish (note this is not the same channel where main track Test Days take place).
+Last weeks' Fit and Finish Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-11_Fit_and_Finish:Peripherals</ref> was on peripherals. Several people turned out to help test, and a variety of different bugs with different types of peripheral were reported.
+Next week's main track Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-20</ref> will be on ABRT changes for Fedora 12<ref>http://fedoraproject.org/wiki/Features/ABRTF12</ref>. ABRT is the Automatic Bug Reporting Tool which helps users file bug reports automatically when applications fail, and it has been extensively improved for Fedora 12. It's an easy component to test and it will help improve the quality of future Fedora releases, so please come along and help out! The Test Day will be held on Thursday 2009-08-20 in IRC #fedora-qa.
+The Fit and Finish<ref>http://fedoraproject.org/wiki/Fit_and_Finish</ref> Test Day track will be holding its own Test Day<ref>https://fedoraproject.org/wiki/Test_Day:2009-08-18_Fit_and_Finish:Printing</ref>, on printing. This is a vital area for many users and has lots of potential quirks with different types of printer connected in different ways, so please come out and help make sure the printing user experience is as smooth as possible! Live images will be available before the Test Day. The Test Day will be held on Tuesday 2009-08-18 in IRC #fedora-fit-and-finish (note this is not the same channel where main track Test Days take place).
 If you would like to propose a main track Test Day for the Fedora 12 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
@@ Line 20: / Line 24: @@
 === Weekly meetings ===
-The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-08-03. The full log is available<ref>http://fedoraproject.org/wiki/QA/Meetings/20090803</ref>. [[User:Jlaska|James Laska]] reported that he had filed a bug<ref>https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=514062</ref> on the pungi problems discussed at the previous meeting which were preventing Rawhide install image composes from working. [[User:jkeating|Jesse Keating]] had already helped to fix the bug.
+The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-08-10. The full log is available<ref>http://fedoraproject.org/wiki/QA/Meetings/20090810</ref>. [[User:Jlaska|James Laska]] asked for feedback on the quality of downloads of the Alpha test compose from the alt.fedoraproject.org server. [[User:Adamwill|Adam Williamson]] reported that his download had been fast and trouble-free. [[User:Kparal|Kamil Paral's]] had been slower, but that was tracked down to bandwidth limitations on his end.
-[[User:poelstra|John Poelstra]] had, as promised, sent an announcement of the previous Friday's Alpha blocker bug review meeting<ref>http://www.redhat.com/archives/fedora-test-list/2009-July/msg00655.html</ref>. [[User:Adamwill|Adam Williamson]] had not yet posted a recap of the blocker meeting (though this was subsequently posted after the QA meeting).
-[[User:Jlaska|James Laska]] had contacted [[User:Wtogami|Warren Togami]] to ask for an entry in the Rawhide Watch blog<ref>http://rawhidewatch.wordpress.com/</ref> about updating Rawhide installations past the change to XZ payloads, and Warren had added an entry<ref>http://rawhidewatch.wordpress.com/2009/07/31/unable-to-update-to-rawhide-rpmlibpayloadisxz</ref>.
+[[User:Jlaska|James Laska]] asked why Rawhide still contained anaconda 12.7, when later versions had been released and built. [[User:jkeating|Jesse Keating]] stated that later versions of anaconda had been entirely broken and thus had not passed his critical path package checks. [[User:Adamwill|Adam Williamson]] asked why major regressions in anaconda seemed to be being introduced during an Alpha freeze. [[User:jkeating|Jesse Keating]] explained that anaconda development was treated as an independent 'upstream project', like rpm, and so did not respect Fedora freezes. Adam suggested that, in that case, Fedora packaging of anaconda should not accept new upstream versions as a matter of course, especially during freezes, but cherry-pick appropriate fixes, due to the sensitivity of anaconda to changes and its position of fundamental importance in any Fedora release.
-[[User:Kparal|Kamil Paral]] had updated the instructions for composing Test Day live CDs<ref>http://fedoraproject.org/wiki/QA/Test_Days/Live_Image</ref> to correctly produce Rawhide, rather than Fedora 11, images.
+[[User:Jlaska|James Laska]] called for those who had filed or were monitoring critical bugs for the Alpha release to continue to work on verifying fixes for them and closing them where appropriate.
-[[User:Johannbg|Jóhann Guðmundsson]] provided an update on the Fedora 12 Dracut feature<ref>http://fedoraproject.org/wiki/Features/Dracut</ref>. [[User:Harald|Harald Hoyer]] is working together with the kernel and anaconda teams to switch to using Dracut by default to generate initramfs images in Rawhide. They expect it to already work fully and transparently for typical situations, and expect testing (including the scheduled Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-08-27</ref>) to focus on complex and unusual situations, like the root partition being on a network drive, or complex RAID/LVM setups.
+[[User:Jlaska|James Laska]] asked for a general overview of Rawhide's readiness for the Alpha release. The consensus was that anaconda was still not yet ready, but most other components were in decent shape. [[User:Adamwill|Adam Williamson]] noted that packages fixing the known major breakage in xorg-x11-server-1.6.99-25 had been tagged into Rawhide over the weekend. James also worried that many features on the Fedora 12 feature list did not seem to be complete in terms of development or have complete test plans yet, but no action was thought to be possible on this.
-The group discussed the state of Rawhide in regards to the Alpha test compose that had been delayed from the previous week. It had still not been possible to create a test compose due to various bugs completely breaking the install process. It seemed that all the complete blocker bugs were cleared, so [[User:jkeating|Jesse Keating]] said he would attempt to build a test compose later in the day. [[User:Jlaska|James Laska]] noted the compose would therefore be four days behind schedule, but thought this did not inevitably mean the release of the Alpha must be delayed, if testing could be completed quickly.
+[[User:Wwoods|Will Woods]] reported on the progress of the AutoQA project. He had completed the automated installation tests, and refactored the pre-existing autoqa tests into the new autotest system. He also had some tests starting to send their results to a mailing list, and hoped to have this process available to the public soon.
-[[User:Jlaska|James Laska]] proposed that a fourth Alpha blocker bug review meeting be scheduled for 2009-08-07, and the Alpha go/no-go meeting scheduled for 2009-08-10 should be a quick decision rather than a full review meeting. [[User:poelstra|John Poelstra]] agreed. [[User:jkeating|Jesse Keating]] felt it was inefficient to hold similar meetings on two successive work days, but did not mind if that path was chosen and would show up for both meetings. The group agreed to follow this plan.
+The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-08-11. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-08-11/fedora-meeting.2009-08-11-15.03.log.html</ref>. [[User:Adamwill|Adam Williamson]] admitted he had not found time to ask the desktop development team for their position on the new triage process, or check which Bugzilla changes generate an email by default.
-[[User:Wwoods|Will Woods]] reported on the progress of the AutoQA project. He has now completed automating all the installation tests in the Rawhide Acceptance Test Plan<ref>http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan</ref>, which means there is now a script capable of performing a fully automated installation up to the stage of disk partitioning, declaring a successful test if disks are found and partitioning completed, or a failed test if not. It also stores all relevant logs as the installation takes place. [[User:Jlaska|James Laska]] provided a link to an example report matrix graphic<ref>http://jlaska.fedorapeople.org/rawhide-report.png</ref> which could be one format used to report results. Will said the next steps would be to make the test run automatically each time a new Rawhide compose was released, and to have organized reporting of the test results. Then tests for the post-installation functionality test cases would be needed, and a front-end for manual results submission for functionality test cases which cannot be automated. Will also noted that eventually the system might be adapted to run on bare metal rather than in a virtual environment, which would allow for testing on the PPC architecture. Will has also started work on Wiki documentation for writing autotest tests<ref>http://fedoraproject.org/wiki/Autotest</ref>.
+The group wanted to take a final decision on the question of changing the process by which bugs are marked as triaged. After a long discussion, it was agreed to go ahead with a plan to switch to using the Triaged keyword rather than the ASSIGNED state, starting with bugs for Fedora 13. [[User:Adamwill|Adam Williamson]] agreed to send a wrap-up email to the mailing list.
-The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-08-04. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-08-04/fedora-meeting.2009-08-04-15.05.log.html</ref>. The group closed out discussion on the critical path component list-based expansion of the priority triage packages list by noting that it had been succesfully merged into the main priority triage package list<ref>http://fedoraproject.org/wiki/BugZappers/Components_and_Triagers</ref>, and thanking Niels Haase for his work.
+[[User:Tk009|Edward Kirk]] brought up the recent fedora-devel-list mail<ref>http://www.redhat.com/archives/fedora-devel-list/2009-August/msg00490.html</ref> which had mentioned the need for triaging of XMMS bugs. However, several group members had looked over the list of bugs on XMMS that were still open or had been closed due to age, and found nothing that could be pursued.
-[[User:Adamwill|Adam Williamson]] gave an update on the status of the kernel triage project. [[User:Rjune|Richard June]] had contacted the kernel maintainer responsible for wireless networking, [[User:linville|John Linville]], to notify him of the project and ask for any advice or requests he had, but had not yet received a reply. Adam said he would contact John himself if a reply was not received soon.
-Brennan Ashton gave an update on the triage metrics project. It had turned out that a bug in FAS was causing some of the problems with his attempted migration to the version of the triage metrics script with FAS integration. He was working with the Infrastructure team to resolve this bug.
-The group discussed again the 'Bugzilla Semantics' proposal. [[User:Adamwill|Adam Williamson]] had sent a mail<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00039.html</ref> to fedora-test-list formally proposing moving to a keyword rather than a status to mark bugs as triaged going forward, but only one day prior to the meeting. He felt it would be best to leave more time for list feedback before making a final decision. Brennan Ashton questioned how many development groups were actively desiring the change. [[User:cebbert|Chuck Ebbert]] avowed that the kernel team would be happy if the change was made, and [[User:Skvidal|Seth Vidal]] already asks for the proposed new policy to be used for yum bugs as a special exception to current practice. [[User:Mcepl|Matej Cepl]] said the KDE and LVM teams would prefer the proposed new system, and [[User:jkeating|Jesse Keating]] said the Desktop team would also.
-The next QA weekly meeting will be held on 2009-08-10 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-08-11 at 1500 UTC in #fedora-meeting.
-<references/>
-=== Alpha test compose availability ===
-[[User:Liam|Liam Li]] announced the availability of the first test compose for Fedora 12 Alpha<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00063.html</ref>. He asked group members to help validate the image, especially as far as installation testing went, following the test matrix<ref>http://fedoraproject.org/wiki/QA:Fedora_12_Alpha_Install_Test_Results</ref>. Testing was carried out by several group members, and identified some significant bugs which were added to the Alpha blocker list, as per a status roundup<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00139.html</ref> by [[User:Jlaska|James Laska]]. In response to a question<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00112.html</ref> from Todd, [[User:Jlaska|James Laska]] clarified that testing did not need to be performed with the test compose images; testing network installation direct from the Rawhide repository was also useful (and necessary, for some test cases).
-<references/>
-=== Xfce spin testing ===
+[[User:Tk009|Edward Kirk]] also worried that meetings were being planned only by himself and [[User:Rjune|Richard June]] and were not being planned according to a defined policy. He intended to write a SOP for planning meetings, and encourage the use of the agenda item submission process to make sure no important issues were not making it to the meeting agenda.
-[[User:maxamillion|Adam Miller]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00089.html</ref> the next Xfce test live image, for Xfce enthusiasts to test and report Xfce-specific issues.
+The next QA weekly meeting will be held on 2009-08-17 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-08-18 at 1500 UTC in #fedora-meeting.
 <references/>
-=== Test Day live image creation ===
+=== Fedora 12 Alpha delay ===
-[[User:Kparal|Kamil Paral]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00116.html</ref> that he had improved the Test Day live image creation guide<ref>https://fedoraproject.org/wiki/QA/Test_Days/Live_Image</ref> to explain workarounds and fixes for some problems related to SELinux that had been discussed on the list.
+At the final go/no-go meeting on Monday 2009-08-10, it was decided with the unanimous agreement of release engineering and QA groups to slip Fedora 12 Alpha's release by one week due to several blocker bugs still outstanding, including several bugs which could cause installations to fail completely in very common circumstances. [[User:jkeating|Jesse Keating]] announced the slip<ref>http://www.redhat.com/archives/fedora-devel-list/2009-August/msg00634.html</ref> to the development mailing list.
 <references/>
-=== Fedora 12 Alpha blocker bug review meetings ===
+=== Fedora 12 Alpha blocker bug reviews ===
-[[User:Adamwill|Adam Williamson]] posted a recap<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00109.html</ref> of the blocker bug review meeting held on 2009-07-31, including details of the bugs discussed and the actions (or lack of action) decided upon for each. [[User:Jlaska|James Laska]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00177.html</ref> the following blocker bug review meeting, to be held on 2009-08-07.
+[[User:Jlaska|James Laska]] reminded the group<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00241.html</ref> that several bugs blocking the Alpha release (as of Sunday 2009-08-09) were in MODIFIED state and required further testing. Later, he sent a follow-up<ref>http://www.redhat.com/archives/fedora-test-list/2009-August/msg00312.html</ref> with updated status on several of the bugs listed.
 <references/>

Search