FWN/Beats/QualityAssurance: Difference between revisions

Revision as of 19:45, 23 November 2009

QualityAssurance

In this section, we cover the activities of the QA team^[1].

↑ http://fedoraproject.org/wiki/QA

Test Days

There was no Test Day last week, and no Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac^[1].

↑ http://fedorahosted.org/fedora-qa/

Weekly meetings

The QA group weekly meeting^[1] was held on 2009-11-16. The full log is available^[2]. Adam Williamson reported that Milos Jakubicek had still not yet followed up on his idea regarding an event to work on FTBFS problems, and the group agreed to table the proposal until he came back with further ideas.

James Laska reported that he had asked Rui He to improve the existing preupgrade test cases to make sure they more accurately reflected real-world use and would hence catch the disk space issues experienced with Fedora 12.

Adam Williamson started a discussion of preparation for the release of Fedora 12, which was to happen the day after the meeting. He highlighted the common bugs page^[3], and James Laska provided a link to a list^[4] of issues which were awaiting addition to that page. James and Adam agreed to work on updating the page. Adam also noted that the Fedora 12 blocker bug should be cleaned up. After some discussion, James and Adam noticed that blocker bugs fell under the remit of the BugZappers group, and agreed to let the following day's BugZappers meeting handle the issue.

James Laska gave a heads-up on his planning for a post-Fedora 12 release retrospective. He was planning to send an email to the mailing list asking for people to identify potential areas for improvement from the Fedora 12 QA cycle, and then sum up the resulting feedback in a wiki page.

Will Woods and Kamil Paral reported on the progress of the AutoQA project. Will had been trying to complete the post-koji-build hook which would allow tests to be triggered by the completion of a build in Koji. He had also talked with the release engineering group about how to create AutoQA tests to help prevent broken dependencies in update repositories, and this had identified the need for a post-bodhi-update hook which would allow tests to be run when Bodhi is used to request a package be added to updates-testing or updates repositories. Will asked Luke Macken what resources Bodhi currently provides that would allow AutoQA to notice when an update is requested, and Luke said at present only RSS feeds are available. Will said he would write a hook that monitored the RSS feeds. Will and Kamil also outlined the current plan for rpmguard integration. Kamil had posted a proposal^[5] on making test development easier, and James Laska had derived an AutoQA use cases page^[6] from it. James also noted that the updated autotest packages had been tested and seemed to be working well.

The Bugzappers group weekly meeting^[7] was held on 2009-11-17. The full log is available^[8]. Edward Kirk announced that the long-planned semantics change would now be going into effect, as Rawhide had separated from Fedora 12 and was driving towards Fedora 13 development. As previously agreed, all bugs filed for Rawhide should be marked as having been triaged by the addition of the Triaged keyword, rather than setting the ASSIGNED status. Steven Parrish volunteered to send an email to the development list announcing the change. Steven also pointed out that the GreaseMonkey script used by most triagers would need updating for the change. Chris Campbell volunteered to follow up with Matej Cepl about updating the script. Adam Williamson volunteered to update the text in the bug workflow page^[9] to reflect the change, and Edward volunteered to change the image.

Edward Kirk introduced the topic of housekeeping updates. He noted that the first release day tasks^[10] - including creating the Fedora 14 blocker bugs, and closing off the Fedora 12 blockers - needed to be done, and said he would take care of that.

The group helped Joerg Stephan with choosing some components to begin his triage work.

Matej Cepl asked for some input on the design of the Greasemonkey script with regards to the new triaging procedure. The group agreed that a single 'smart' button which made the appropriate changes depending on the distribution version for which the bug in question was reported would be better than separate buttons for pre-Fedora 13 and Fedora 13-and-later bugs would be a better design.

The next QA weekly meeting will be held on 2009-11-23 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-24 at 1500 UTC in #fedora-meeting.

Improving the release criteria

John Poelstra submitted a proposal^[1] for improving the release criteria^[2] for future releases. The new proposed criteria ^[3] splits the old single page into an introductory / outline page and separate pages for each public release in the upcoming cycle. Adam Williamson^[4] and James Laska^[5] both replied to welcome to idea and post some suggestions for refinement. John plans to further refine the proposal and then have a session to discuss it at the upcoming FUDCon Toronto.

[1] ttp://fedoraproject.org/wiki/QA

[2] ttp://fedorahosted.org/fedora-qa/

[3] ttp://fedoraproject.org/wiki/QA/Meetings

[4] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-11-16/fedora-meeting.2009-11-16-16.01.log.html

[5] ttp://fedoraproject.org/wiki/Common_F12_bugs

[6] ttp://fedoraproject.org/wiki/QA/Meetings/20091116#Common_F12_Bugs

[7] ttp://fedorahosted.org/pipermail/autoqa-devel/2009-November/000018.html

[8] ttp://fedoraproject.org/wiki/AutoQA_Use_Cases

[9] ttp://fedoraproject.org/wiki/BugZappers/Meetings

[10] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-11-17/fedora-meeting.2009-11-17-15.00.log.html

[11] ttp://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow

[12] ttp://fedoraproject.org/wiki/BugZappers/HouseKeeping/FirstDayDevel

[13] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00926.html

[14] ttp://fedoraproject.org/wiki/QA/ReleaseCriteria

[15] ttp://fedoraproject.org/wiki/Fedora_Release_Criteria

[16] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00933.html

[17] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00992.html

[1]

[1]

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[1]

[2]

[3]

[4]

[5]

@@ Line 10: / Line 10: @@
 === Test Days ===
-There have been no Test Days for the last two weeks, due to the pressures of the Fedora 12 release.
+There was no Test Day last week, and no Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
-No Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
 <references/>
@@ Line 18: / Line 16: @@
 === Weekly meetings ===
-As the QA beat was unfortunately not present for Fedora Weekly News #201, we will cover two weeks' worth of events below.
+The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-11-16. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-16/fedora-meeting.2009-11-16-16.01.log.html</ref>. [[User:Adamwill|Adam Williamson]] reported that Milos Jakubicek had still not yet followed up on his idea regarding an event to work on FTBFS problems, and the group agreed to table the proposal until he came back with further ideas.
-QA group weekly meetings<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> were held on 2009-11-02 and 2009-11-09. The full logs are available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-02/fedora-meeting.2009-11-02-16.01.log.html</ref>, <ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-09/fedora-meeting.2009-11-09-16.00.log.html</ref>.
+[[User:Jlaska|James Laska]] reported that he had asked [[User:Rhe|Rui He]] to improve the existing preupgrade test cases to make sure they more accurately reflected real-world use and would hence catch the disk space issues experienced with Fedora 12.
-During the meeting of 2009-11-02, [[User:Adamwill|Adam Williamson]] reported that Milos Jakubicek had not yet followed up on his idea regarding an event to work on FTBFS problems. [[User:Johannbg|Jóhann Guðmundsson]] was also not present to report on his work on creating a test case for keyboard layout issues<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452</ref>. [[User:Wwoods|Will Woods]] was making good progress on the action items for AutoQA from the previous meeting.
+[[User:Adamwill|Adam Williamson]] started a discussion of preparation for the release of Fedora 12, which was to happen the day after the meeting. He highlighted the common bugs page<ref>http://fedoraproject.org/wiki/Common_F12_bugs</ref>, and [[User:Jlaska|James Laska]] provided a link to a list<ref>http://fedoraproject.org/wiki/QA/Meetings/20091116#Common_F12_Bugs</ref> of issues which were awaiting addition to that page. James and Adam agreed to work on updating the page. Adam also noted that the Fedora 12 blocker bug should be cleaned up. After some discussion, James and Adam noticed that blocker bugs fell under the remit of the BugZappers group, and agreed to let the following day's BugZappers meeting handle the issue.
-The group reviewed the status of the Fedora 12 code base with regards to the then-impeding release candidate phase. It was generally agreed that the status was promising and it should be possible to make the release candidate phase on time, based on a review of the blocker bug list.
+[[User:Jlaska|James Laska]] gave a heads-up on his planning for a post-Fedora 12 release retrospective. He was planning to send an email to the mailing list asking for people to identify potential areas for improvement from the Fedora 12 QA cycle, and then sum up the resulting feedback in a wiki page.
-[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Will had been working on revising the AutoQA code to provide a Python library interface<ref>http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=lib/python;hb=wwoods-autotest</ref>. He had been moving all shared or potentially shareable code from all current AutoQA tests into the library. He hoped to have it merged into the master branch by the end of the week. He also noted that a newer version of autotest was currently being packaged and implemented into the AutoQA system, which may cause strange results if any bugs emerged.
+[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Will had been trying to complete the post-koji-build hook which would allow tests to be triggered by the completion of a build in Koji. He had also talked with the release engineering group about how to create AutoQA tests to help prevent broken dependencies in update repositories, and this had identified the need for a post-bodhi-update hook which would allow tests to be run when Bodhi is used to request a package be added to updates-testing or updates repositories. Will asked [[User:Lmacken|Luke Macken]] what resources Bodhi currently provides that would allow AutoQA to notice when an update is requested, and Luke said at present only RSS feeds are available. Will said he would write a hook that monitored the RSS feeds. Will and Kamil also outlined the current plan for rpmguard integration. Kamil had posted a proposal<ref>http://fedorahosted.org/pipermail/autoqa-devel/2009-November/000018.html</ref> on making test development easier, and [[User:Jlaska|James Laska]] had derived an AutoQA use cases page<ref>http://fedoraproject.org/wiki/AutoQA_Use_Cases</ref> from it. James also noted that the updated autotest packages had been tested and seemed to be working well.
-[[User:Adamwill|Adam Williamson]] reviewed upcoming events. The release candidate date was Wednesday 2009-11-04 and the go/no-go date Monday 2009-11-09. [[User:jkeating|Jesse Keating]] clarified that for an RC build to be done, the blocker bug list must be clear, but new issues that emerged during RC compose and testing could be resolved up until the date of the go/no-go meeting. [[User:Jlaska|James Laska]] promised to co-ordinate with the anaconda team to ensure there were no remaining blocker issues in installation.
+The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-11-17. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-17/fedora-meeting.2009-11-17-15.00.log.html</ref>. [[User:Tk009|Edward Kirk]] announced that the long-planned semantics change would now be going into effect, as Rawhide had separated from Fedora 12 and was driving towards Fedora 13 development. As previously agreed, all bugs filed for Rawhide should be marked as having been triaged by the addition of the Triaged keyword, rather than setting the ASSIGNED status. [[User:StevenParrish|Steven Parrish]] volunteered to send an email to the development list announcing the change. Steven also pointed out that the GreaseMonkey script used by most triagers would need updating for the change. [[User:campbecg|Chris Campbell]] volunteered to follow up with [[User:Mcepl|Matej Cepl]] about updating the script. [[User:Adamwill|Adam Williamson]] volunteered to update the text in the bug workflow page<ref>http://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow</ref> to reflect the change, and Edward volunteered to change the image.
-The meeting of 2009-11-09 was held during the final run-up to the Fedora 12 go/no-go meeting, so there was some last-minute blocker bug discussion. [[User:Johannbg|Jóhann Guðmundsson]] had not yet been able to work on creating a test case for keyboard layout issues<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452</ref>. [[User:Jlaska|James Laska]] had followed up with the anaconda team and verified no blocker bugs remained in the installation process. [[User:Adamwill|Adam Williamson]] noted that one of the anaconda bugs that definitely wasn't left had been fixed the previous day.
+[[User:Tk009|Edward Kirk]] introduced the topic of housekeeping updates. He noted that the first release day tasks<ref>http://fedoraproject.org/wiki/BugZappers/HouseKeeping/FirstDayDevel</ref> - including creating the Fedora 14 blocker bugs, and closing off the Fedora 12 blockers - needed to be done, and said he would take care of that.
-[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Kamil had added a check to rpmguard for the case where an old version of rpmdiff is installed. Will had the new python library ready but wanted more testing before merging it into master. The new Koji watcher (for running AutoQA tests on new builds as they hit Koji) was now functional.
+The group helped Joerg Stephan with choosing some components to begin his triage work.
-[[User:Jlaska|James Laska]] pointed out that [[MatthiasClasen|Matthias Clasen]] had asked the group to test Fedora 12 0-day updates by enabling the Fedora 12 updates and possibly updates-testing repositories and updating their systems. James thought it would be a good idea to create a test case for testing the update repositories for a release before they were generally enabled.
+[[User:Mcepl|Matej Cepl]] asked for some input on the design of the Greasemonkey script with regards to the new triaging procedure. The group agreed that a single 'smart' button which made the appropriate changes depending on the distribution version for which the bug in question was reported would be better than separate buttons for pre-Fedora 13 and Fedora 13-and-later bugs would be a better design.
-A Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-11-03. No meeting was held on 2009-11-10. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-03/fedora-meeting.2009-11-03-15.03.log.html</ref>. [[User:Rjune|Richard June]] reported that he was continuing to work on kernel triage. He had not been in touch with Jeff Hann regarding his volunteering to help out yet, but would attempt to co-ordinate via the mailing list.
+The next QA weekly meeting will be held on 2009-11-23 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-24 at 1500 UTC in #fedora-meeting.
-[[User:Adamwill|Adam Williamson]] asked if anyone had concerns about unaddressed issues for the Fedora 12 release, and no-one did. Adam asked [[User:Mcepl|Matej Cepl]] how he was coping with X.org triage while [[User:Fcami|François Cami]] was mostly unable to help, and he said it was difficult to stay on top of the large number of bugs. Adam promised to continue to try and manage the nouveau driver bugs, and [[user:thomasj|Thomas Janssen]] volunteered to help out with others. Matej said he would work with Thomas to bring him up to speed on X triaging.
-[[User:Tk009|Edward Kirk]] reported that he had worked on outstanding Fedora 10 bugs, and managed to update some and close others. He also reported that the maintainer warning email for the upcoming Fedora 12 housekeeping Bugzilla changes had been sent out. [[User:poelstra|John Poelstra]] was ready to do the Rawhide bug rebase (moving open Rawhide bugs to Fedora 12) and Fedora 10 bug end-of-life warning operations.
-The next QA weekly meeting will be held on 2009-11-16 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-17 at 1500 UTC in #fedora-meeting.
 <references/>
-=== Fedora 12 release ===
+=== Improving the release criteria ===
-Much of the group activity in the last two weeks was centred around Fedora 12 release testing and validation. [[User:Jlaska|James Laska]] posted a recap<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00266.html</ref> of the final blocker bug review meeting for Fedora 12. James, [[User:Adamwill|Adam Williamson]] and [[User:Wwoods|Will Woods]] worked with the release engineering team throughout the final week before release to continuously monitor blocker bug status, test fixes, and monitor for newly identified blocker bugs and regressions.
-Four release candidate builds were produced by the release engineering group with the help of testing and feedback from QA. [[User:Liam|Liam Li]] co-ordinated the planned installation testing through the test compose and release candidate process<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00080.html</ref> <ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00235.html</ref>, and maintained the test results matrix<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_12_RC4_Install</ref>. He also sent a post mortem on the testing process<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00585.html</ref>. Many members of the group contributed valuable test reports to the matrix. Several group members, including Gianluca Cecchi, Gene Czarcinski and others posted test installation reports which helped identify important issues that were fixed or documented during the release process.
+[[User:poelstra|John Poelstra]] submitted a proposal<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00926.html</ref> for improving the release criteria<ref>http://fedoraproject.org/wiki/QA/ReleaseCriteria</ref> for future releases. The new proposed criteria <ref>http://fedoraproject.org/wiki/Fedora_Release_Criteria</ref> splits the old single page into an introductory / outline page and separate pages for each public release in the upcoming cycle. [[User:Adamwill|Adam Williamson]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00933.html</ref> and [[User:Jlaska|James Laska]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00992.html</ref> both replied to welcome to idea and post some suggestions for refinement. John plans to further refine the proposal and then have a session to discuss it at the upcoming FUDCon Toronto.
 <references/>

Search