From Fedora Project Wiki

< FWN‎ | Beats

(fwn 204 beat)
(fwn 206 beat)
Line 16: Line 16:
=== Weekly meetings ===
=== Weekly meetings ===


The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-11-23. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-23/fedora-meeting.2009-11-23-16.00.log.html</ref>. [[User:Jlaska|James Laska]] noted that a common bugs page entry had been added<ref>http://fedoraproject.org/wiki/Common_F12_bugs#preupgrade-boot</ref> to cover the known issue with preupgrade and free space in the /boot partition, and [[User:Rhe|Rui He]] had been working to update the preupgrade test cases to catch similar problems in future<ref>http://fedorahosted.org/fedora-qa/ticket/30</ref>.
As the QA beat was unfortunately not present for Fedora Weekly News #205, we will cover two weeks' worth of events below.


[[User:Jlaska|James Laska]] admitted that he had not yet sent out the request for feedback for the Fedora 12 QA retrospective, but promised to do it soon. [[User:poelstra|John Poelstra]] asked whether the group would be interested in a project-wide retrospective at the upcoming FUDCon; James offered to discuss the idea with John after the meeting.
A QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-11-30. No meeting was held on 2009-12-07 due to many group members being absent at FUDCon or elsewhere. The full log of the 2009-11-30 meeting is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-30/qa.2009-11-30-16.00.log.html</ref>. [[User:Adamwill|Adam Williamson]] reported that he had started a development mailing list conversation regarding the proposed privilege escalation policy<ref>http://www.redhat.com/archives/fedora-devel-list/2009-November/msg01745.html</ref>.


The group discussed the question of privilege escalation testing, following the PackageKit installation permission controversy<ref>http://bugzilla.redhat.com/show_bug.cgi?id=534047</ref>. [[User:Jlaska|James Laska]] wanted to discuss the plan [[TomCallaway|Tom 'spot' Callaway]] had proposed via a blog post<ref>http://spot.livejournal.com/312216.html</ref> and create a test plan based around it. [[User:Adamwill|Adam Williamson]] felt it was too early to begin planning testing, since Tom's blog post was only a proposal, and there was no official policy or guideline for privilege escalation issues on which a test plan could be based. Adam was also worried about defining the scope of testing, as checking every package in the distribution would be impractical given the size of the QA team. The group agreed that for any useful testing to be done, two things would be needed: a project-wide policy or set of policies and guidelines, and a tool for generating a list of packages which are capable of privilege escalation. Adam agreed to start a discussion of this on the development and security mailing lists. [[User:Wwoods|Will Woods]] offered to work on the tool for identifying escalation-capable packages.
[[User:Jlaska|James Laska]] thanked the group for their feedback on [[User:poelstra|John Poelstra's]] plan to improve the release criteria<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00926.html</ref>, and asked for comments on the planned next steps. John said he was planning to work in all the received feedback, send a revised draft of the page to the mailing list for comment, and do the final touches in a hackfest at the upcoming FUDCon. [[User:Adamwill|Adam Williamson]] volunteered to write a proposed paragraph to cover subjective judgment of configuration-dependent problems.


[[User:Jlaska|James Laska]] brought up [[User:poelstra|John Poelstra's]] plan to improve the release criteria<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00926.html</ref>, and asked the group to provide feedback. John noted that he was hoping people could get together to work on finalizing the new criteria at FUDCon.
[[User:Adamwill|Adam Williamson]] gave a more detailed update on the privilege escalation policy topic. He said the development mailing list discussion had generated some useful points and potential issues that should be dealt with by any policy, but no clear road forward. With some help from [[User:Kevin|Kevin Fenzi]], a plan was made for Adam to escalate the issue for consideration by FESco with reference to the previous discussion, and possibly a proposed policy created with the help of the security team.


[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Will had completed the redesign of the autoqa code to be based around a Python shared library containing functions commonly used in multiple watchers and tests. The new post-koji-build test hook is also included, and autoqa is currently running an rpmlint test on every Koji build to test the hook. He said the next objective was to solidify the post-koji-build hook, help package maintainers add post-build tests, and get the rpmguard test running. A later objective is to work on a post-bodhi-update hook and dependency check test so that all updates submitted to Bodhi will be checked for dependency consistency, to hopefully end the situation where updates are pushed which break dependency chains. Kamil had been working on the Wiki documentation, and had created a new front page<ref>http://fedoraproject.org/wiki/AutoQA</ref> which briefly explains the project and contains links to the most important relevant pages. He also pointed out that [[User:Jlaska|James Laska]] had been drafting further improvements to this page<ref>http://fedoraproject.org/wiki/User:Jlaska/Draft</ref>.
[[User:Jlaska|James Laska]] gave an update on the Fedora 12 QA retrospective. He thanked the group for their feedback, and outlined his plans. He intends to organize the feedback into topic groups to try and identify trends, and then discuss what can be improved for the Fedora 13 cycle based on that feedback.


[[User:jkeating|Jesse Keating]] proposed a talk during FUDCon to explain how several new ideas across the release engineering and QA groups - no frozen rawhide, autoqa, autosigning, and new milestones - would fit together in upcoming Fedora release cycles. The group thought this was a good idea, and Jesse said he would take the lead in arranging it.
[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Kamil had written some patches intended to make test development easier, providing --help and --dry-run parameters for each watcher. He now plans to document these improvements in the wiki once they are accepted. He had also worked on integrating rpmguard into AutoQA. Will had added --local and --dry-run parameters to the test harness so that tests can be run locally by developers. He had fixed various watchers to run tests only once when repositories are changed or builds are run, even if multiple architectures are changed. He was planning to write some notes on AutoQA for Fedora developrs for use at the upcoming FUDCon, and help Kamil with rpmguard integration. [[User:Jlaska|James Laska]] reported that packaging of the israwhidebroken code was now complete, with help from [[ToshioKuratomi|Toshio Kuratomi]].


The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-11-24. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-24/fedora-meeting.2009-11-24-15.11.log.html</ref>. The group discussed housekeeping tasks, particularly updating the components and triagers page<ref>http://fedoraproject.org/wiki/BugZappers/Components_and_Triagers</ref>. [[User:Adamwill|Adam Williamson]] thought the list of triagers should be kept (rather than being emptied as was previously the case with each new release) but pruned, with triagers known to be inactive being removed. [[User:Tk009|Edward Kirk]] volunteered to look into a method for updating the component list, based on the current critical path package list.
[[User:Adamwill|Adam Williamson]] proposed that Glögg<ref>http://loupgaroublond.blogspot.com/2009/11/glogg.html</ref> be adopted as the official QA team drink. [[User:jkeating|Jesse Keating]] suggested magic hobo gravy<ref>http://hijinksensue.com/2009/11/27/the-special-sauce/</ref> as an alternative. No clear consensus was reached on the issue.


The group then discussed the topic of mentoring new members, with [[User:Tk009|Edward Kirk]] encouraging experienced group members to help mentor new ones to make sure they got a good start on their triaging careers. He also thought it would be good for existing members to join in welcoming new members to the group when they posted their introduction emails. [[User:Adamwill|Adam Williamson]] suggested doing this via private mail to avoid cluttering up the list.
A Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-12-01. No meeting was held on 2009-12-08 due to many members being absent. The full log of the 2009-12-01 meeting is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-12-01/fedora-meeting.2009-12-01-15.06.log.html</ref>. [[User:Rjune|Richard June]] noted that updating the components and triagers page<ref>http://fedoraproject.org/wiki/BugZappers/Components_and_Triagers</ref> had been left with [[User:Tk009|Edward Kirk]], who was not present to report on it. [[User:Adamwill|Adam Williamson]] encouraged all group members to make sure they were listed on the page next to the correct components.


[[User:Mcepl|Matej Cepl]] brought up a problem related to the recently-implemented change in the method of marking bugs that had been triaged. He had found that the fact that this was now being done differently for different releases made it impossible to construct a Bugzilla search for all triaged or un-triaged bugs in a given component across all releases. To address this problem, he proposed adding the new Triaged keyword to all bugs in ASSIGNED state for existing supported releases (Fedora 10 through 12), which would allow searches to be performed using the keyword in all releases. The group could see no problems with this idea, as long as it was done without generating a large amount of email, and approved the plan for Matej to approach the Bugzilla maintainer for help in implementing it.
[[User:Adamwill|Adam Williamson]] started a discussion on the topic of anaconda triage. He explained that the anaconda team could benefit from having a triage volunteer, as [[User:andyl|Andy Lindeberg]] was no longer working on anaconda triage. This would take a significant amount of time and require knowledge of anaconda, but the anaconda team was prepared to help train volunteers. [[User:Rjune|Richard June]] volunteered to send a mail to the mailing list with details of the requirements, to ask for volunteers.


[[User:Mcepl|Matej Cepl]] pointed out that the level of duplicate bugs being filed via the abrt<ref>http://fedorahosted.org/abrt/wiki</ref> automated bug reporting tool was increasing the triage workload on some components significantly. After a long discussion, the group agreed a plan to try and address this. [[User:Wwoods|Will Woods]] would talk to the abrt team about the idea of reporting issues to an intermediate, abrt-specific server rather than directly to Bugzilla, based on the kerneloops.org<ref>http://www.kerneloops.org</ref> model. Matej would talk to the abrt team about their plans to improve abrt's own automatic duplicate detection and about having abrt format its reports in ways that would aid triagers in manual duplicate detection. [[User:Adamwill|Adam Williamson]] would respond to the existing thread on the development mailing list about the problem to raise the group's concerns, and ask the abrt team whether future improvements to abrt's duplicate detection logic could be retrospectively applied to bugs already filed by older versions of abrt.
[[User:Adamwill|Adam Williamson]] took another look at the triage metrics situation. No-one had heard from Brennan Ashton lately, but Adam would try to check in with him at the upcoming FUDCon event. Adam also pointed out the fairly new statistics system within Bugzilla<ref>http://bugzilla.redhat.com/browse.cgi</ref> as a possibility for the group to explore.


The next QA weekly meeting will be held on 2009-11-30 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-12-01 at 1500 UTC in #fedora-meeting.
[[User:Mcepl|Matej Cepl]] reported that he had completed the agreed plan to add the Triaged keyword to all ASSIGNED bugs in Fedora 10 through 12.
 
The next QA weekly meeting may be held on 2009-11-07 at 1600 UTC in #fedora-meeting, although several group members are at FUDCon during that time and may not be able to attend. The next Bugzappers weekly meeting will be held on 2009-12-08 at 1500 UTC in #fedora-meeting.
 
<references/>
 
=== FUDCon Toronto ===
 
The QA and BugZappers groups were both well represented at FUDCon Toronto<ref>http://fedoraproject.org/wiki/FUDCon:Toronto_2009</ref>. [[User:Jlaska|James Laska]], [[User:Wwoods|Will Woods]], [[User:Adamwill|Adam Williamson]], [[User:poelstra|John Poelstra]], [[User:StevenParrish|Steven M. Parrish]], [[User:Bashton|Brennan Ashton (comphappy)]] all attended, and a brand new BugZapper - Patrick Ian - was welcomed during the event. Steven, Will and Adam all gave talks: Steven<ref>http://meetbot.fedoraproject.org/fudcon-room-5/2009-12-05/fudcon-room-5.2009-12-05-17.11.log.html</ref> on effective bug reporting, Will<ref>http://meetbot.fedoraproject.org/fudcon-room-2/2009-12-05/fudcon-room-2.2009-12-05-21.12.log.html</ref> on AutoQA, and Adam<ref>http://meetbot.fedoraproject.org/fudcon-room-5/2009-12-05/fudcon-room-5.2009-12-05-19.09.log.html</ref> on getting involved with QA and BugZappers. Several group members also posted write-ups of the event: John<ref>http://poelcat.wordpress.com/2009/12/09/fudcon-toronto-trip-report/</ref>, Steven<ref>http://smparrish.livejournal.com/11639.html</ref>, and Adam<ref>http://www.happyassassin.net/2009/12/10/fudcon-toronto-2009-wrap-up/</ref>.
 
=== BugZappers triaged bugs policy change ===
 
[[User:Adamwill|Adam Williamson]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01228.html</ref> a small modification to the recently-implemented triaged bug policy change. To make searching easier, triagers should now add the Triaged keyword to all Fedora 11 and Fedora 12 bugs when they have been triaged, as well as marking them as ASSIGNED.
 
<references/>
 
=== Release criteria revision ===
 
[[User:poelstra|John Poelstra]] posted a final request<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00047.html</ref> for feedback on the proposed new release criteria<ref>https://fedoraproject.org/wiki/Fedora_Release_Criteria</ref>, noting that the pages would be finalized and made active at the upcoming FUDCon Toronto. At FUDCon, a group including John, [[User:Jlaska|James Laska]], [[User:Adamwill|Adam Williamson]], [[User:Notting|Bill Nottingham]] and [[User:tburke|Tim Burke]] revised the pages including all feedback provided from the list, and then made further revisions based on feedback by the anaconda and desktop teams. Adam announced the changes to the list<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00137.html</ref>. Some further changes were then made based on suggestions from A. Mani<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00140.html</ref> and others.
 
<references/>
 
=== Bugzilla 3.4 public beta ===
 
[[User:Jlaska|James Laska]] passed on an announcement<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00099.html</ref> from Red Hat's Bugzilla team that a public beta of Bugzilla 3.4 for the Red Hat Bugzilla instance (which Fedora shares) is now available<ref>http://partner-bugzilla.redhat.com/</ref>. Later, James passed on the announcement for the second beta<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00212.html</ref>.


<references/>
<references/>


=== Increasing the grub timeout ===
=== Fedora 13 schedule ===


Scott Robbins started a long thread<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01012.html</ref> with the suggestion to increase the default timeout for the Fedora boot loader from its current default setting of 0 (which causes the boot loader menu never to be shown at all). There were many opinions on this idea, but the general response was positive enough for Scott to file a feature request<ref>http://bugzilla.redhat.com/show_bug.cgi?id=541315</ref> on the idea, where some compromises were suggested. Richard Ryniker suggested having the system detect unclean shutdowns and force the boot menu to be displayed on the next boot (much as Windows does). Stewart Adam suggested having grub initially installed with a non-zero timeout, and have firstboot change it to zero on the assumption that a system that can get to firstboot must have a properly configured bootloader.
[[User:poelstra|John Poelstra]] provided<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00101.html</ref> a draft schedule of QA tasks for the Fedora 13 release. He later provided an updated schedule<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00176.html</ref> based on discussion at FUDCon Toronto. [[User:Jlaska|James Laska]] requested<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00177.html</ref> an extra pre-alpha acceptance milestone on 2009-01-21.


<references/>
<references/>


=== Fedora 12 QA retrospective ===
=== Fedora 13 Alpha release notes ===


[[User:Jlaska|James Laska]] posted a request<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01126.html</ref> for feedback on the Fedora 12 QA cycle from anyone, both on things that went well and areas that could be improved. Many group members posted replies, including [[User:Adamwill|Adam Williamson]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01127.html</ref>, [[User:Johannbg|Jóhann Guðmundsson]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01149.html</ref>, and [[User:Sundaram|Rahul Sundaram]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg01128.html</ref>.  
[[User:Sundaram|Rahul Sundaram]] posted<ref>http://www.redhat.com/archives/fedora-test-list/2009-December/msg00138.html</ref> a preliminary draft of the Fedora 13 Alpha release notes for comment and discussion.


<references/>
<references/>

Revision as of 00:31, 12 December 2009

QualityAssurance

In this section, we cover the activities of the QA team[1].

Contributing Writer: Adam Williamson

Test Days

There was no Test Day last week, and no Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac[1].

Weekly meetings

As the QA beat was unfortunately not present for Fedora Weekly News #205, we will cover two weeks' worth of events below.

A QA group weekly meeting[1] was held on 2009-11-30. No meeting was held on 2009-12-07 due to many group members being absent at FUDCon or elsewhere. The full log of the 2009-11-30 meeting is available[2]. Adam Williamson reported that he had started a development mailing list conversation regarding the proposed privilege escalation policy[3].

James Laska thanked the group for their feedback on John Poelstra's plan to improve the release criteria[4], and asked for comments on the planned next steps. John said he was planning to work in all the received feedback, send a revised draft of the page to the mailing list for comment, and do the final touches in a hackfest at the upcoming FUDCon. Adam Williamson volunteered to write a proposed paragraph to cover subjective judgment of configuration-dependent problems.

Adam Williamson gave a more detailed update on the privilege escalation policy topic. He said the development mailing list discussion had generated some useful points and potential issues that should be dealt with by any policy, but no clear road forward. With some help from Kevin Fenzi, a plan was made for Adam to escalate the issue for consideration by FESco with reference to the previous discussion, and possibly a proposed policy created with the help of the security team.

James Laska gave an update on the Fedora 12 QA retrospective. He thanked the group for their feedback, and outlined his plans. He intends to organize the feedback into topic groups to try and identify trends, and then discuss what can be improved for the Fedora 13 cycle based on that feedback.

Will Woods and Kamil Paral reported on the progress of the AutoQA project. Kamil had written some patches intended to make test development easier, providing --help and --dry-run parameters for each watcher. He now plans to document these improvements in the wiki once they are accepted. He had also worked on integrating rpmguard into AutoQA. Will had added --local and --dry-run parameters to the test harness so that tests can be run locally by developers. He had fixed various watchers to run tests only once when repositories are changed or builds are run, even if multiple architectures are changed. He was planning to write some notes on AutoQA for Fedora developrs for use at the upcoming FUDCon, and help Kamil with rpmguard integration. James Laska reported that packaging of the israwhidebroken code was now complete, with help from Toshio Kuratomi.

Adam Williamson proposed that Glögg[5] be adopted as the official QA team drink. Jesse Keating suggested magic hobo gravy[6] as an alternative. No clear consensus was reached on the issue.

A Bugzappers group weekly meeting[7] was held on 2009-12-01. No meeting was held on 2009-12-08 due to many members being absent. The full log of the 2009-12-01 meeting is available[8]. Richard June noted that updating the components and triagers page[9] had been left with Edward Kirk, who was not present to report on it. Adam Williamson encouraged all group members to make sure they were listed on the page next to the correct components.

Adam Williamson started a discussion on the topic of anaconda triage. He explained that the anaconda team could benefit from having a triage volunteer, as Andy Lindeberg was no longer working on anaconda triage. This would take a significant amount of time and require knowledge of anaconda, but the anaconda team was prepared to help train volunteers. Richard June volunteered to send a mail to the mailing list with details of the requirements, to ask for volunteers.

Adam Williamson took another look at the triage metrics situation. No-one had heard from Brennan Ashton lately, but Adam would try to check in with him at the upcoming FUDCon event. Adam also pointed out the fairly new statistics system within Bugzilla[10] as a possibility for the group to explore.

Matej Cepl reported that he had completed the agreed plan to add the Triaged keyword to all ASSIGNED bugs in Fedora 10 through 12.

The next QA weekly meeting may be held on 2009-11-07 at 1600 UTC in #fedora-meeting, although several group members are at FUDCon during that time and may not be able to attend. The next Bugzappers weekly meeting will be held on 2009-12-08 at 1500 UTC in #fedora-meeting.

FUDCon Toronto

The QA and BugZappers groups were both well represented at FUDCon Toronto[1]. James Laska, Will Woods, Adam Williamson, John Poelstra, Steven M. Parrish, Brennan Ashton (comphappy) all attended, and a brand new BugZapper - Patrick Ian - was welcomed during the event. Steven, Will and Adam all gave talks: Steven[2] on effective bug reporting, Will[3] on AutoQA, and Adam[4] on getting involved with QA and BugZappers. Several group members also posted write-ups of the event: John[5], Steven[6], and Adam[7].

BugZappers triaged bugs policy change

Adam Williamson announced[8] a small modification to the recently-implemented triaged bug policy change. To make searching easier, triagers should now add the Triaged keyword to all Fedora 11 and Fedora 12 bugs when they have been triaged, as well as marking them as ASSIGNED.

Release criteria revision

John Poelstra posted a final request[1] for feedback on the proposed new release criteria[2], noting that the pages would be finalized and made active at the upcoming FUDCon Toronto. At FUDCon, a group including John, James Laska, Adam Williamson, Bill Nottingham and Tim Burke revised the pages including all feedback provided from the list, and then made further revisions based on feedback by the anaconda and desktop teams. Adam announced the changes to the list[3]. Some further changes were then made based on suggestions from A. Mani[4] and others.

Bugzilla 3.4 public beta

James Laska passed on an announcement[1] from Red Hat's Bugzilla team that a public beta of Bugzilla 3.4 for the Red Hat Bugzilla instance (which Fedora shares) is now available[2]. Later, James passed on the announcement for the second beta[3].

Fedora 13 schedule

John Poelstra provided[1] a draft schedule of QA tasks for the Fedora 13 release. He later provided an updated schedule[2] based on discussion at FUDCon Toronto. James Laska requested[3] an extra pre-alpha acceptance milestone on 2009-01-21.

Fedora 13 Alpha release notes

Rahul Sundaram posted[1] a preliminary draft of the Fedora 13 Alpha release notes for comment and discussion.