From Fedora Project Wiki

< FWN‎ | Beats

(create 178 qa beat)
(create 179 qa beat)
Line 12: Line 12:
There was no Test Day last week, as we are deep in the Fedora 11 final release run-up.
There was no Test Day last week, as we are deep in the Fedora 11 final release run-up.


Currently, no Test Day is scheduled for next week - it is too close to the scheduled release of Fedora 11 for any testing to produce results directly in Fedora 11 final release, but if you would like to propose a test day which could result in changes for post-release updates, or an early test day for Fedora 12, please contact the QA team via email or IRC.
Currently, no Test Day is scheduled for next week - it is too close to the scheduled release of Fedora 11 for any testing to produce results directly in Fedora 11 final release, but if you would like to propose a test day which could result in changes for post-release updates, or an early test day for Fedora 12, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>https://fedorahosted.org/fedora-qa/</ref>.


<references/>
<references/>
Line 18: Line 18:
=== Weekly meetings ===
=== Weekly meetings ===


The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-05-27. The full log is available<ref>http://fedoraproject.org/wiki/QA/Meetings/20090527</ref>. [[User:Adamwill|Adam Williamson]] reported that he had again not yet remembered to ask the Bugzilla team to add a link to the Fedora bug workflow page.
The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-06-03. The full log is available<ref>http://fedoraproject.org/wiki/QA/Meetings/20090603</ref>. [[User:Adamwill|Adam Williamson]] reported that he had finally remembered to ask the Bugzilla team to add a link to the Fedora bug workflow page<ref>http://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow</ref> from the Bugzilla page<ref>https://bugzilla.redhat.com/page.cgi?id=fields.html</ref>. This has been done, and the link will show up with the next refresh of Bugzilla.


[[User:Jlaska|James Laska]] reported that he was still not yet ready to send out a Test Day feedback survey to previous participants, but continued to work on it.
[[User:Jlaska|James Laska]] reported that he has now sent out the survey about Fedora 11 Test Days, asking participants for feedback on how the events went and any possible improvements that could be made<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00160.html</ref>. Some feedback had already been received, and much more was expected.


[[User:poelstra|John Poelstra]] reported that he had updated the current Fedora 12 schedule<ref>http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng-tasks.html</ref>.
[[User:Wwoods|Will Woods]] reported that he had added two test cases for preupgrade <ref>http://fedoraproject.org/wiki/QA:Testcase_Preupgrade</ref>, <ref>http://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release</ref>, and updated the release candidate test matrix for RC3<ref>http://fedoraproject.org/wiki/QA:Fedora_11_RC3_Install_Test_Results</ref>.


[[User:Wwoods|Will Woods]] reported that he had added a test case for upgrading from one Fedora release to the next with an encrypted root partition<ref>https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Encrypted_Root</ref>.
The group discussed how to handle the installation test result matrix wiki page<ref>https://fedoraproject.org/wiki/QA:Fedora_11_RC2_Install_Test_Results</ref> between release candidate revisions. [[User:Jlaska|James Laska]] committed to work out his best solution and send it to the mailing list.


The group discussed how to handle the installation test result matrix wiki page<ref>https://fedoraproject.org/wiki/QA:Fedora_11_RC2_Install_Test_Results</ref> between release candidate revisions. [[User:Jlaska|James Laska]] committed to work out his best solution and send it to the mailing list.
[[User:Adamwill|Adam Williamson]] reported that he had added an entry to the Fedora 11 Common Bugs page<ref>http://fedoraproject.org/wiki/Common_F11_bugs</ref> for bug #502077<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=502077</ref>, but that the bug would now be fixed for final release and so the note should be removed. He clarified that issues which will be fixed for final release should just be removed from the page, not moved to the planned 'Resolved Issues' section.
 
The group discussed the state of Fedora 11 final release preparation. In general building of release candidates and testing was progressing smoothly. [[User:Jlaska|James Laska]] asked that the group make an effort to confirm the fixes for the nine release-critical issues marked as MODIFIED in Bugzilla.
 
The group then discussed the appropriate way to document bug #503824<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=503824</ref>, where installation fails in certain circumstances on an x86-64 system with only 512MB of memory. In the end it was decided the most appropriate way to address this would be in the minimum hardware requirements. [[User:Adamwill|Adam Williamson]] volunteered to add a request for some appropriate text to be added to an existing bug report on revision of the minimum requirements.
 
[[User:Jlaska|James Laska]] then started a brainstorming session for a general review of QA's role during the Fedora 11 cycle. Many ideas were contributed by the entire group. A summary of these is available on the meeting page<ref>http://fedoraproject.org/wiki/QA/Meetings/20090603#F-11_QA_Post-mortem_discussion</ref>.
 
The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-06-02. The full log is available<ref>http://fedoraproject.org/wiki/BugZappers/Meetings/Minutes-2009-Jun-02</ref>. [[User:poelstra|John Poelstra]] reported on progress of the housekeeping changes for Fedora 11's release, and the group agreed that he was doing a fine job and should keep it up.
 
[[User:Adamwill|Adam Williamson]] reported on the progress of the triage metric system. The system<ref>http://publictest14.fedoraproject.org/triageweb/</ref> is now running on the real Bugzilla data, updated nightly. The system is now in its beta stage, and the developer Brennan Ashton asks that people experiment with it and report bugs or feature requests to trac<ref>http://fedorahosted.org/triage</ref> (component triageweb).
 
[[User:Adamwill|Adam Williamson]] also reported on the progress of the proposal to include setting the priority / severity fields as part of triage. It is now waiting on a change by the Bugzilla maintainers to restrict access to the priority and severity fields. This is being tracked in a bug report<ref>http://bugzilla.redhat.com/show_bug.cgi?id=495985</ref>. [[User:arxs | Niels Haase]] noted that he had already begun setting severity on reports he is triaging, according to the policy, and had not yet met with any resistance on the part of reporters. The group agreed that triagers could go ahead and begin setting the severity field ahead of the change to Bugzilla, if they would like to.


[[User:Adamwill|Adam Williamson]] reported that the cleaning and revising of the Fedora 11 Common Bugs page<ref>https://fedoraproject.org/wiki/Common_F11_bugs</ref> was complete. [[User:Jlaska|James Laska]] added that he had as promised been adding significant installation issues to the page. Adam said that he had added the X.org issues of which he was aware, and sound-related issues. He noted that François Cami had created an initial draft of a list of ATI-related issues, but had not yet completed it.
[[User:arxs | Niels Haase]] flagged up a bug<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=459323</ref> for possible inclusion in the Fedora 11 Common Bugs page. It involves resume from suspend failing when using the nouveau graphics driver. After some discussion, the group agreed it should be added to the list.


[[User:Wwoods|Will Woods]] clarified that his preferred title in relation to autoqa issues is Cap'n Autoqa. The minutes do not relate whether or not there is a parrot.
[[User:poelstra|John Poelstra]] announced that he would be stepping back from some of his leadership role within the BugZappers group, though remaining involved in many ways. The group thanked him for all his efforts so far. [[User:Adamwill|Adam Williamson]], [[User:Tk009|Edward Kirk]] and [[User:arxs | Niels Haase]] will cover meeting arrangements for the foreseeable future.


The group reviewed the current status of Fedora 11 GA (final release) from its perspective. (Note that this meeting took place before the latest delay in the final ship date). They went over the list of currently open release blocker bugs, and agreed it seemed possible to make the final deadline for initial RC generation with all of the bugs at least tentatively resolved. There was detailed discussion of two bugs (502077 and 498553). Action plans were developed for both issues to have them addressed within the one-day deadline the team was at this point working with.
[[User:StevenParrish | Steven Parrish]] mentioned that he intended to go through all still-open Fedora 9 bugs for the components he triages, and try to determine whether they were still valid for a current release (and if so change them to that release), in advance of the automated closing of Fedora 9 bugs for EOL. [[User:Adamwill|Adam Williamson]] suggested he also mention this idea on the mailing list.


The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-05-26. The full log is available<ref>http://fedoraproject.org/wiki/BugZappers/Meetings/Minutes-2009-May-26</ref>. [[User:poelstra|John Poelstra]] reported on progress of the housekeeping changes for Fedora 11's release, and the group agreed that he was doing a fine job and should keep it up.
The next QA weekly meeting will be held on 2009-06-10 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-06-09 at 1500 UTC in #fedora-meeting.


[[User:Adamwill|Adam Williamson]] reported on the progress of the triage metric system. Significant progress had been made during the week by the author, Brennan Ashton. The system is now fully working on the official Fedora infrastructure hosting server<ref>http://publictest14.fedoraproject.org/triageweb/</ref>. It is currently working with a test snapshot of data rather than with the live Bugzilla data, but it should already be theoretically capable of working with the live data. The project will now enter a tidying-up and beta testing phase during which it will be brought up to a state where it can be declared fully usable. This should take two weeks or so. The group noted that the list of triagers was based on the FAS 'triagers' group, which leads back to the existing question of how to rationalize the 'fedorabugs' and 'triagers' groups. Brennan will work with [[User:Jstanley|Jon Stanley]] to address this issue.
<references/>


[[User:Adamwill|Adam Williamson]] also reported on the progress of the proposal to include setting the priority / severity fields as part of triage. As no feedback opposing the Cepl Method<ref>http://fedoraproject.org/wiki/User:Beland/Bugzilla_Legend#Proposal_B:</ref> had been received on the mailing list, the group agreed that it could now go ahead and adopt this as the official method of setting severity at the triage stage. Adam said he would work with the Bugzilla team to restrict access to the priority and severity fields as had been agreed as part of the proposal, and then adjust all the relevant documentation on the Wiki to put the severity policy into place.
=== Release candidate build availability ===


The next QA weekly meeting will be held on 2009-06-03 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-06-02 at 1500 UTC in #fedora-meeting.
Following on from last week's discussion of the availability of release candidate builds, Andre Robatino announced<ref>https://www.redhat.com/archives/fedora-test-list/2009-May/msg01372.html</ref> that he had built and made available delta ISOs - files containing the difference between two ISO images, allowing the reconstruction of the latest final image - for RC2, from Fedora 11 Preview. He later made delta ISOs available for RC3 and RC4. The group continued to discuss the feasibility of getting quickly-revised pre-release builds available from the public mirror system using various methods, but no conclusion has yet been reached.


<references/>
<references/>


=== Unified Greasemonkey triage script ===
=== Bugzilla statistics ===


Matej Cepl announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01131.html</ref> that he had released a new revised and unified Greasemonkey script for triagers incorporating all features of all previously released scripts. [[User:Tk009|Edward Kirk]] thanked him for his work<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01132.html</ref>. [[User:StevenParrish | Steven Parrish]] noted<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01163.html</ref> that GreaseMonkey did not yet work unmodified with the current Firefox 3.5 pre-release as found in Fedora 11. Matej suggested<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01174.html</ref> the Nightly Tester Tools extension as an easy way to work around this limitation.
Brennan Ashton released<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00093.html</ref> the first weekly Bugzilla statistics roundup, derived from the new triage metrics system. The response was enthusiastic, with requests and suggestions for more information from [[User:Johannbg|Jóhann Guðmundsson]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00097.html</ref> and [[User:Beland|Christopher Beland]]<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00121.html</ref>. There were also several positive responses on the development mailing list, where the information was also posted.


<references/>
<references/>


=== Merging Fedora 11 FAQ into other pages ===
=== 'How to report bugs' page revised ===


[[User:Beland|Christopher Beland]] revived<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01226.html</ref> the idea of merging the Fedora 11 FAQ<ref>http://fedoraproject.org/wiki/Fedora_11_FAQ</ref>, maintained by [[User:Sundaram | Rahul Sundaram]], into other pages, as most of its content could more appropriately be located in various other places, including the Release Notes, Installation Guide, Common Bugs page and other places. Rahul explained<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01228.html</ref> that he was happy for any content that could be moved to a more appropriate place to be removed from the FAQ page. The documentation team's [[User:Laubersm|Susan Lauber]] contributed some suggestions<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01246.html</ref> on other appropriate places the content could be moved to, and in a later thread she provided<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01328.html</ref> some more useful information on adding information to the Release Notes post-freeze.
[[User:Adamwill|Adam Williamson]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00137.html</ref> that he had made some changes to the main Wiki page on how to report bugs<ref>https://fedoraproject.org/wiki/Bugs_and_feature_requests</ref>. In particular, he had revised the section providing advice on what information to include in particular types of bug report to be more consistent. He encouraged everyone to contribute this type of information: if you know of specific information which is usually required when filing a particular type of bug (or a bug on a particular component), add this information following the layout used in the appropriate section of the page<ref>http://fedoraproject.org/wiki/Bugs_and_feature_requests#Tips_by_Type_of_Bug</ref>.


<references/>
<references/>


=== Release Candidate testing ===
=== Fedora 11 Test Day survey ===


[[User:Jlaska|James Laska]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01272.html</ref> testing for the first release candidate build for Fedora 11 (and, later, for the second<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01333.html</ref>). He asked for installation-related issues to be reported to the Wiki test matrix page<ref>https://fedoraproject.org/wiki/QA:Fedora_11_RC2_Install_Test_Results</ref>. This led indirectly to questions about where to find the release candidate images (their location is buried within the matrix page in order to try and limit demand for the images) and why release candidate images are not more widely promoted and distributed<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01308.html</ref>. [[User:jkeating|Jesse Keating]] explained <ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01309.html</ref> that the amounts of data were too great, the available storage and bandwidth resources too small, and the timeframes too tight for release candidate images to be meaningfully distributed for public testing. He did emphasize<ref>http://www.redhat.com/archives/fedora-test-list/2009-May/msg01286.html</ref>, however, that the community could contribute useful testing through use of the Rawhide repositories and installer images, which currently are synchronized with the release candidate builds.
[[User:Jlaska|James Laska]] posted a survey<ref>http://www.redhat.com/archives/fedora-test-list/2009-June/msg00160.html</ref> on the Fedora 11 Test Day process, asking for feedback on various facets of the process and suggestions for future improvements. The response was wide and enthusiastic, across both the QA and the development mailing lists, with many useful and constructive suggestions from testers and developers alike. James and [[User:Adamwill|Adam Williamson]] responded to several of the suggestions, affirming that many would be considered for implementation during the Fedora 12 Test Day cycle.


<references/>
<references/>

Revision as of 08:45, 6 June 2009

QualityAssurance

In this section, we cover the activities of the QA team[1].

Contributing Writer: Adam Williamson

Test Days

There was no Test Day last week, as we are deep in the Fedora 11 final release run-up.

Currently, no Test Day is scheduled for next week - it is too close to the scheduled release of Fedora 11 for any testing to produce results directly in Fedora 11 final release, but if you would like to propose a test day which could result in changes for post-release updates, or an early test day for Fedora 12, please contact the QA team via email or IRC, or file a ticket in QA Trac[1].

Weekly meetings

The QA group weekly meeting[1] was held on 2009-06-03. The full log is available[2]. Adam Williamson reported that he had finally remembered to ask the Bugzilla team to add a link to the Fedora bug workflow page[3] from the Bugzilla page[4]. This has been done, and the link will show up with the next refresh of Bugzilla.

James Laska reported that he has now sent out the survey about Fedora 11 Test Days, asking participants for feedback on how the events went and any possible improvements that could be made[5]. Some feedback had already been received, and much more was expected.

Will Woods reported that he had added two test cases for preupgrade [6], [7], and updated the release candidate test matrix for RC3[8].

The group discussed how to handle the installation test result matrix wiki page[9] between release candidate revisions. James Laska committed to work out his best solution and send it to the mailing list.

Adam Williamson reported that he had added an entry to the Fedora 11 Common Bugs page[10] for bug #502077[11], but that the bug would now be fixed for final release and so the note should be removed. He clarified that issues which will be fixed for final release should just be removed from the page, not moved to the planned 'Resolved Issues' section.

The group discussed the state of Fedora 11 final release preparation. In general building of release candidates and testing was progressing smoothly. James Laska asked that the group make an effort to confirm the fixes for the nine release-critical issues marked as MODIFIED in Bugzilla.

The group then discussed the appropriate way to document bug #503824[12], where installation fails in certain circumstances on an x86-64 system with only 512MB of memory. In the end it was decided the most appropriate way to address this would be in the minimum hardware requirements. Adam Williamson volunteered to add a request for some appropriate text to be added to an existing bug report on revision of the minimum requirements.

James Laska then started a brainstorming session for a general review of QA's role during the Fedora 11 cycle. Many ideas were contributed by the entire group. A summary of these is available on the meeting page[13].

The Bugzappers group weekly meeting[14] was held on 2009-06-02. The full log is available[15]. John Poelstra reported on progress of the housekeeping changes for Fedora 11's release, and the group agreed that he was doing a fine job and should keep it up.

Adam Williamson reported on the progress of the triage metric system. The system[16] is now running on the real Bugzilla data, updated nightly. The system is now in its beta stage, and the developer Brennan Ashton asks that people experiment with it and report bugs or feature requests to trac[17] (component triageweb).

Adam Williamson also reported on the progress of the proposal to include setting the priority / severity fields as part of triage. It is now waiting on a change by the Bugzilla maintainers to restrict access to the priority and severity fields. This is being tracked in a bug report[18]. Niels Haase noted that he had already begun setting severity on reports he is triaging, according to the policy, and had not yet met with any resistance on the part of reporters. The group agreed that triagers could go ahead and begin setting the severity field ahead of the change to Bugzilla, if they would like to.

Niels Haase flagged up a bug[19] for possible inclusion in the Fedora 11 Common Bugs page. It involves resume from suspend failing when using the nouveau graphics driver. After some discussion, the group agreed it should be added to the list.

John Poelstra announced that he would be stepping back from some of his leadership role within the BugZappers group, though remaining involved in many ways. The group thanked him for all his efforts so far. Adam Williamson, Edward Kirk and Niels Haase will cover meeting arrangements for the foreseeable future.

Steven Parrish mentioned that he intended to go through all still-open Fedora 9 bugs for the components he triages, and try to determine whether they were still valid for a current release (and if so change them to that release), in advance of the automated closing of Fedora 9 bugs for EOL. Adam Williamson suggested he also mention this idea on the mailing list.

The next QA weekly meeting will be held on 2009-06-10 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-06-09 at 1500 UTC in #fedora-meeting.

Release candidate build availability

Following on from last week's discussion of the availability of release candidate builds, Andre Robatino announced[1] that he had built and made available delta ISOs - files containing the difference between two ISO images, allowing the reconstruction of the latest final image - for RC2, from Fedora 11 Preview. He later made delta ISOs available for RC3 and RC4. The group continued to discuss the feasibility of getting quickly-revised pre-release builds available from the public mirror system using various methods, but no conclusion has yet been reached.

Bugzilla statistics

Brennan Ashton released[1] the first weekly Bugzilla statistics roundup, derived from the new triage metrics system. The response was enthusiastic, with requests and suggestions for more information from Jóhann Guðmundsson[2] and Christopher Beland[3]. There were also several positive responses on the development mailing list, where the information was also posted.

'How to report bugs' page revised

Adam Williamson announced[1] that he had made some changes to the main Wiki page on how to report bugs[2]. In particular, he had revised the section providing advice on what information to include in particular types of bug report to be more consistent. He encouraged everyone to contribute this type of information: if you know of specific information which is usually required when filing a particular type of bug (or a bug on a particular component), add this information following the layout used in the appropriate section of the page[3].

Fedora 11 Test Day survey

James Laska posted a survey[1] on the Fedora 11 Test Day process, asking for feedback on various facets of the process and suggestions for future improvements. The response was wide and enthusiastic, across both the QA and the development mailing lists, with many useful and constructive suggestions from testers and developers alike. James and Adam Williamson responded to several of the suggestions, affirming that many would be considered for implementation during the Fedora 12 Test Day cycle.