From Fedora Project Wiki

< FWN‎ | Beats

(https fix)
(create fwn 221 qa beat)
Line 2: Line 2:
== QualityAssurance ==
== QualityAssurance ==


In this section, we cover the activities of the QA team<ref>http://fedoraproject.org/wiki/QA</ref>. It seems like the new layout has been a success, so we'll stick with it for the future.
In this section, we cover the activities of the QA team<ref>http://fedoraproject.org/wiki/QA</ref>. For more information on the work of the QA team and how you can get involved, see the Joining page<ref>http://fedoraproject.org/wiki/QA/Join</ref>.


Contributing Writer: [[User:Adamwill|Adam Williamson]]
Contributing Writer: [[User:Adamwill|Adam Williamson]]
Line 10: Line 10:
=== Test Days ===
=== Test Days ===


We had two Test Days last week. The first<ref>http://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault</ref>, on Tuesday 2010-03-30, was on the implementation of SSSD by default<ref>http://fedoraproject.org/wiki/Features/SSSDByDefault</ref>. This feature is very useful to those who use accounts on a remote server which may not always be accessible from their system. A small group of dedicated testers were able to expose five bugs, which the developers are now investigating. The second<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT</ref> was on ABRT<ref>http://fedoraproject.org/wiki/Features/ABRT</ref>, the automated bug report tool which has been included with Fedora by default since the release of Fedora 12. This event resulted in mostly positive test results, but also exposed six bugs, three of which have already been closed by the developers.
Last week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-08_Virtualization</ref> was on virtualization<ref>http://fedoraproject.org/wiki/Virtualization</ref>. This was mainly focused on the Fedora virtualization stack, based around KVM, libvirt, and virt-manager. A small band of hardened virtualization testers were able to expose 14 bugs, which the developers are now investigating. Thanks to everyone who came out to help with the testing.


This week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-08_Virtualization</ref>, on Thursday 2010-04-08, will be on virtualization<ref>http://fedoraproject.org/wiki/Virtualization</ref>. This is mainly focused on the Fedora virtualization stack, based around KVM, libvirt, and virt-manager. There've been many improvements throughout the stack for Fedora 13, so there's a lot of new stuff to test as well as making sure the basic functions are working properly. Virtualization is a very important area these days, so please come out to help us test it! The event will take place all day in the #fedora-test-day channel on Freenode IRC. If you can't make it on the day, you can still provide your results on the Wiki page before or after the event.
This week is a big moment in the Test Day schedule: Graphics Test Week. There will be three Test Days focusing on the three major graphics drivers: NVIDIA Test Day on Tuesday 2010-04-13<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-13_Nouveau</ref>, ATI/AMD Test Day on Wednesday 2010-04-14<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-14_Radeon</ref>, and Intel graphics Test Day on Thursday 2010-04-15<ref>http://fedoraproject.org/wiki/Test_Day:2010-04-15_Intel</ref>. As always, widespread graphics testing is critical to the development of these drivers. Around 75% of all bugs reported in the last Graphics Test Week have been closed (either as fixed, or as duplicates), so the information gathered isn't ignored! Testing can be done with a live image, so there's no need to have an unstable Fedora installation to do the testing, and the tests are easy to do and come with full instructions. Almost everyone has an NVIDIA, AMD/ATI or Intel graphics adapter, so please come out to help us test! The events will take place all day in the #fedora-test-day channel on Freenode IRC (if you're not sure how to use IRC, there's an instruction page<ref>http://fedoraproject.org/wiki/How_to_use_IRC</ref>, or you can use WebIRC<ref>http://webchat.freenode.net/?channels=fedora-test-day</ref>. If you can't make it on the day, you can still provide your results on the Wiki page before or after the event.


If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
Line 20: Line 20:
=== Fedora 13 testing ===
=== Fedora 13 testing ===


Once again, the week was occupied with much testing of Fedora 13 Beta candidate builds. The second release candidate was announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089807.html</ref> on Wednesday 2010-03-31, and closely followed by the third release candidate on the same day<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089826.html</ref>. As usual, Andre Robatino provided delta ISOs<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089833.html</ref>. Installation validation testing on RC3<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC3_Install</ref> revealed a blocker bug in support for encrypted partitions<ref>http://bugzilla.redhat.com/show_bug.cgi?id=578633</ref>. This led the group, along with release engineering and development groups, to take the decision to delay the Beta release by a week<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089828.html</ref>.
This week saw the group wrap up Fedora 13 Beta validation testing. After the previous week's delay, the fourth<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089919.html</ref> and fifth<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089962.html</ref> release candidate builds for the Beta arrived during the week. Installation<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC4_Install</ref> and desktop<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC4_Desktop</ref> validation testing for the RC4 build were both broadly successful, but [[User:Adamwill|Adam Williamson]] realized that the build included a critical bug which would cause systems containing a certain common network adapter to be unable to boot<ref>http://bugzilla.redhat.com/show_bug.cgi?id=577463</ref>, so the RC5 build provided an updated kernel to fix that issue. Adam posted a call for testing of the updated kernel<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089944.html</ref> which drew an overwhelming response, with dozens of group members confirming the kernel worked on their systems. The group re-ran the validation tests<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC5_Install</ref>, and subsequently agreed with the development and release engineering groups at the go/no-go meeting<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-04-08/f-13-beta-eng-readiness.2010-04-08-00.01.html</ref> that the RC5 build met all the release criteria<ref>http://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria</ref> and so was suitable for release as Fedora 13 Beta. [[User:Rhe|Rui He]] summarized the validation test results<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090015.html</ref> and encouraged more group members to be involved in the validation testing for future releases.


<references/>
<references/>


=== Update acceptance testing ===
=== Testing non-English keyboard layouts ===


[[User:maxamillion|Adam Miller]] revised his proventesters SOP proposal<ref>http://fedoraproject.org/wiki/QA/JoinProvenTesters:Draft</ref> based on feedback received from [[User:Adamwill|Adam Williamson]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089801.html</ref> and [[User:jkeating|Jesse Keating]]<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089780.html</ref>.
Petri Laine reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089995.html</ref> that he had experienced problems using a non-default keyboard mapping in Fedora 13 Beta RC5. [[User:Adamwill|Adam Williamson]] replied<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089998.html</ref> that similar bugs had occurred during previous release periods, and then announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090001.html</ref> that he had extended an installation validation test case<ref>http://fedoraproject.org/wiki/QA:Testcase_Anaconda_autopart_%28encrypted%29_install</ref> and created a desktop validation test case<ref>http://fedoraproject.org/wiki/QA:Testcase_desktop_login</ref> to try to ensure that similar issues are caught in future testing rounds. Petri appended his report to an existing bug report<ref>http://bugzilla.redhat.com/show_bug.cgi?id=571900</ref> and followed up on the problem there.


<references/>
<references/>


=== Stock response for ABRT symbol failures ===
=== Ensuring packages are signed ===


[[User:Beland|Christopher Beland]] proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089875.html</ref> a BugZappers stock response for the situation where ABRT fails to download the packages to provide debugging symbols in an automatically-generated bug report. After receiving no objections, he later reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089902.html</ref> that he had added this to the page<ref>http://fedoraproject.org/wiki/BugZappers/StockBugzillaResponses#No_Debugging_Symbols</ref>.
[[User:Jlaska|James Laska]] proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090025.html</ref> a new release criterion and validation test to ensure that all packages are signed with a valid Feora GPG signature. [[User:Notting|Bill Nottingham]] pointed out<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090026.html</ref> that this would not slot easily into the existing package release workflow. He also noted that Bodhi is supposed to reject un-signed packages. [[User:jkeating|Jesse Keating]] explained<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090028.html</ref> that this was a mash configuration option which had been disabled intentionally for initial Branched composes as some packages were known not to be signed at that time. Bill ultimately suggested<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/090032.html</ref> that the signature check should be re-enabled in the relevant mash configurations.


<references/>
<references/>


=== BugZapping classes ===
=== Bugzappers screencasts ===


During the weekly BugZappers meeting<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-03-30/fedora-meeting.2010-03-30-15.00.log.html</ref>, the group discussed the possibility of providing more ways for new members to get started. [[User:campbecg|Chris Campbell]] volunteered to look into the possibility of running a test Bugzapping class. [[User:Shakthimaan|Shakthi Kannan]] suggested creating screencasts as examples of each step in the triage process, and [[User:Adamwill|Adam Williamson]] said he would forward this idea to the mailing list.
At the weekly Bugzappers meeting<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-04-06/fedora-meeting.2010-04-06-15.00.log.html</ref>, [[User:Adamwill|Adam Williamson]] noted that he had not yet forwarded [[User:Shakthimaan|Shakthi Kannan's]] suggestion of making Bugzapping screencasts to the mailing list. The next day, he did so<ref>http://lists.fedoraproject.org/pipermail/test/2010-April/089972.html</ref>. [[User:etank|Eric Lake]], [[User:campbecg|Chris Campbell]] and James Gledhill all posted in support of the idea, but no-one yet had the combination of free time and expertise to make the screencasts.
 
<references/>
 
=== AutoQA ===
 
With Fedora 13 validation testing winding down, work on AutoQA was picking up steam again, with the team working on dependency checking<ref>http://fedorahosted.org/autoqa/milestone/autoqa depcheck</ref>, tests<ref>http://fedorahosted.org/autoqa/milestone/Package%20Sanity%20Tests</ref> to implement the Package Sanity Test Plan<ref>http://fedoraproject.org/wiki/QA:Package_Sanity_Test_Plan</ref>, the results database idea<ref>http://fedorahosted.org/autoqa/milestone/Resultdb</ref> and the automated installation test plan<ref>http://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan</ref>.


<references/>
<references/>

Revision as of 23:45, 12 April 2010

QualityAssurance

In this section, we cover the activities of the QA team[1]. For more information on the work of the QA team and how you can get involved, see the Joining page[2].

Contributing Writer: Adam Williamson

  1. ↑ http://fedoraproject.org/wiki/QA
  2. ↑ http://fedoraproject.org/wiki/QA/Join

Test Days

Last week's Test Day[1] was on virtualization[2]. This was mainly focused on the Fedora virtualization stack, based around KVM, libvirt, and virt-manager. A small band of hardened virtualization testers were able to expose 14 bugs, which the developers are now investigating. Thanks to everyone who came out to help with the testing.

This week is a big moment in the Test Day schedule: Graphics Test Week. There will be three Test Days focusing on the three major graphics drivers: NVIDIA Test Day on Tuesday 2010-04-13[3], ATI/AMD Test Day on Wednesday 2010-04-14[4], and Intel graphics Test Day on Thursday 2010-04-15[5]. As always, widespread graphics testing is critical to the development of these drivers. Around 75% of all bugs reported in the last Graphics Test Week have been closed (either as fixed, or as duplicates), so the information gathered isn't ignored! Testing can be done with a live image, so there's no need to have an unstable Fedora installation to do the testing, and the tests are easy to do and come with full instructions. Almost everyone has an NVIDIA, AMD/ATI or Intel graphics adapter, so please come out to help us test! The events will take place all day in the #fedora-test-day channel on Freenode IRC (if you're not sure how to use IRC, there's an instruction page[6], or you can use WebIRC[7]. If you can't make it on the day, you can still provide your results on the Wiki page before or after the event.

If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac[8].

  1. ↑ http://fedoraproject.org/wiki/Test_Day:2010-04-08_Virtualization
  2. ↑ http://fedoraproject.org/wiki/Virtualization
  3. ↑ http://fedoraproject.org/wiki/Test_Day:2010-04-13_Nouveau
  4. ↑ http://fedoraproject.org/wiki/Test_Day:2010-04-14_Radeon
  5. ↑ http://fedoraproject.org/wiki/Test_Day:2010-04-15_Intel
  6. ↑ http://fedoraproject.org/wiki/How_to_use_IRC
  7. ↑ http://webchat.freenode.net/?channels=fedora-test-day
  8. ↑ http://fedorahosted.org/fedora-qa/

Fedora 13 testing

This week saw the group wrap up Fedora 13 Beta validation testing. After the previous week's delay, the fourth[1] and fifth[2] release candidate builds for the Beta arrived during the week. Installation[3] and desktop[4] validation testing for the RC4 build were both broadly successful, but Adam Williamson realized that the build included a critical bug which would cause systems containing a certain common network adapter to be unable to boot[5], so the RC5 build provided an updated kernel to fix that issue. Adam posted a call for testing of the updated kernel[6] which drew an overwhelming response, with dozens of group members confirming the kernel worked on their systems. The group re-ran the validation tests[7], and subsequently agreed with the development and release engineering groups at the go/no-go meeting[8] that the RC5 build met all the release criteria[9] and so was suitable for release as Fedora 13 Beta. Rui He summarized the validation test results[10] and encouraged more group members to be involved in the validation testing for future releases.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089919.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089962.html
  3. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC4_Install
  4. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC4_Desktop
  5. ↑ http://bugzilla.redhat.com/show_bug.cgi?id=577463
  6. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089944.html
  7. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC5_Install
  8. ↑ http://meetbot.fedoraproject.org/fedora-meeting/2010-04-08/f-13-beta-eng-readiness.2010-04-08-00.01.html
  9. ↑ http://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
  10. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090015.html

Testing non-English keyboard layouts

Petri Laine reported[1] that he had experienced problems using a non-default keyboard mapping in Fedora 13 Beta RC5. Adam Williamson replied[2] that similar bugs had occurred during previous release periods, and then announced[3] that he had extended an installation validation test case[4] and created a desktop validation test case[5] to try to ensure that similar issues are caught in future testing rounds. Petri appended his report to an existing bug report[6] and followed up on the problem there.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089995.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089998.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090001.html
  4. ↑ http://fedoraproject.org/wiki/QA:Testcase_Anaconda_autopart_%28encrypted%29_install
  5. ↑ http://fedoraproject.org/wiki/QA:Testcase_desktop_login
  6. ↑ http://bugzilla.redhat.com/show_bug.cgi?id=571900

Ensuring packages are signed

James Laska proposed[1] a new release criterion and validation test to ensure that all packages are signed with a valid Feora GPG signature. Bill Nottingham pointed out[2] that this would not slot easily into the existing package release workflow. He also noted that Bodhi is supposed to reject un-signed packages. Jesse Keating explained[3] that this was a mash configuration option which had been disabled intentionally for initial Branched composes as some packages were known not to be signed at that time. Bill ultimately suggested[4] that the signature check should be re-enabled in the relevant mash configurations.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090025.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090026.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090028.html
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/090032.html

Bugzappers screencasts

At the weekly Bugzappers meeting[1], Adam Williamson noted that he had not yet forwarded Shakthi Kannan's suggestion of making Bugzapping screencasts to the mailing list. The next day, he did so[2]. Eric Lake, Chris Campbell and James Gledhill all posted in support of the idea, but no-one yet had the combination of free time and expertise to make the screencasts.

  1. ↑ http://meetbot.fedoraproject.org/fedora-meeting/2010-04-06/fedora-meeting.2010-04-06-15.00.log.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-April/089972.html

AutoQA

With Fedora 13 validation testing winding down, work on AutoQA was picking up steam again, with the team working on dependency checking[1], tests[2] to implement the Package Sanity Test Plan[3], the results database idea[4] and the automated installation test plan[5].

  1. ↑ http://fedorahosted.org/autoqa/milestone/autoqa depcheck
  2. ↑ http://fedorahosted.org/autoqa/milestone/Package%20Sanity%20Tests
  3. ↑ http://fedoraproject.org/wiki/QA:Package_Sanity_Test_Plan
  4. ↑ http://fedorahosted.org/autoqa/milestone/Resultdb
  5. ↑ http://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan
Retrieved from "https://fedoraproject.org/w/index.php?title=FWN/Beats/QualityAssurance&oldid=165743"