From Fedora Project Wiki

< FWN‎ | Beats

(create fwn 237 qa beat)
(create fwn 240 qa beat)
Line 8: Line 8:
<references/>
<references/>


=== Localization testing ===
=== Test Days ===


Discussion on the topic of localization testing continued, with [[User:Rhe|Rui He]] proposing to run a Test Day during the Fedora 14 cycle for localization testing, particularly of keyboard layouts<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092261.html</ref>. [[User:Igor|Igor Pires Soares]] thought this was a great idea and proposed some dates<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092274.html</ref>. Eventually, Rui and Igor agreed on 2010-09-09 as the date, and added the event to the schedule<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092304.html</ref>.
This week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-08-26_OpenSCAP</ref> on 2010-08-26 will be on OpenSCAP<ref>http://fedoraproject.org/wiki/Features/OpenSCAP</ref>, an open implementation of SCAP, which aims to provide a standardized approach to maintaining the security of systems. This Test Day is most likely to be of interest to those who already have some knowledge of SCAP and are interested in an open source implementation within the Fedora project. As always, the Test Day will run all day in the #fedora-test-day IRC channel.


<references/>
Next week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-09-02_Preupgrade</ref> on 2010-09-02 will be on preupgrade<ref>http://fedoraproject.org/wiki/Preupgrade</ref>, the Fedora in-place upgrade system. This is always an important test day as we attempt to ensure the upgrade mechanisms for the next release work as expected. As upgrading is a complex process and highly dependent on the installed system, it would help to have as many testers as possible to help track down any bugs we can find. As always, the Test Day will run all day in the #fedora-test-day IRC channel. You will need to have an installed Fedora 13 system you don't mind hurting in order to help out with the testing - but remember, testing in a virtual machine is easy and non-destructive!
 
=== Upstream bug reporting ===
 
Discussion also continued on the topic of providing instructions for reporting bugs to upstream bug trackers. [[User:Ankursinha|Ankur Sinha]] proposed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092266.html</ref> a rough draft<ref>http://fedoraproject.org/wiki/User:Ankursinha/Reporting_Bugs</ref>. Jonathan Kamens felt it encouraged users to report bugs upstream instead of reporting them to Fedora, which he thought was a bad idea<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092269.html</ref>. [[User:Adamwill|Adam Williamson]] suggested focusing solely on the actual steps for reporting bugs upstream, rather than providing any potential reasons why someone might want to do so<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092271.html</ref>. [[User:Johannbg|Jóhann Guðmundsson]] suggested that the instructions should be incorporated into the pages for particular components in the Wiki, rather than being a separate page of their own<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092272.html</ref>.
 
<references/>
 
=== Rawhide acceptance testing ===
 
[[User:Jlaska|James Laska]] reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092270.html</ref> on the third (and final) automated Rawhide acceptance test plan<ref>http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan</ref> run for Fedora 14, and linked to the full results<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_14_Pre-Alpha_Rawhide_Acceptance_Test_3</ref>. The run encountered only a single bug, and the test images were declared 'last known good'. However, James cautioned that the images had not included systemd or Python 2.7, which could cause significant changes for the test composes.


<references/>
If you would like to propose a main track Test Day for the Fedora 14 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.


=== SELinux use in testing ===
=== Fedora 14 Alpha testing ===


[[User:Adamwill|Adam Williamson]] reminded the group that SELinux should always be enabled when testing Fedora, if possible<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092273.html</ref>. He explained that since SELinux being enabled is the default configuration of Fedora, it is best for testers to enable SELinux so problems related to it can be caught and fixed as soon as possible.
As always, the QA team was very active in testing the latest pre-release, Fedora 14 Alpha, conducting desktop<ref>http://fedoraproject.org/wiki/QA:Desktop_validation_testing</ref> and installation<ref>http://fedoraproject.org/wiki/QA:Installation_validation_testing</ref> validation testing (with the valued help of the desktop SIGs) for each build as we worked through the test candidate<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092367.html</ref>, RC1<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092468.html</ref>, RC2<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092493.html</ref>, RC3<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092573.html</ref> and RC4<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092700.html</ref>. At the initial go/no-go meeting on 2010-08-12<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-08-12/gonogo.2010-08-12-00.00.html</ref>, the QA representative [[User:Adamwill|Adam Williamson]] had to propose a slip due to a remaining blocker issue<ref>https://bugzilla.redhat.com/show_bug.cgi?id=623129</ref>. This issue was resolved in RC4. A second potential blocker lingered, but extensive feedback from many QA group members to Adam's request for testing<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092681.html</ref> was instrumental in identifying it as not blocking the release, and at the second go/no-go meeting, QA along with the development and release engineering groups agreed to go ahead with the Alpha release<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-08-19/fedora-meeting.2010-08-19-00.00.html</ref>.


<references/>
<references/>


=== Proven tester special testing procedures ===
=== Release criteria update ===


[[User:Adamwill|Adam Williamson]] recalled a previous proposal to have special testing procedures for the kernel for proven testers, and suggested the idea be put into place as a general concept and proposed some specific procedures for PackageKit, in light of a recent update which had broken update notification for some users<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092292.html</ref>. The general response to the proposal was positive. Bob Lightfoot suggested integration with fedora-easy-karma, so that it would display the special testing procedure, or a link to it, when requesting feedback on a package for which a special procedure existed<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092313.html</ref>. Adam thought this was a good idea and promised to look into it<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092316.html</ref>.
The blocker review process for Fedora 14 Alpha made it clear that there were several areas where the coverage of the release criteria<ref>http://fedoraproject.org/wiki/Fedora_Release_Criteria</ref> was incomplete, so [[User:Adamwill|Adam Williamson]] proposed several new criteria in two mailing list threads<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092615.html</ref> <ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092617.html</ref>, to cover booting to a console and the 'basic graphics mode' of the installer. The proposals were generally positively received, so Adam went ahead and added them to the criteria<ref>http://lists.fedoraproject.org/pipermail/test/2010-August/092784.html</ref>.


<references/>
<references/>


=== Automated bug checking tools release criteria ===
=== Learning lessons from updates ===


[[User:Adamwill|Adam Williamson]] proposed adding release criteria to cover the functionality of default automated bug checking tools (such as abrt and sealert)<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092322.html</ref>. [[User:Jdulaney|John Dulaney]] wondered why making such tools submit reports at an early stage was a high priority. Adam clarified<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092326.html</ref> that the tools should usually always be able to submit reports, and when they cannot, it's an unexpected bug: no regular groundwork is required for the tools to be able to submit reports for new releases.
[[User:Kevin|Kevin Fenzi]] created a page<ref>http://fedoraproject.org/wiki/Updates_Lessons</ref> to track problems caused by updates to stable Fedora releases, and brought it up for discussion during the weekly QA meeting of 2010-08-16<ref>http://fedoraproject.org/wiki/QA/Meetings/20100816</ref>. [[User:Jlaska|James Laska]] suggested that issues tracked there should be discussed at the FESCo level, and any issues of concern for QA passed back down to the QA group by FESCo. James also volunteered to adjust the wiki page to track recommendations and results for each issue.


<references/>
<references/>


=== Proven testers working on Fedora 14 ===
=== AutoQA ===


[[User:Adamwill|Adam Williamson]] announced that, with the branching of Fedora 14, proven testers are now needed to test and approve critical path package updates for this release<ref>http://lists.fedoraproject.org/pipermail/test/2010-July/092324.html</ref>, just as with the stable Fedora 13 release. He explained that the procedure for proven testers to follow was just the same as that for the existing stable release.
Vojtěch Aschenbrenner created a test case<ref>https://fedorahosted.org/pipermail/autoqa-devel/2010-August/000967.html</ref> for checking if a new package for a given release would be considered a 'newer' package than the newest version of the same package available in the repositories for later releases, a situation which causes problems with upgrades. [[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] continued to work on the complex dependency check tests, and Will recently posted a summary of the current status and next steps<ref>http://fedorahosted.org/pipermail/autoqa-devel/2010-August/001010.html</ref>.


<references/>
<references/>

Revision as of 00:18, 26 August 2010

QualityAssurance

In this section, we cover the activities of the QA team[1]. For more information on the work of the QA team and how you can get involved, see the Joining page[2].

Contributing Writer: Adam Williamson

  1. ↑ http://fedoraproject.org/wiki/QA
  2. ↑ http://fedoraproject.org/wiki/QA/Join

Test Days

This week's Test Day[1] on 2010-08-26 will be on OpenSCAP[2], an open implementation of SCAP, which aims to provide a standardized approach to maintaining the security of systems. This Test Day is most likely to be of interest to those who already have some knowledge of SCAP and are interested in an open source implementation within the Fedora project. As always, the Test Day will run all day in the #fedora-test-day IRC channel.

Next week's Test Day[3] on 2010-09-02 will be on preupgrade[4], the Fedora in-place upgrade system. This is always an important test day as we attempt to ensure the upgrade mechanisms for the next release work as expected. As upgrading is a complex process and highly dependent on the installed system, it would help to have as many testers as possible to help track down any bugs we can find. As always, the Test Day will run all day in the #fedora-test-day IRC channel. You will need to have an installed Fedora 13 system you don't mind hurting in order to help out with the testing - but remember, testing in a virtual machine is easy and non-destructive!

If you would like to propose a main track Test Day for the Fedora 14 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac[5].

Fedora 14 Alpha testing

As always, the QA team was very active in testing the latest pre-release, Fedora 14 Alpha, conducting desktop[6] and installation[7] validation testing (with the valued help of the desktop SIGs) for each build as we worked through the test candidate[8], RC1[9], RC2[10], RC3[11] and RC4[12]. At the initial go/no-go meeting on 2010-08-12[13], the QA representative Adam Williamson had to propose a slip due to a remaining blocker issue[14]. This issue was resolved in RC4. A second potential blocker lingered, but extensive feedback from many QA group members to Adam's request for testing[15] was instrumental in identifying it as not blocking the release, and at the second go/no-go meeting, QA along with the development and release engineering groups agreed to go ahead with the Alpha release[16].

  1. ↑ http://fedoraproject.org/wiki/Test_Day:2010-08-26_OpenSCAP
  2. ↑ http://fedoraproject.org/wiki/Features/OpenSCAP
  3. ↑ http://fedoraproject.org/wiki/Test_Day:2010-09-02_Preupgrade
  4. ↑ http://fedoraproject.org/wiki/Preupgrade
  5. ↑ http://fedorahosted.org/fedora-qa/
  6. ↑ http://fedoraproject.org/wiki/QA:Desktop_validation_testing
  7. ↑ http://fedoraproject.org/wiki/QA:Installation_validation_testing
  8. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092367.html
  9. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092468.html
  10. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092493.html
  11. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092573.html
  12. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092700.html
  13. ↑ http://meetbot.fedoraproject.org/fedora-meeting/2010-08-12/gonogo.2010-08-12-00.00.html
  14. ↑ https://bugzilla.redhat.com/show_bug.cgi?id=623129
  15. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092681.html
  16. ↑ http://meetbot.fedoraproject.org/fedora-meeting/2010-08-19/fedora-meeting.2010-08-19-00.00.html

Release criteria update

The blocker review process for Fedora 14 Alpha made it clear that there were several areas where the coverage of the release criteria[1] was incomplete, so Adam Williamson proposed several new criteria in two mailing list threads[2] [3], to cover booting to a console and the 'basic graphics mode' of the installer. The proposals were generally positively received, so Adam went ahead and added them to the criteria[4].

  1. ↑ http://fedoraproject.org/wiki/Fedora_Release_Criteria
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092615.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092617.html
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-August/092784.html

Learning lessons from updates

Kevin Fenzi created a page[1] to track problems caused by updates to stable Fedora releases, and brought it up for discussion during the weekly QA meeting of 2010-08-16[2]. James Laska suggested that issues tracked there should be discussed at the FESCo level, and any issues of concern for QA passed back down to the QA group by FESCo. James also volunteered to adjust the wiki page to track recommendations and results for each issue.

  1. ↑ http://fedoraproject.org/wiki/Updates_Lessons
  2. ↑ http://fedoraproject.org/wiki/QA/Meetings/20100816

AutoQA

Vojtěch Aschenbrenner created a test case[1] for checking if a new package for a given release would be considered a 'newer' package than the newest version of the same package available in the repositories for later releases, a situation which causes problems with upgrades. Will Woods and Kamil Paral continued to work on the complex dependency check tests, and Will recently posted a summary of the current status and next steps[2].

  1. ↑ https://fedorahosted.org/pipermail/autoqa-devel/2010-August/000967.html
  2. ↑ http://fedorahosted.org/pipermail/autoqa-devel/2010-August/001010.html
Retrieved from "https://fedoraproject.org/w/index.php?title=FWN/Beats/QualityAssurance&oldid=194253"