From Fedora Project Wiki

< FWN‎ | Beats

(create fwn 218 qa beat)
(create fwn 219 qa beat)
Line 10: Line 10:
=== Test Days ===
=== Test Days ===


Last week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-03-18_Palimpsest</ref> was on Fedora 13 changes to disk management, via the udisks (previously DeviceKit-disks) backend and the Palimpsest front end<ref>http://fedoraproject.org/wiki/Features/UdisksImprovements</ref>. The turnout was solid and resulted in 12 bugs being filed, of which two have already been fixed. [[User:Davidz|David Zeuthen]] was there to help with testing throughout the test day and will be working to fix the remaining bugs.
Last week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing</ref> was on printing, including the implementation of automatic print driver installation<ref>http://fedoraproject.org/wiki/Features/AutomaticPrintDriverInstallation</ref>in Fedora 13. There was a disappointingly low turnout, despite [[User:twaugh|Tim Waugh's]] extensive efforts to organize and promote the event. We theorize that printing works so well for most people that they didn't think it was necessary to turn up! Nevertheless, thanks to those who did come out to test, and reported five bugs for Tim to work on.


Next week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing</ref> will be on printing, including the implementation of automatic print driver installation<ref>http://fedoraproject.org/wiki/Features/AutomaticPrintDriverInstallation</ref>in Fedora 13. This is a nice mix of making sure the existing printer support is working well and testing out a shiny new feature. Printing is important to nearly everyone, so if you've got a printer, please come along and help test! As usual, you can test with an installed Fedora 13 or Rawhide system, or a live image which is available on the Test Day page. [[User:twaugh|Tim Waugh]] has been working hard to arrange the test day, and he and Jiri Popelka, along with QA's [[User:Ykopkova|Yulia Kopkova]], will be on hand to help with any questions or problems. The Test Day will run all day on Thursday 2010-03-25 in the #fedora-test-day IRC channel.
This week sees two Test Days. The first<ref>https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault</ref>, on Tuesday 2010-03-30, will have passed by the time you read this; it will have been on the implementation of SSSD by default<ref>http://fedoraproject.org/wiki/Features/SSSDByDefault</ref>. This feature is very useful to those who use accounts on a remote server which may not always be accessible from their system. We'll bring you a report on this event next week.
 
The second<ref>https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT</ref> will be on ABRT<ref>http://fedoraproject.org/wiki/Features/ABRT</ref>, the automated bug report tool which has been included with Fedora by default since the release of Fedora 12. We'll be testing Fedora 13 enhancements to ABRT and making sure the system is working correctly for the upcoming Fedora 13 release. ABRT is important to all Fedora users and developers, so if you have a few minutes, please come along and help test! As usual, you can test with an installed Fedora 13 or Rawhide system, or a live image which is available on the Test Day page. [[User:Zprikryl|Zdenek Prikryl]], [[User:Jmoskovc|Jiri Moskovc]] and [[User:Adamwill|Adam Williamson]] will be on hand during the event, which will run all day on Thursday 2010-04-01 in the #fedora-test-day IRC channel.


If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
Line 20: Line 22:
=== Fedora 13 testing ===
=== Fedora 13 testing ===


The first acceptance test plan<ref>http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan</ref> run of the Beta period was performed on the Fedora 13 tree of 2010-03-15<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Pre-Beta_Acceptance_Test_1</ref>, resulting in a pass, with two incidental issues noted and reported.
The first test compose for Fedora 13 Beta was announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089526.html</ref> by [[User:Rhe|Rui He]] on 2010-03-23. The group helped to fill out the planned installation<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_TC1_Install</ref> and desktop<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_TC1_Desktop</ref> test matrices. [[User:Rhe|Rui He]] provided a summary<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089681.html</ref> of the TC test results. Shortly after the test compose, the first release candidate build followed<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089680.html</ref> on 2010-03-26. Andre Robatino provided delta ISOs between TC1 and RC1<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089684.html</ref>. Again, the group quickly filled out the desktop<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC1_Desktop</ref> and install<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC1_Install</ref> matrices. The testers found several blocker issues.
 
During the weekly QA meeting<ref>http://fedoraproject.org/wiki/QA/Meetings/20100315</ref>, [[User:Jlaska|James Laska]] noted that release engineering should be working on the first Beta test compose for 2010-03-18, but it did not arrive during the week. [[User:Rhe|Rui He]] announced<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089340.html</ref> the installation and desktop acceptance testing matrices for the candidate in preparation for its expected arrival.
 
The second blocker bug review meeting for Fedora 13 Beta was held on 2010-03-19<ref>http://meetbot.fedoraproject.org/fedora-bugzappers/2010-03-19/f13beta-blocker-review.2010-03-19-16.04.html</ref>. All outstanding Beta blocker bugs were reviewed, and developers were consulted on the remaining open bugs to ensure fixes should be available in time for the release candidate process to begin the following week.


Robert P.J. Day<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089175.html</ref> and Felix Miata<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089297.html</ref> noted issues with the package customization choices in the Fedora 13 installation process. Robert noticed that the package group customization screen only allowed one major package group to be chosen, leading to a discussion on whether this was a sensible decision or whether it should be possible to select multiple high-level groups. [[User:dcantrel|David Cantrell]] explained<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089213.html</ref> that the Anaconda team had noticed that, in previous releases where multiple selections were allowed, "most users would either (a) leave the default
The third blocker bug review meeting for Fedora 13 Beta was held on 2010-03-26<ref>http://meetbot.fedoraproject.org/fedora-bugzappers/2010-03-26/f-13-beta-blocker-review.2010-03-26-16.05.html</ref>. All outstanding Beta blocker bugs were reviewed, and developers were consulted on the remaining open bugs to ensure fixes should be available in time for the release candidate process to begin the following week.
choices in place and continue or (b) check them all for fear of missing something". However, he was still open to the possibility of changing to multiple selection after "look[ing] at the tasks as
defined now [to] see if they should be changed, expanded, or reduced".


Felix noticed that the minimal installation option resulted in a non-working network connection on first boot. [[User:jkeating|Jesse Keating]] explained<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089307.html</ref> that this is because the minimal install does not include NetworkManager, and the installer does not have code to detect when this is the case and enable the traditional-style ''network'' service instead. Some debate ensued over whether this feature should be implemented, and about whether minimal installations had had working network connections or not on earlier Fedora releases.
'John H' reported<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089584.html</ref> that the test candidate build failed to install correctly to an Intel X25-M SSD. [[User:Jlaska|James Laska]] suggested<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089595.html</ref> he file a bug report on the issue.


Several group members, including Jim Haynes<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089210.html</ref>, Tom Horsley<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089223.html</ref> and Robert Lightfoot<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089385.html</ref>, were engaged in testing Fedora 13 images and gave some valuable feedback on the issues they encountered.
[[User:Kparal|Kamil Paral]], Joachim Backes<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089715.html</ref> and others found a bug<ref>https://bugzilla.redhat.com/show_bug.cgi?id=577789</ref> which prevented the boot process from completing successfully with the Plymouth graphical boot system enabled. The bug was later tracked down and resolved in a build which will be brought into the second Beta release candidate.


<references/>
<references/>
Line 38: Line 34:
=== Update acceptance testing ===
=== Update acceptance testing ===


During the weekly QA meeting, [[User:Jlaska|James Laska]] noted that the group needed to make progress on defining a process for update acceptance testing. Packages intended for Fedora 13 were already being held in the updates-testing repository for testing, and FESCo had agreed to a policy requiring proposed updates for critical path packages in stable releases to be held for testing as well. The QA group had the responsibility to decide how the group of trusted testers whose feedback would be used to approve updates should be defined. [[User:maxamillion|Adam Miller]] had already provided an initial draft policy<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/088980.html</ref>, but this was waiting on feedback from the rest of the group.
During the weekly QA meeting<ref>http://fedoraproject.org/wiki/QA/Meetings/20100322</ref>, the group discussed the status of the various proposed changes and policies regarding updates. [[User:Adamwill|Adam Williamson]] summarized that "we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given". [[User:maxamillion|Adam Miller]] provided<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089764.html</ref> a second draft of the proventesters SOP proposal, and [[User:Adamwill|Adam Williamson]] posted a discussion of testing procedure<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089690.html</ref> which posited that a policy for testing updates was impossible within the current Bodhi system, and the best way forward would be to revise the way Bodhi works. [[User:Bochecha|Mathieu Bridon]] pointed out<ref>http://lists.fedoraproject.org/pipermail/devel/2010-March/134089.html</ref> that Williamson's proposal was very similar to Doug Ledford's earlier proposal<ref>http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html</ref>, and explained that the infrastructure team was already working Doug's ideas into their plans for 'Bodhi 2', the next major revision of Bodhi.


<references/>
<references/>


=== Privilege escalation testing ===
=== Target bug trackers ===


During the weekly QA meeting, [[User:Adamwill|Adam Williamson]] noted that, since the privilege escalation policy<ref>http://fedoraproject.org/wiki/Privilege_escalation_policy</ref> which the group had worked on had been accepted, no progress had yet been made on implementing testing. He felt that the logical next step would be to produce the planned script to identify packages which potentially perform privilege escalation; this would give an idea of the problem space and potentially allow basic manual testing during the Fedora 13 cycle. He planned to talk to [[User:Wwoods|Will Woods]] to see if this could be implemented quickly.
Following on from discussion the previous week, [[User:Adamwill|Adam Williamson]] posted a proposal<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089575.html</ref> on the use of the Target bug trackers, suggesting either discontinuing their use or repurposing them to track bugs which did not constitute release blockers, but for which fixes would be accepted through release freezes. The thread turned into quite a wide-ranging discussion about freeze procedures and update acceptance. Later, Adam summarized by suggesting the key issue to decide is 'whether we want to have a time-defined stage' where only fixes for blockers and certain specifically chosen bugs would be accepted, and if a tracker bug is the most sensible way to keep track of those bugs<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089664.html</ref>.


<references/>
<references/>


=== New release process Wiki documentation ===
=== Bugzapping in the classroom ===


During the weekly Bugzappers meeting<ref>http://meetbot.fedoraproject.org/fedora-meeting/2010-03-16/fedora-meeting.2010-03-16-15.03.log.html</ref>, [[User:Beland|Christopher Beland]] announced that he had revised several pages on the Wiki to reflect the new, No Frozen Rawhide-based<ref>http://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal</ref> release process. He had updated the Rawhide<ref>http://fedoraproject.org/wiki/Releases/Rawhide</ref> and Fedora Release Life Cycle<ref>http://fedoraproject.org/wiki/Fedora_Release_Life_Cycle</ref> pages, and created a new Branched page<ref>http://fedoraproject.org/wiki/Releases/Branched</ref> to document the ''Branched'' release (which will always be the upcoming stable Fedora release, so is currently Fedora 13). [[User:Jlaska|James Laska]] noted that he had an older proposal<ref>http://fedoraproject.org/wiki/User:Jlaska/Draft</ref> to split the Rawhide page into several smaller pages. James, Christopher and [[User:Adamwill|Adam Williamson]] discussed whether this would be a good idea, and eventually agreed that the Rawhide and Branched pages could benefit from streamlining and possibly from splitting, but detailed proposals should be sent to the mailing list.
[[User:Vedranm|Vedran Miletić]] asked the group<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089589.html<ref> for advice and support on teaching Bugzapping in a university environment. [[User:Beland|Christopher Beland]] applauded the idea, and suggested Firefox, Evolution, Nautilus and Rhythmbox as good components for a class group to work on. [[User:Adamwill|Adam Williamson]] offered<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089604.html</ref> to provide support and assistance.


<references/>
<references/>


=== Target bug trackers ===
=== rsync for test builds ===
 
During the weekly Bugzappers meeting, [[User:Beland|Christopher Beland]] raised the question of what should be done with the Target bug trackers - F12Target, F13Target and so on. These are intended to track bugs which are not quite release blockers but are important and should receive greater developer attention in a best effort to resolve them before release. However, the group felt that they are not serving this purpose, and are mostly ignored by developers. They also noted that there was significant overlap with the ''severity'' field, since the group had defined a policy for the use of that field and begun to implement it in triaging. As no-one had ideas for reviving the usefulness of the Target trackers, the group agreed that [[User:Adamwill|Adam Williamson]] should draft a proposal to drop them, to be forward to the development group.


Later in the week, at the blocker bug review meeting, [[User:jkeating|Jesse Keating]] suggested the Target tracker could be used during tight freeze periods to track bugs which were not strictly release blockers, but which QA and release engineering would accept fixes for, through the freeze.
Andre Robatino asked<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089669.html</ref> if it would be possible to set up rsync access for the server on which test builds are stored, to make it easier to convert a DVD build into a multi-CD build for testing purposes. He then suggested<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089735.html</ref> that zsync would serve the purpose even better. [[User:Adamwill|Adam Williamson]] pointed out<ref>http://lists.fedoraproject.org/pipermail/test/2010-March/089798.html</ref> that zsync was still not packaged in Fedora due to a problem with bundled libraries.


<references/>
<references/>

Revision as of 06:27, 31 March 2010

QualityAssurance

In this section, we cover the activities of the QA team[1]. This week, we are trying out a new topic-focused layout, without the topic-by-topic weekly meeting recaps. Please let me know if you particularly like or dislike the new layout!

Contributing Writer: Adam Williamson

  1. ↑ http://fedoraproject.org/wiki/QA

Test Days

Last week's Test Day[1] was on printing, including the implementation of automatic print driver installation[2]in Fedora 13. There was a disappointingly low turnout, despite Tim Waugh's extensive efforts to organize and promote the event. We theorize that printing works so well for most people that they didn't think it was necessary to turn up! Nevertheless, thanks to those who did come out to test, and reported five bugs for Tim to work on.

This week sees two Test Days. The first[3], on Tuesday 2010-03-30, will have passed by the time you read this; it will have been on the implementation of SSSD by default[4]. This feature is very useful to those who use accounts on a remote server which may not always be accessible from their system. We'll bring you a report on this event next week.

The second[5] will be on ABRT[6], the automated bug report tool which has been included with Fedora by default since the release of Fedora 12. We'll be testing Fedora 13 enhancements to ABRT and making sure the system is working correctly for the upcoming Fedora 13 release. ABRT is important to all Fedora users and developers, so if you have a few minutes, please come along and help test! As usual, you can test with an installed Fedora 13 or Rawhide system, or a live image which is available on the Test Day page. Zdenek Prikryl, Jiri Moskovc and Adam Williamson will be on hand during the event, which will run all day on Thursday 2010-04-01 in the #fedora-test-day IRC channel.

If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac[7].

  1. ↑ http://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing
  2. ↑ http://fedoraproject.org/wiki/Features/AutomaticPrintDriverInstallation
  3. ↑ https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault
  4. ↑ http://fedoraproject.org/wiki/Features/SSSDByDefault
  5. ↑ https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT
  6. ↑ http://fedoraproject.org/wiki/Features/ABRT
  7. ↑ http://fedorahosted.org/fedora-qa/

Fedora 13 testing

The first test compose for Fedora 13 Beta was announced[1] by Rui He on 2010-03-23. The group helped to fill out the planned installation[2] and desktop[3] test matrices. Rui He provided a summary[4] of the TC test results. Shortly after the test compose, the first release candidate build followed[5] on 2010-03-26. Andre Robatino provided delta ISOs between TC1 and RC1[6]. Again, the group quickly filled out the desktop[7] and install[8] matrices. The testers found several blocker issues.

The third blocker bug review meeting for Fedora 13 Beta was held on 2010-03-26[9]. All outstanding Beta blocker bugs were reviewed, and developers were consulted on the remaining open bugs to ensure fixes should be available in time for the release candidate process to begin the following week.

'John H' reported[10] that the test candidate build failed to install correctly to an Intel X25-M SSD. James Laska suggested[11] he file a bug report on the issue.

Kamil Paral, Joachim Backes[12] and others found a bug[13] which prevented the boot process from completing successfully with the Plymouth graphical boot system enabled. The bug was later tracked down and resolved in a build which will be brought into the second Beta release candidate.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089526.html
  2. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_TC1_Install
  3. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_TC1_Desktop
  4. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089681.html
  5. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089680.html
  6. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089684.html
  7. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC1_Desktop
  8. ↑ http://fedoraproject.org/wiki/Test_Results:Fedora_13_Beta_RC1_Install
  9. ↑ http://meetbot.fedoraproject.org/fedora-bugzappers/2010-03-26/f-13-beta-blocker-review.2010-03-26-16.05.html
  10. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089584.html
  11. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089595.html
  12. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089715.html
  13. ↑ https://bugzilla.redhat.com/show_bug.cgi?id=577789

Update acceptance testing

During the weekly QA meeting[1], the group discussed the status of the various proposed changes and policies regarding updates. Adam Williamson summarized that "we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given". Adam Miller provided[2] a second draft of the proventesters SOP proposal, and Adam Williamson posted a discussion of testing procedure[3] which posited that a policy for testing updates was impossible within the current Bodhi system, and the best way forward would be to revise the way Bodhi works. Mathieu Bridon pointed out[4] that Williamson's proposal was very similar to Doug Ledford's earlier proposal[5], and explained that the infrastructure team was already working Doug's ideas into their plans for 'Bodhi 2', the next major revision of Bodhi.

  1. ↑ http://fedoraproject.org/wiki/QA/Meetings/20100322
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089764.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089690.html
  4. ↑ http://lists.fedoraproject.org/pipermail/devel/2010-March/134089.html
  5. ↑ http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html

Target bug trackers

Following on from discussion the previous week, Adam Williamson posted a proposal[1] on the use of the Target bug trackers, suggesting either discontinuing their use or repurposing them to track bugs which did not constitute release blockers, but for which fixes would be accepted through release freezes. The thread turned into quite a wide-ranging discussion about freeze procedures and update acceptance. Later, Adam summarized by suggesting the key issue to decide is 'whether we want to have a time-defined stage' where only fixes for blockers and certain specifically chosen bugs would be accepted, and if a tracker bug is the most sensible way to keep track of those bugs[2].

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089575.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089664.html

Bugzapping in the classroom

Vedran Miletić asked the groupCite error: Closing </ref> missing for <ref> tag to provide support and assistance.


rsync for test builds

Andre Robatino asked[1] if it would be possible to set up rsync access for the server on which test builds are stored, to make it easier to convert a DVD build into a multi-CD build for testing purposes. He then suggested[2] that zsync would serve the purpose even better. Adam Williamson pointed out[3] that zsync was still not packaged in Fedora due to a problem with bundled libraries.

  1. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089669.html
  2. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089735.html
  3. ↑ http://lists.fedoraproject.org/pipermail/test/2010-March/089798.html
Retrieved from "https://fedoraproject.org/w/index.php?title=FWN/Beats/QualityAssurance&oldid=162620"