FWN/Beats/QualityAssurance

QualityAssurance

In this section, we cover the activities of the QA team.

Contributing Writer: Adam Williamson

Test Days

There was no Test Day last week, and No Test Day is currently scheduled for next week. If you would like to propose a main track Test Day for the Fedora 12 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac.

Weekly meetings

The QA group weekly meeting was held on 2009-07-27. The full log is available. James Laska reported that he had contacted the infrastructure team's Mike McGrath regarding whether alt.fedoraproject.org has enough resources to make test composes and release candidate builds publicly available. Mike believes it does, so the test composes for Fedora 12 Alpha will be made publicly available as a test. They will be announced to fedora-test-list. A ticket is being used to track this.

Jóhann Guðmundsson had not yet been able to contact Harald Hoyer regarding testing and contingency plans for the Fedora 12 Dracut feature. He will update again at the next meeting.

The group discussed the state of Rawhide in regards to the Alpha test compose that was due the Wednesday following the meeting. John Poelstra noted that a bug was currently preventing install images from being generated in Rawhide's daily updates. Jesse Keating said a bug should be filed on this and added to the Alpha blocker bug list. James Laska mentioned there were known to be two bugs entirely blocking Rawhide installation from working. Kamil Paral mentioned two more bugs which were breaking installation in KVM-based virtual machines. As these are essentially QA's reference platform, the group asked Kamil to add them to the Alpha release blocker list.

James Laska noted that a Fedora 12 Alpha blocker bug review meeting was scheduled for Friday 2009-07-31. It was agreed that John Poelstra would send out an announcement of the meeting, and Adam Williamson would send out a recap after it had finished.

Will Woods reported on the progress of the AutoQA project. He has now automated seven test cases in the Rawhide Acceptance Test Plan, and is working on a script to send out test status emails. He is also working on a blog post and possibly some Wiki documentation regarding the project. David Pravec noted that the latest autotest packages had some problems. Jesse Keating would work on fixing those.

Jesse Keating alerted the group that a Rawhide mass rebuild would be starting during the week, due to the arch change from i586 to i686, new compression format for RPM payloads (XZ), new glibc, and new gcc. He asked everyone to be on the lookout for rebuild-related bugs. Will Woods noted that, after the rpm package itself was rebuilt with an XZ format payload, upgrades from Rawhide installs before the XZ change to current Rawhide would no longer work normally. The best workaround for this issue is to install the updated rpm package from Fedora 11 updates-testing, then update Rawhide as normal. The issue does not affect upgrades from Fedora 11 via yum or anaconda, only upgrades from older to newer Rawhide. James Laska volunteered to submit the issue to Warren Togami for inclusion in the Rawhide Watch blog.

Kamil Paral noted that, at the time of the meeting, the guide to creating live images for test days had a problem which would cause anyone following it to create a live CD based on Fedora 11, not Rawhide. James Laska thanked him for the report, and promised to look into the problem. (Editor's note: since the time of the meeting, this problem has been fixed, and the guide as it stands works correctly).

Jóhann Guðmundsson brought up an old request he had filed for the QA team to have a web page / blog for hosting announcements of QA-related projects, and articles on QA-related topics. The issue was tabled for further discussion as it was not clear exactly what the scope of this page should be, or whether existing Fedora Project pages already sufficiently covered the perceived need. Adam Williamson and Will Woods suggested that a QA team blog aggregator (a Planet) may be another way to achieve this goal.

The Bugzappers group weekly meeting was held on 2009-07-28. The full log is available. The group discussed the latest revision of the critical path component list-based expansion of the priority triage packages list. Since last week, Niels Haase had reduced the size of the list by removing dependencies. The group decided this was a sensible approach given the triaging resources available, and approved merging it into the main priority triage package list.

Adam Williamson gave an update on the status of the kernel triage project. Richard June had volunteered to start triaging wireless-related kernel bugs, as a test of the viability of the plan. Adam asked him to contact the kernel maintainer with responsibility for wireless, John Linville, to notify him of the project and ask for any advice or requests he had, and then to start triaging bugs. Adam would also send a follow-up mail to the group of people interested in the kernel triage project with this current status.

Brennan Ashton gave an update on the triage metrics project. His development version of the code was not functional at the time of a meeting due to problems with its database code. He had reverted the public instance of the triage system to the last stable working code, but it had only one day's data available at the time of the meeting. More data would be available shortly after. Adam Williamson asked if, when the development code was ready for production, it would be able to use the existing data, or whether the data would have to be re-generated again. Brennan said that since the development version could use more information than the current stable version, re-generating the data would be faster. Brennan also plans in future to branch the code to use TurboGears 2.0. Adam also asked if Brennan was happy to have co-maintainers on the project, to speed up the work and ensure more reliable availability of maintainers. Brennan said that this was fine, and Adam and Brennan agreed to work together to put out a call for volunteers to help work on the project.

The group discussed again the 'Bugzilla Semantics' proposal Adam Williamson had made to the mailing list. The current feeling on the mailing list and in the meeting seemed to be most in favour of the second option presented in Adam's last email on the topic. The group agreed to propose option #2 as the way forward on the mailing list, and proceed with it if no serious objections were raised.

The next QA weekly meeting will be held on 2009-08-03 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-08-04 at 1500 UTC in #fedora-meeting.

Benchmarking discussion

Adam Pribyl brought up Phoronix's benchmarking of Fedora Rawhide in comparison with OpenSUSE, Ubuntu and Mandriva, suggesting that Fedora's supposed poor performance in these benchmarks should be an issue for concern, in terms of the image of the distribution if nothing else. Jóhann Guðmundsson felt that benchmarking of a development distribution was fundamentally meaningless. He said ongoing performance monitoring of Rawhide may be useful to development, but would have to properly managed. Frank Murphy agreed with Adam that, irrespective of the quality of Phoronix's benchmarks, the fact that the site was widely read meant apparent 'poor performance' in Fedora was a problem. As a source of the apparent poor performance, several people pointed out that Rawhide has debugging code enabled that stable releases don't have, and Dave Jones went into more detail about some debugging code Fedora enables and which other distributions do not.

Test Day live image creation

Kamil Paral announced that he had fixed the Test Day live image creation guide so that images are generated from Rawhide as intended, rather than Fedora 11 as was previously the case.

Fedora 12 Alpha test compose delay

Liam Li announced that the test compose for Fedora 12 Alpha had been delayed from 2009-07-29 to 2009-08-06, due to multiple bugs entirely blocking Rawhide installation from working.