FWN/Beats/QualityAssurance

QualityAssurance

In this section, we cover the activities of the QA team.

Contributing Writer: Adam Williamson

Test Days

Last week's main track Test Day was on Dracut, the new initrd generation tool. There was a solid turnout of testers and developers. Many cases were tested to work without problems, but some problem cases were identified, and bugs were filed.

Next week's main track Test Day will be on Sugar on a Stick, the Fedora-derived USB stick distribution which features the Sugar desktop environment that is the default desktop for the OLPC project. This Test Day is being led by the Sugar developers. If you're interested in this exciting and innovative desktop environment, please come along and help test it! The testing will be on Sugar on a Stick v2 Beta, which should be available in time for the Test Day. The Test Day will be held on Thursday 2009-09-03 in IRC #fedora-test-day.

Next week's Fit and Finish project Test Day will be on Sectool, the security audit and intrusion detection tool. The Fit and Finish team are working throughout the Fedora 12 cycle to file the rough edges off Fedora's desktop experience, so please come along and help them test! The Test Day will be held on Tuesday 2009-09-01 in IRC #fedora-test-day.

If you would like to propose a main track Test Day for the Fedora 12 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac.

Weekly meetings

The QA group weekly meeting was held on 2009-08-24. The full log is available. James Laska led a post-Alpha release recap (on the assumption the Fedora 12 Alpha would in fact be released on time). The group agreed that the process had been handled quite well. Jesse Keating was happy with the level of communication between release engineering and QA. James felt the blocker bug review meetings had gone smoothly and been a positive contribution. Adam Williamson thought the Alpha process had flagged up the need for a better process for filtering Anaconda updates into Fedora. James summarized areas of possible improvement: he felt planned testing could be extended to areas beyond installation. The group agreed, but generally felt that installation was the most important area by a significant margin. James committed to trying to extend the test plan to cover X.org testing for the Fedora 12 Beta release. Will Woods pointed out that basic X functionality was part of the Rawhide acceptance test plan, and suggested that the Rawhide acceptances tests should be considered a prerequisite to the installation testing.

Will Woods reported on the progress of the AutoQA project. He noted that the automated tests were still running and sending results to the mailing list. He had fixed bugs in several of the tests, and improved the subject lines of the result emails. He was still engaged in tracing other bugs in the existing tests, and writing documentation for creating tests and hooks. Jesse Keating pointed out that a new upstream release of autotest was available, and committed to getting it packaged and made available through the infrastructure team for testing. Adam Williamson asked whether the current state of the project was sufficient for the planned israwhidebroken.com website to be created. Will explained that some bits were still missing, particularly a method for getting data from autotest into the page.

David Pravec proposed creating a fedora-test-announce mailing list for those who wanted to be informed of events such as Test Days, but did not want to follow the traffic of fedora-test-list. Adam Williamson suggested using the list to announce test composes and changes to release schedules. Jesse Keating worried about the principle of creating more and more mailing lists, and suggested posting announcements to fedora-devel-announce instead, but James Laska said he had been asked to stop posting Test Day announcements to that list in the past. In the end the group agreed on the proposal, and David took responsibility for creating the list.

James Laska asked for an update on Test Day status. Adam Williamson reported that the Fit and Finish team's Printing Test Day had gone smoothly, from what he had seen. James linked to his report on the ABRT Test Day, and thanked David Pravec and Kamil Paral for organizing the event. James also reported on the readiness of the upcoming Dracut Test Day.

Adam Williamson raised the topic of the recently-introduced nightly live builds of Rawhide, and asked the group to support him in publicising their existence. Jesse Keating worried that the limited resources of the server on which they are hosted would be put under serious strain if they become too widely used. This led to another discussion of the best way to distribute regularly updated large images to a mass user base. As usual, no definite answers were discovered. Kevin Fenzi wondered if DeltaISOs would help, but Jesse explained they would not, due to the contents of a live image as compared to an installation image (live images essentially contain one large file that is an image of an entire filesystem, while installation images contain individual package files, and hence are much more amenable to having their size reduced by DeltaISOs).

David Pravec wanted to improve on the reporting of results of Test Days. He felt that having a results table which was essentially a set of Bugzilla links at the bottom of each Test Day page was unnecessary repetition of work. Adam Williamson pointed out that the results tables for some Test Days contained significantly more information than simply links to bug reports. David's suggestion was to automate the linking of Bugzilla reports to the Test Day Wiki pages in some way. Adam felt this might be theoretically possible, but technically difficult without undesirable significant modifications to Bugzilla. James Laska noted that reporting results to the Wiki pages was only ever intended to be an interim solution, and the group was still officially committed to implementing a proper test case management system, which should render the problem irrelevant. In the meantime, James and Adam were both happy to accept any improvements anyone could propose for the Wiki-based system. David promised to work on providing a practical proposal.

The Bugzappers group weekly meeting was held on 2009-08-25. The full log is available. Adam Williamson gave an update on the proposal to add the semantics switchover to the QA team calendar. He noted that the public Google calendar the QA team had run for a short time was now mostly unused and had only been intended as a test. He further noted that the Infrastructure group was still working on providing a project-wide calendaring solution. Niels Haase clarified that he had in mind the short lists of tasks and dates related to specific groups that are published by the release engineering team. Adam said he could have the switchover added to these Fedora 13 schedules once they were created.

Richard June gave an update on the kernel triage project. He had started on his work of triaging wireless related bugs. So far he had found that most reports were either very old, or were valid reports which already included all necessary information and hence did not need to be triaged. Adam Williamson suggested that he continue on wireless bugs for a while, and if the same pattern persisted, try a different kernel component instead. If several kernel components all seemed to be in the same state, the value of continuing with the kernel triage project could be re-evaluated.

Edward Kirk said that he was working on an SOP (standard operating procedure) detailing all aspects of arranging the Bugzappers group meetings, and asked the group if it had particular ideas or suggestions about any part of the process. In general everyone agreed the current process was good and was happy that Edward was working on officially documenting it. Edward promised to submit a draft of the SOP to the mailing list or a future meeting for review.

Edward Kirk suggested having meetbot announce Bugzappers meetings in related channels shortly ahead of the meeting. Kevin Fenzi and Adam Williamson worried that this might annoy people, and also considered the dystopian possibilities of a world where all projects announced all their meetings in all relevant channels. The proposal was not taken further.

The next QA weekly meeting will be held on 2009-08-31 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-09-01 at 1500 UTC in #fedora-meeting.

ABRT Test Day report

David Pravec and Kamil Paral reported on the ABRT Test Day held on 2009-08-20, with a list of all bugs reported during the Test Day and their current statuses. They were happy with the success of the Test Day.