FWN/Beats/QualityAssurance

QualityAssurance

In this section, we cover the activities of the QA team.

Contributing Writer: Adam Williamson

Test Days

There was no Test Day last week. Next week's main track Test Day will be on NetworkManager. NetworkManager will see some significant changes and improvements for Fedora 12, so we want to test all its capabilities. Testing will include regular and wireless ethernet connections, IPv6, cellular data modems, and cellphone tethering both via USB and Bluetooth. A testing live image will be available on the Test Day page prior to the event. Just about everyone needs to network with something, so please come along and make sure your networking needs will be met in Fedora 12! The Test Day will be held on 2009-08-13 in IRC #fedora-qa.

The Fit and Finish Test Day track will be holding its third event, on peripherals. This is a broad-based day where we want to look at experiences involving any kind of peripheral - an external device you can attach to your computer (via a cable or even Bluetooth). Some test cases for the more common types of peripheral are already up on the page, and live images will be available before the Test Day. Please come along to contribute your experiences and suggestion improvements on how Fedora deals with peripherals! The Test Day will be held on 2009-08-11 in IRC #fedora-fit-and-finish (note this is not the same channel where main track Test Days take place).

If you would like to propose a main track Test Day for the Fedora 12 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac.

Weekly meetings

The QA group weekly meeting was held on 2009-08-03. The full log is available. James Laska reported that he had filed a bug on the pungi problems discussed at the previous meeting which were preventing Rawhide install image composes from working. Jesse Keating had already helped to fix the bug.

John Poelstra had, as promised, sent an announcement of the previous Friday's Alpha blocker bug review meeting. Adam Williamson had not yet posted a recap of the blocker meeting (though this was subsequently posted after the QA meeting).

James Laska had contacted Warren Togami to ask for an entry in the Rawhide Watch blog about updating Rawhide installations past the change to XZ payloads, and Warren had added an entry.

Kamil Paral had updated the instructions for composing Test Day live CDs to correctly produce Rawhide, rather than Fedora 11, images.

Jóhann Guðmundsson provided an update on the Fedora 12 Dracut feature. Harald Hoyer is working together with the kernel and anaconda teams to switch to using Dracut by default to generate initramfs images in Rawhide. They expect it to already work fully and transparently for typical situations, and expect testing (including the scheduled Test Day) to focus on complex and unusual situations, like the root partition being on a network drive, or complex RAID/LVM setups.

The group discussed the state of Rawhide in regards to the Alpha test compose that had been delayed from the previous week. It had still not been possible to create a test compose due to various bugs completely breaking the install process. It seemed that all the complete blocker bugs were cleared, so Jesse Keating said he would attempt to build a test compose later in the day. James Laska noted the compose would therefore be four days behind schedule, but thought this did not inevitably mean the release of the Alpha must be delayed, if testing could be completed quickly.

James Laska proposed that a fourth Alpha blocker bug review meeting be scheduled for 2009-08-07, and the Alpha go/no-go meeting scheduled for 2009-08-10 should be a quick decision rather than a full review meeting. John Poelstra agreed. Jesse Keating felt it was inefficient to hold similar meetings on two successive work days, but did not mind if that path was chosen and would show up for both meetings. The group agreed to follow this plan.

Will Woods reported on the progress of the AutoQA project. He has now completed automating all the installation tests in the Rawhide Acceptance Test Plan, which means there is now a script capable of performing a fully automated installation up to the stage of disk partitioning, declaring a successful test if disks are found and partitioning completed, or a failed test if not. It also stores all relevant logs as the installation takes place. James Laska provided a link to an example report matrix graphic which could be one format used to report results. Will said the next steps would be to make the test run automatically each time a new Rawhide compose was released, and to have organized reporting of the test results. Then tests for the post-installation functionality test cases would be needed, and a front-end for manual results submission for functionality test cases which cannot be automated. Will also noted that eventually the system might be adapted to run on bare metal rather than in a virtual environment, which would allow for testing on the PPC architecture. Will has also started work on Wiki documentation for writing autotest tests.

The Bugzappers group weekly meeting was held on 2009-08-04. The full log is available. The group closed out discussion on the critical path component list-based expansion of the priority triage packages list by noting that it had been succesfully merged into the main priority triage package list, and thanking Niels Haase for his work.

Adam Williamson gave an update on the status of the kernel triage project. Richard June had contacted the kernel maintainer responsible for wireless networking, John Linville, to notify him of the project and ask for any advice or requests he had, but had not yet received a reply. Adam said he would contact John himself if a reply was not received soon.

Brennan Ashton gave an update on the triage metrics project. It had turned out that a bug in FAS was causing some of the problems with his attempted migration to the version of the triage metrics script with FAS integration. He was working with the Infrastructure team to resolve this bug.

The group discussed again the 'Bugzilla Semantics' proposal. Adam Williamson had sent a mail to fedora-test-list formally proposing moving to a keyword rather than a status to mark bugs as triaged going forward, but only one day prior to the meeting. He felt it would be best to leave more time for list feedback before making a final decision. Brennan Ashton questioned how many development groups were actively desiring the change. Chuck Ebbert avowed that the kernel team would be happy if the change was made, and Seth Vidal already asks for the proposed new policy to be used for yum bugs as a special exception to current practice. Matej Cepl said the KDE and LVM teams would prefer the proposed new system, and Jesse Keating said the Desktop team would also.

The next QA weekly meeting will be held on 2009-08-10 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-08-11 at 1500 UTC in #fedora-meeting.

Alpha test compose availability

Liam Li announced the availability of the first test compose for Fedora 12 Alpha. He asked group members to help validate the image, especially as far as installation testing went, following the test matrix. Testing was carried out by several group members, and identified some significant bugs which were added to the Alpha blocker list, as per a status roundup by James Laska. In response to a question from Todd, James Laska clarified that testing did not need to be performed with the test compose images; testing network installation direct from the Rawhide repository was also useful (and necessary, for some test cases).

Xfce spin testing

Adam Miller announced the next Xfce test live image, for Xfce enthusiasts to test and report Xfce-specific issues.

Test Day live image creation

Kamil Paral announced that he had improved the Test Day live image creation guide to explain workarounds and fixes for some problems related to SELinux that had been discussed on the list.

Fedora 12 Alpha blocker bug review meetings

Adam Williamson posted a recap of the blocker bug review meeting held on 2009-07-31, including details of the bugs discussed and the actions (or lack of action) decided upon for each. James Laska announced the following blocker bug review meeting, to be held on 2009-08-07.