Fedora 12 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 12. The feedback will be used as a basis for identifying areas for improvement for Fedora 13 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to fedora-test-list@redhat.com.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

• jlaska - Having QA/SOP_Test_Day_management was invaluable and resulted in a consistent look'n'feel for Fedora 12 test days
• jlaska - The creation of the test-announce@lists.fedorahosted.org mailing list made it much easier to announce test events. With a single mailing list for announcements, it's harder to miss any events.
• jlaska - Consistent announcement of ISO test events from Liam was great to ensure a consistent testing experience (see User:Liam/Draft_Install_Test_SOP).
• jlaska - We had more ISO install test runs, and they were earlier in each milestone
• jlaska - F-12 ISO Install testing solicited more testing from outside the core install test group. 27 different people contributed test feedback on the wiki test matrix during F-12. . 19 different people contributed test feedback on the wiki test matrix during F-11.
• jlaska - Automated test cases provided for the Power Management test day == awesome!
• Viking-Ice - Increase in SIG and maintainers for hosting and holding their own test days and specific application testing.
• Viking-Ice - Nightly-composes of rawhide test images.
• Viking-Ice - Improvement in advertisement testing/test day's
• Viking-Ice - Increase in request from sig/maintainers in Fedora QA Trac system.
• Viking-Ice - Increase in how to debug <component> for reporters to follow and provide better bug report.
• Viking-Ice - Consistency in how to debug <component> pages for better look and feel.
• Viking-Ice - Automatic bug reporting tool develop to help provide better bug reporting and allow technical challenge people to participate report and provide feedback.
• Viking-Ice - Better collaboration and participation from developers/maintainers with QA.
• He Rui - The SOP management of both test days and install tests
• He Rui - Traceback can be saved or reported automatically
• He Rui - Common f12 bugs page
• He Rui - Tickets could be created and traced in Trac system
• Adam Williamson - test days went well again. it was nice to see how many independent test days there were.

Could have been better

• anonymous

uname -a hera.localdomain 2.6.31.12-174.2.3.fc12.x86_64 #1 SMP Mon Jan 18 19:52:07 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux language brazilian portuguese

this works for me ls -l *.rpm

look at "./" preceding the char "*" this works for me ls -l ./*.pps

but

this does not works for me

ls -l *.pps ls: invalid option -- 'P' Experimente "ls --help" para mais informações.

I think this is a bug

• jlaska - it may be, but I believe you might want to confirm what the asterix is expanding to in the directory you are running the command ls -l *.pps. It's possible it is expanding to a filename that has -P as the first 2 characters, which confuses the ls command. What does echo *.pps yield? You may wish to enclose the command in quotes, ls -l "*.pps". If problems remain, please raise the issue on the users or test list.
• jlaska - The blocker bug meetings during the Final milestone were too long.
• Kparal - The preupgrade could receive more testing/test cases
• Kparal - test-announce wasn't used for announcing RC candidates IIRC, we should make sure all important events are mentioned there
• He Rui - Post-event review python could be used for install test like in test day
• He Rui - Transfer from manual-installation to Auto-installation test
• He Rui - Test days/Install test events could be put up obviously on wiki so that people who wanna contribute can find it easily from wiki except from mail lists.
• He Rui - 'Edit' on wiki could be more automatically by filling in and submitting a form without wiki syntax.
• Adam Williamson - in the end, we focused heavily on just three component areas for testing: anaconda, kernel, and X.org. This is primarily a function of the fact that these are the most vital bits; it feels like we're still at the point of doing let's make sure it's not totally broken testing than let's make sure it's really good testing. We didn't do stuff like making sure the desktop was polished.
• Adam Williamson - there was clearly a lot of uncertainty about RAID issues; it's something we obviously don't as a team test well enough (and some of us personally don't understand enough :>). In the end there didn't turn out to be any horrible issues, but the confusion was evident, and we did miss the Intel BIOS RAID stuff-ups. For F13 we should have better RAID testing both in Test Days and in pre-release test cycles.
• Adam Williamson - We weren't completely on top of X.org bugs for this release. The ones that wound up getting promoted to release blocker level were kind of an arbitrary selection. I think we got nouveau mostly right as I had a reasonable grip on nouveau triage, but we just had too few people to triage server / intel / ati bugs during the cycle, so when we hit beta / RC stage, we didn't have the whole bug set well enough triaged to be able to be sure we picked the most important bugs as blockers. For F13 we should stay on top of triage better so we can do blocker identification accurately. happily, matej is more active on X triage again now and we have some more assistance from Chris Campbell (thank you Chris!) further volunteers would be great. I will try to stay on top of nouveau, again.
• Adam Williamson - we have the big security thing (aka lack of a defined security policy and test plan) to deal with. I did start a thread on fedora-devel-list about that.

Wishlist

• 66.187.233.202 - I would like a pony
• jlaska - I also want a pony
• Kparal - I would like to see more people in beta/RC testing. New templates are going to solve the practical part (display multiple results easily), now the issue is to attract more people. I would like to visit get-fedora page and see there "Fedora 13 Release Candidate available! Download <<here>>, help with testing <<here>>". And links to our wiki pages describing how to contribute on testing. I think Ubuntu is doing similar thing (large banners "Beta available" on their front page). That could bring a lot of new people.
• Kparal - Searching through our wiki I could not find a page named "Fedora 12", "Fedora 13", etc. Simply a page summarizing (linking) everything important about a release - schedules, features, current progress, (release notes). It's scattered in pages like these and these, but no page is standing out as an "outpost" of that particular release. I imagine if we have some page like that, we could add there links about the current testing efforts, so people would be easily navigated exactly where we need them to help ("current QA efforts: Fedora 13 RC2 testing"...). This "Fedora 13" overview page would of course be referenced from main wiki page, so people can easily find it.
• He Rui - More Asian testers could participate in the test events. (could inform them by forwarding announcements to their mail list/irc in native language)

Recommendations

After enough feedback has been collected, the QA team will discuss and make recommendations on changes for Fedora 13. This section will be filled in at that time.

References

• See Fedora_11_Retrospective_Notes