From Fedora Project Wiki

(Add Liam Li feedback)
Line 45: Line 45:
 
* [[User:Rhe|He Rui]] - Tickets could be created and traced in Trac system  
 
* [[User:Rhe|He Rui]] - Tickets could be created and traced in Trac system  
 
* [[User:Adamwill|Adam Williamson]] - test days went well again. it was nice to see how many ''independent'' test days there were.
 
* [[User:Adamwill|Adam Williamson]] - test days went well again. it was nice to see how many ''independent'' test days there were.
 +
* [[User:Liam|Liam Li]] - The schedule of Fedora 12 Quality Tasks is accurate,from there, we can exactly know when to start the install testing. 2. The ticket to request Compose or media is very convenient and has high performance
 +
* [[User:Liam|Liam Li]] - People who participate in install test are increasing
  
 
== Could have been better ==
 
== Could have been better ==
Line 69: Line 71:
 
* '''Jim Haynes''' - I've learned it is necessary to install foomatic-db to have my particular printer show up in system->administration->printing.  According to [[User:Adamwill|Adam Williamson]], This has been documented in common bugs, but it does seem like something we ought to have caught - thanks for bringing it up. Perhaps we need more testing of common basic peripherals like printers.
 
* '''Jim Haynes''' - I've learned it is necessary to install foomatic-db to have my particular printer show up in system->administration->printing.  According to [[User:Adamwill|Adam Williamson]], This has been documented in common bugs, but it does seem like something we ought to have caught - thanks for bringing it up. Perhaps we need more testing of common basic peripherals like printers.
 
* '''CAI Qian''' - Power Management Test Day is boring, because it provides a automated scripts to run, and not sure how to inspect that data from the testers. As the results, it lose the fun to hunt bugs. It is important to have some instructions to inspect the data we received in the future, so we can see if there any regression or issues.
 
* '''CAI Qian''' - Power Management Test Day is boring, because it provides a automated scripts to run, and not sure how to inspect that data from the testers. As the results, it lose the fun to hunt bugs. It is important to have some instructions to inspect the data we received in the future, so we can see if there any regression or issues.
 +
* [[User:Liam|Liam Li]] - Install test on ppc platform is not sufficient. I mean some cases can not be executed comparing with i386/x86_64
 +
* [[User:Liam|Liam Li]] - Speed of test Media for download is snow,people are unwilling to download DVD to test.could we put this to mirror?or think about other way to solve this problem?
 +
* [[User:Liam|Liam Li]] - The install time is very long,most of the time is spent on package install stage,but most of the bugs are not occurred on packaging install stage,can we build a CD which is very similar with LiveCD, but it just for install test,only install gnome and other basic packages?By this way, we can find anaconda bugs, but the size of this media is less then 700M. Or we can add this function to LiveCD,there is an option to start install on boot stage,not to start install after login desktop like liveCD.
 +
* [[User:Liam|Liam Li]] - The bug maintenance takes lots of time.Can we build a bug report instructions about what logs need to attach when report a bug for specified component.or this can build to bugzilla? when selected a component,some logs/information are recommended to attach to save the communication time between the developers and testers. By this way,at least some necessary information will not be missing because of the carelessness of testers, and the reproduce time will be reduced when some necessary information missed
  
 
== Wishlist ==
 
== Wishlist ==
Line 81: Line 87:
 
* [[User:Kparal|Kparal]] - People attending Test Days presentation asked if test day results are presented and published externally, like in magazines, etc.
 
* [[User:Kparal|Kparal]] - People attending Test Days presentation asked if test day results are presented and published externally, like in magazines, etc.
 
* [[User:Kparal|Kparal]] - People attending Test Days presentation asked if Fedora Test Days are on Facebook and Twitter, which could attract larger audience.
 
* [[User:Kparal|Kparal]] - People attending Test Days presentation asked if Fedora Test Days are on Facebook and Twitter, which could attract larger audience.
 +
* [[User:Liam|Liam Li]] - Have more method and more broadcast way to let more people know our testing
  
 
= Recommendations =
 
= Recommendations =

Revision as of 04:13, 7 December 2009

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 12. The feedback will be used as a basis for identifying areas for improvement for Fedora 13 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to fedora-test-list@redhat.com.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

  1. Login to the wiki
  2. Select [Edit] for the appropriate section below.
  3. Add your feedback using the format:
    * ~~~ - I like ____ about the new ____ process
  4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

  1. Select the appropriate page for your feedback...
  2. Add your feedback using the format:
    * ~~~ - I like ____ about the new ____ process
  3. When done, Submit your changes

Feedback

Things that went well

  • jlaska - Having QA/SOP_Test_Day_management was invaluable and resulted in a consistent look'n'feel for Fedora 12 test days
  • jlaska - The creation of the test-announce@lists.fedorahosted.org mailing list made it much easier to announce test events. With a single mailing list for announcements, it's harder to miss any events.
  • jlaska - Consistent announcement of ISO test events from Liam was great to ensure a consistent testing experience (see User:Liam/Draft_Install_Test_SOP).
  • jlaska - We had more ISO install test runs, and they were earlier in each milestone
  • jlaska - F-12 ISO Install testing solicited more testing from outside the core install test group. 27 different people contributed test feedback on the wiki test matrix during F-12. . 19 different people contributed test feedback on the wiki test matrix during F-11.
  • jlaska - Automated test cases provided for the Power Management test day == awesome!
  • Viking-Ice - Increase in SIG and maintainers for hosting and holding their own test days and specific application testing.
  • Viking-Ice - Nightly-composes of rawhide test images.
  • Viking-Ice - Improvement in advertisement testing/test day's
  • Viking-Ice - Increase in request from sig/maintainers in Fedora QA Trac system.
  • Viking-Ice - Increase in how to debug <component> for reporters to follow and provide better bug report.
  • Viking-Ice - Consistency in how to debug <component> pages for better look and feel.
  • Viking-Ice - Automatic bug reporting tool develop to help provide better bug reporting and allow technical challenge people to participate report and provide feedback.
  • Viking-Ice - Better collaboration and participation from developers/maintainers with QA.
  • He Rui - The SOP management of both test days and install tests
  • He Rui - Traceback can be saved or reported automatically
  • He Rui - Common f12 bugs page
  • He Rui - Tickets could be created and traced in Trac system
  • Adam Williamson - test days went well again. it was nice to see how many independent test days there were.
  • Liam Li - The schedule of Fedora 12 Quality Tasks is accurate,from there, we can exactly know when to start the install testing. 2. The ticket to request Compose or media is very convenient and has high performance
  • Liam Li - People who participate in install test are increasing

Could have been better

  • anonymous

uname -a hera.localdomain 2.6.31.12-174.2.3.fc12.x86_64 #1 SMP Mon Jan 18 19:52:07 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux language brazilian portuguese

this works for me ls -l *.rpm

look at "./" preceding the char "*" this works for me ls -l ./*.pps

but

this does not works for me

ls -l *.pps ls: invalid option -- 'P' Experimente "ls --help" para mais informações.

I think this is a bug

  • jlaska - it may be, but I believe you might want to confirm what the asterix is expanding to in the directory you are running the command ls -l *.pps. It's possible it is expanding to a filename that has -P as the first 2 characters, which confuses the ls command. What does echo *.pps yield? You may wish to enclose the command in quotes, ls -l "*.pps". If problems remain, please raise the issue on the users or test list.
  • jlaska - The blocker bug meetings during the Final milestone were too long.
  • Kparal - The preupgrade could receive more testing/test cases
  • Kparal - test-announce wasn't used for announcing RC candidates IIRC, we should make sure all important events are mentioned there
  • He Rui - Post-event review python could be used for install test like in test day
  • He Rui - Transfer from manual-installation to Auto-installation test
  • He Rui - Test days/Install test events could be put up obviously on wiki so that people who wanna contribute can find it easily from wiki except from mail lists.
  • He Rui - 'Edit' on wiki could be more automatically by filling in and submitting a form without wiki syntax.
  • Adam Williamson - in the end, we focused heavily on just three component areas for testing: anaconda, kernel, and X.org. This is primarily a function of the fact that these are the most vital bits; it feels like we're still at the point of doing let's make sure it's not totally broken testing than let's make sure it's really good testing. We didn't do stuff like making sure the desktop was polished.
    • Matthias Clasen - I agree with this assessment. I would go even further and say it is to a large extent 'lets make sure the installer is not totally broken for exotic cases' testing.
  • Adam Williamson - there was clearly a lot of uncertainty about RAID issues; it's something we obviously don't as a team test well enough (and some of us personally don't understand enough :>). In the end there didn't turn out to be any horrible issues, but the confusion was evident, and we did miss the Intel BIOS RAID stuff-ups. For F13 we should have better RAID testing both in Test Days and in pre-release test cycles.
  • Adam Williamson - We weren't completely on top of X.org bugs for this release. The ones that wound up getting promoted to release blocker level were kind of an arbitrary selection. I think we got nouveau mostly right as I had a reasonable grip on nouveau triage, but we just had too few people to triage server / intel / ati bugs during the cycle, so when we hit beta / RC stage, we didn't have the whole bug set well enough triaged to be able to be sure we picked the most important bugs as blockers. For F13 we should stay on top of triage better so we can do blocker identification accurately. happily, matej is more active on X triage again now and we have some more assistance from Chris Campbell (thank you Chris!) further volunteers would be great. I will try to stay on top of nouveau, again.
  • Adam Williamson - we have the big security thing (aka lack of a defined security policy and test plan) to deal with. I did start a thread on fedora-devel-list about that.
  • Matthias Clasen - From my perspective, the two main avenues to a new Fedora release are the live installer and preupgrade, and those two should get all the attention they can get.
  • Rahul Sundaram - IMO, RCs needs to advertised loudly. We need all the testing we can get and more alpha or beta snapshots would help as well, I think.
  • Rahul Sundaram - PackageKit signed install policy was a major issue and although QA was not really responsible for it, the team does need to do it all can do to avoid anything like this in the future . Signing Rawhide packages automatically would have caught this. Not many users in forum or fedora-list complained about it however.
  • Rahul Sundaram - Lot more users are trying Preupgrade and QA needs strong focus on the upgrade story including test days directed towards it. The small /boot is a major issue and another common bug (not listed in the wiki but I think needs to be added) is https://bugzilla.redhat.com/show_bug.cgi?id=538118 which makes the problem worse. We really need to make sure Anaconda creates a bigger /boot for Fedora 13, preupgrade is explicitly tested well and has solid workarounds for the small /boot case.
  • Rahul Sundaram - KMS is still flaky in some cases. In particular, a few Intel users seem to be reporting lower resolution by default without nomodeset and ATI performance seems to have regressed. Although I am no fan of proprietary drivers, I must note that installing the proprietary Nvidia driver has become a bit more of a hassle.
  • Scott Robbins - Although it's not much of a factor in the US, apparently bandwidth limits are quite common in Australia--there were several who seemed rather disgruntled about the somewhat confusing SHA1/SHA256 sum issue. Although it's now clear in the release notes, those who verify, as a rule, have done it before with other distributions, and are used to downloading an ISO, looking at the sums file, and seeing that it's md5 (if anyone's still using it), or SHAwhatever. Upon getting what seems to be a bad checksum, they download again, which in fairness, if you get one bad checksum, is probably the logical thing to do.
  • Jim Haynes - I've learned it is necessary to install foomatic-db to have my particular printer show up in system->administration->printing. According to Adam Williamson, This has been documented in common bugs, but it does seem like something we ought to have caught - thanks for bringing it up. Perhaps we need more testing of common basic peripherals like printers.
  • CAI Qian - Power Management Test Day is boring, because it provides a automated scripts to run, and not sure how to inspect that data from the testers. As the results, it lose the fun to hunt bugs. It is important to have some instructions to inspect the data we received in the future, so we can see if there any regression or issues.
  • Liam Li - Install test on ppc platform is not sufficient. I mean some cases can not be executed comparing with i386/x86_64
  • Liam Li - Speed of test Media for download is snow,people are unwilling to download DVD to test.could we put this to mirror?or think about other way to solve this problem?
  • Liam Li - The install time is very long,most of the time is spent on package install stage,but most of the bugs are not occurred on packaging install stage,can we build a CD which is very similar with LiveCD, but it just for install test,only install gnome and other basic packages?By this way, we can find anaconda bugs, but the size of this media is less then 700M. Or we can add this function to LiveCD,there is an option to start install on boot stage,not to start install after login desktop like liveCD.
  • Liam Li - The bug maintenance takes lots of time.Can we build a bug report instructions about what logs need to attach when report a bug for specified component.or this can build to bugzilla? when selected a component,some logs/information are recommended to attach to save the communication time between the developers and testers. By this way,at least some necessary information will not be missing because of the carelessness of testers, and the reproduce time will be reduced when some necessary information missed

Wishlist

  • 66.187.233.202 - I would like a pony
  • jlaska - I also want a pony
  • Kparal - I would like to see more people in beta/RC testing. New templates are going to solve the practical part (display multiple results easily), now the issue is to attract more people. I would like to visit get-fedora page and see there "Fedora 13 Release Candidate available! Download <<here>>, help with testing <<here>>". And links to our wiki pages describing how to contribute on testing. I think Ubuntu is doing similar thing (large banners "Beta available" on their front page). That could bring a lot of new people.
  • Kparal - Searching through our wiki I could not find a page named "Fedora 12", "Fedora 13", etc. Simply a page summarizing (linking) everything important about a release - schedules, features, current progress, (release notes). It's scattered in pages like these and these, but no page is standing out as an "outpost" of that particular release. I imagine if we have some page like that, we could add there links about the current testing efforts, so people would be easily navigated exactly where we need them to help ("current QA efforts: Fedora 13 RC2 testing"...). This "Fedora 13" overview page would of course be referenced from main wiki page, so people can easily find it.
  • He Rui - More Asian testers could participate in the test events. (could inform them by forwarding announcements to their mail list/irc in native language)
  • Jim Haynes - Over all this is the easiest Fedora version upgrade I have ever done; and when I did have problems I got quick and correct answers from this list (fedora-test-list@redhat.com) that I was not getting anywhere else. (Maybe that suggests we should have a current-version-install ombudsman list, where anybody can present questions but only authoritative people can give answers.)
  • Kparal - People attending Test Days presentation asked which sources are most attractive for Test Day announcements and allure many people. Where certainly not to forget to advertise when arranging a test day.
  • Kparal - People attending Test Days presentation asked if test day results are presented and published externally, like in magazines, etc.
  • Kparal - People attending Test Days presentation asked if Fedora Test Days are on Facebook and Twitter, which could attract larger audience.
  • Liam Li - Have more method and more broadcast way to let more people know our testing

Recommendations

After enough feedback has been collected, the QA team will discuss and make recommendations on changes for Fedora 13. This section will be filled in at that time.


References