From Fedora Project Wiki

(update recommendations section with non-ticketable recommendations, tickets filed for ticketable items, and one action already taken)
 
(17 intermediate revisions by 8 users not shown)
Line 28: Line 28:
=== Things that went well ===
=== Things that went well ===
{{Test_Results:Fedora_35_QA_Retrospective/good}}
{{Test_Results:Fedora_35_QA_Retrospective/good}}
[[User:Adamwill|adamwill]] - We slipped both Beta and Final appropriately and did not rush the releases.
* [[User:Adamwill|adamwill]] - We slipped both Beta and Final appropriately and did not rush the releases.
[[User:Adamwill|adamwill]] - The final RC was built plenty of time before Go/No-Go, allowing for a good amount of testing.
* [[User:Adamwill|adamwill]] - The final RC was built plenty of time before Go/No-Go, allowing for a good amount of testing.
[[User:Adamwill|adamwill]] - We were able to resolve all the KDE-related blockers with good co-operation from Fedora and upstream KDE folks, especially Aleix Pol who resolved a lot of the Discover blockers.
* [[User:Adamwill|adamwill]] - We were able to resolve all the KDE-related blockers with good co-operation from Fedora and upstream KDE folks, especially Aleix Pol who resolved a lot of the Discover blockers.
[[User:Jpbn|Jpbn]] ([[User talk:Jpbn|talk]]) I like the James Laska's old list of leading questions to prompt feedback, because it's still pretty good: it shows the way to improve my knowledge about the test proces.
*[[User:Jpbn|Jpbn]] ([[User talk:Jpbn|talk]]) I like the James Laska's old list of leading questions to prompt feedback, because it's still pretty good: it shows the way to improve my knowledge about the test proces.
[[User:Ahmedalmeleh|Ahmedalmeleh]] I only started contributing to the QA team for Fedora 35 Final, I enjoyed submitting my viewpoints in IRC and joining the GO/No-go meeting. I know how to use the blocker bugs tool and Bodhi quite well (Fedora Updates System).
*[[User:Ahmedalmeleh|Ahmedalmeleh]] - I only started contributing to the QA team for Fedora 35 Final, I enjoyed submitting my viewpoints in IRC and joining the Go/No-go meeting. I know how to use the blocker bugs tool and Bodhi quite well (Fedora Updates System). I am glad we took our time to perfect the Final Fedora 35 release and overcame any obstacles (blocker bugs) that stood in our way. By the way I used some of "James Laska's old list of leading questions to prompt feedback" to answer some of this, it still works.
* [[User:Mattdm|Mattdm]] ([[User talk:Mattdm|talk]]) - Overall release is very solid and well-received. I feel confident that we're doing a good job in making sure upgrades (and new install first impressions) are pleasant for users.
* [[User:sumantrom|sumantrom]] ([[User talk:sumantrom|talk]]) - Fedora CoreOS aligned with our release cycle and we got time to capture bugs in the test week. [https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/message/XB3XP3X5CYTK5Z7VMVHWDZLKARKAWDLP/ Read More]
* [[User:sumantrom|sumantrom]] ([[User talk:sumantrom|sumantrom]]) - There is a growth in QA testers in terms of test days. The test days app hasn't gone down even ONCE ''*KUDOS frantizek*''
 
=== Could have been better ===
=== Could have been better ===
{{Test_Results:Fedora_35_QA_Retrospective/bad}}
{{Test_Results:Fedora_35_QA_Retrospective/bad}}
[[User:Adamwill|adamwill]] - It took a long time to work through all the iterations of the [https://bugzilla.redhat.com/show_bug.cgi?id=2016510 fedora-third-party/SELinux bug]. Seems like we could have tightened the loop on that somehow.
* [[User:Adamwill|adamwill]] - It took a long time to work through all the iterations of the [https://bugzilla.redhat.com/show_bug.cgi?id=2016510 fedora-third-party/SELinux bug]. Seems like we could have tightened the loop on that somehow.
[[User:Adamwill|adamwill]] - Movement on the KDE blockers was initially slow, if we could have resolved them earlier we would have had more time to find and fix the bugs that eventually delayed Final.
* [[User:Adamwill|adamwill]] - Movement on the KDE blockers was initially slow, if we could have resolved them earlier we would have had more time to find and fix the bugs that eventually delayed Final.
[[User:Ahmedalmeleh|Ahmedalmeleh]] - Bugzilla is challenging to me and still getting used to it. I wish I was able to learn how to operate OpenQA's automated tests.
* [[User:Ahmedalmeleh|Ahmedalmeleh]] - Bugzilla is challenging to me and still getting used to it. I wish I was able to learn how to operate OpenQA's automated tests, in the given timeframe and was given guidance sooner.
* [[User:kparal|kparal]] - After F34, we talked in our team about doing a selected compose testing in a complete full run (all test cases), way ahead of the milestones (Beta and Final), and as a group effort to encourage people to really fill out all test cases. Yet we never got to it in F35. Some of the bugs might've been discovered earlier this way.
* [[User:Mattdm|Mattdm]] ([[User talk:Mattdm|talk]]) - Seconded on the KDE blockers. As I understand it, KDE SIG is working on getting KDE upstream folks directly involved and aware of potentially-blocking issues sooner, which will really help.
* [[User:Mattdm|Mattdm]] ([[User talk:Mattdm|talk]]) - The [https://community.kde.org/Schedules/Plasma_5 timing of KDE Plasma releases] is not well-aligned with our schedule. We were trying to stabilize with 5.22, when upstream was really on to 5.23 (which is now in our updates-testing.) With F36, we're probably okay, with the 5.24 release coming the same day branch and continuing with 5.24.z bugfixes right through shortly after our release day. But with F37, 5.26.0 drops just a week before our "Early' Final target date.
* [[User:Mattdm|Mattdm]] ([[User talk:Mattdm|talk]]) - Lots of people (as non-scientifically measured by my personal impressions of help requests I'm seeing) are hitting the [https://fedoraproject.org/wiki/Common_F35_bugs#No_sound_after_upgrade wireplummer-not-enabled] problem. Maybe we should have blocked on that, actually.
* [[User:copperi|copperi]] agree on weird sound issues could have been tested more.
* [[User:sumantrom|sumantrom]] ([[User talk:sumantrom|talk]]) - Sending some testing event updates to our "users" list. I see surge in contributors when there is an article posted in Fedora Magazine which means Fedora's user base wants to contribute/ try out things. Posting things in the user-list will help.
* [[User:sumantrom|sumantrom]] ([[User talk:sumantrom|talk]]) - Onboarding session on Bugzilla and test case writing.
* [[User:frantisekz|frantisekz]] - Some testing could have been conducted earlier in the development, especially KDE/Discover blockers and bugs have been present for a long time before the release.


=== Wishlist ===
=== Wishlist ===
{{Test_Results:Fedora_35_QA_Retrospective/wishlist}}
{{Test_Results:Fedora_35_QA_Retrospective/wishlist}}  
* [[User:Ahmedalmeleh|Ahmedalmeleh]] - For Fedora 36 Beta I wish documentation on how to use OpenQA and Bugzilla was more easily accessible and we would give new participants even more range of tools and knowledge from the start.


== Recommendations ==
== Recommendations ==
Line 46: Line 59:


In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the [https://pagure.io/fedora-qa/ QA issue tracker].
In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the [https://pagure.io/fedora-qa/ QA issue tracker].
* Be more proactive about co-ordinating complex SELinux fixes between SELinux and component maintainers in future
* Keep in touch with newer KDE contributors from the Fedora 35 cycle so we can fix future KDE blockers promptly
* [https://pagure.io/fedora-qa/issue/684 Make a Bugzilla how-to video]
* [https://pagure.io/fedora-qa/issue/685 Improve Bugzilla instructions]
* [https://pagure.io/fedora-qa/issue/686 Fedora 36 Beta validation test event]
* [https://pagure.io/fedora-qa/issue/687 Onboarding session on Bugzilla and test case writing]
* [https://pagure.io/fedora-qa/issue/688 Create test cases for graphical package management]
* [https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/254 Automate graphical package manager test cases]
* Update [[QA/SOP_Test_Day_management]] "Promote the Test Day" section - [https://fedoraproject.org/w/index.php?title=QA/SOP_Test_Day_management&diff=632222&oldid=595200 done]


== References ==
== References ==

Latest revision as of 23:52, 2 December 2021

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 35. The feedback will be used as a basis for identifying areas for improvement for Fedora 36 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

  • Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

  1. Login to the wiki
  2. Select [Edit] for the appropriate section below.
  3. Add your feedback using the format:
    * ~~~ - I like ____ about the new ____ process
  4. When done, Submit your changes

Anonymous

If you do not have a Fedora account, follow the instructions below to submit anonymous feedback. Please note, mediawiki records the IP address associated with any anonymous page edits.

  1. Select the appropriate page for your feedback...
  2. Add your feedback using the format:
    * ~~~ - I like ____ about the new ____ process
  3. When done, Submit your changes

Feedback

Things that went well

  • adamwill - We slipped both Beta and Final appropriately and did not rush the releases.
  • adamwill - The final RC was built plenty of time before Go/No-Go, allowing for a good amount of testing.
  • adamwill - We were able to resolve all the KDE-related blockers with good co-operation from Fedora and upstream KDE folks, especially Aleix Pol who resolved a lot of the Discover blockers.
  • Jpbn (talk) I like the James Laska's old list of leading questions to prompt feedback, because it's still pretty good: it shows the way to improve my knowledge about the test proces.
  • Ahmedalmeleh - I only started contributing to the QA team for Fedora 35 Final, I enjoyed submitting my viewpoints in IRC and joining the Go/No-go meeting. I know how to use the blocker bugs tool and Bodhi quite well (Fedora Updates System). I am glad we took our time to perfect the Final Fedora 35 release and overcame any obstacles (blocker bugs) that stood in our way. By the way I used some of "James Laska's old list of leading questions to prompt feedback" to answer some of this, it still works.
  • Mattdm (talk) - Overall release is very solid and well-received. I feel confident that we're doing a good job in making sure upgrades (and new install first impressions) are pleasant for users.
  • sumantrom (talk) - Fedora CoreOS aligned with our release cycle and we got time to capture bugs in the test week. Read More
  • sumantrom (sumantrom) - There is a growth in QA testers in terms of test days. The test days app hasn't gone down even ONCE *KUDOS frantizek*

Could have been better

  • adamwill - It took a long time to work through all the iterations of the fedora-third-party/SELinux bug. Seems like we could have tightened the loop on that somehow.
  • adamwill - Movement on the KDE blockers was initially slow, if we could have resolved them earlier we would have had more time to find and fix the bugs that eventually delayed Final.
  • Ahmedalmeleh - Bugzilla is challenging to me and still getting used to it. I wish I was able to learn how to operate OpenQA's automated tests, in the given timeframe and was given guidance sooner.
  • kparal - After F34, we talked in our team about doing a selected compose testing in a complete full run (all test cases), way ahead of the milestones (Beta and Final), and as a group effort to encourage people to really fill out all test cases. Yet we never got to it in F35. Some of the bugs might've been discovered earlier this way.
  • Mattdm (talk) - Seconded on the KDE blockers. As I understand it, KDE SIG is working on getting KDE upstream folks directly involved and aware of potentially-blocking issues sooner, which will really help.
  • Mattdm (talk) - The timing of KDE Plasma releases is not well-aligned with our schedule. We were trying to stabilize with 5.22, when upstream was really on to 5.23 (which is now in our updates-testing.) With F36, we're probably okay, with the 5.24 release coming the same day branch and continuing with 5.24.z bugfixes right through shortly after our release day. But with F37, 5.26.0 drops just a week before our "Early' Final target date.
  • Mattdm (talk) - Lots of people (as non-scientifically measured by my personal impressions of help requests I'm seeing) are hitting the wireplummer-not-enabled problem. Maybe we should have blocked on that, actually.
  • copperi agree on weird sound issues could have been tested more.
  • sumantrom (talk) - Sending some testing event updates to our "users" list. I see surge in contributors when there is an article posted in Fedora Magazine which means Fedora's user base wants to contribute/ try out things. Posting things in the user-list will help.
  • sumantrom (talk) - Onboarding session on Bugzilla and test case writing.
  • frantisekz - Some testing could have been conducted earlier in the development, especially KDE/Discover blockers and bugs have been present for a long time before the release.

Wishlist

  • Ahmedalmeleh - For Fedora 36 Beta I wish documentation on how to use OpenQA and Bugzilla was more easily accessible and we would give new participants even more range of tools and knowledge from the start.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 36. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the QA issue tracker.

References