Proven tester: Difference between revisions

Latest revision as of 03:44, 6 February 2013

In other languages: English (en) italiano (it) português do Brasil (pt-br) [edit]

Proven testers on hiatus
As of 2013-02-05, the proven testers system is currently 'on hiatus'. Contrary to what's written below, proven tester feedback is at present treated no differently from any other feedback for the purposes of approving Fedora updates. This means that, at present, proven tester status has no consequence. The proven tester group continues to exist and proven tester feedback will still have (proventester) when listed in Bodhi, but please be aware there is no actual reason to do so under the current updates policy. We are keeping the group and documentation alive in the expectation that future improvements to Bodhi will allow us to make the group 'active' once more. We are not currently accepting applications to join the proventesters group: if it becomes active again, we will open up the application process again.

A proven tester, also known as a critical path wrangler, verifies and reports on the stability of test updates to critical path packages. They retrieve their updates from the QA:Updates_Testing repository and report their findings as karma using Bodhi. Positive karma from a proven tester is required for each critical path update before it can be pushed to the stable repository.

A proven tester is a member of the proventesters group. Individuals who wish to join this group must first be mentored and approved.

Testing process

Proven testers verify a basic level of stability before releasing an update to the public. They do not need to test for total correctness or ensure complete test coverage. Some tests will vary depending on the type of the package, while others must pass for all updates. Generally speaking, an update should successfully execute all applicable critical path actions. The Fedora release criteria is another useful guide for minimum testing criteria.

Proven testers verify updates by installing them from the updates-testing repository. For instructions on using this repository, see this page. To ensure rapid detection of regressions, you should perform a full system update from this repository at least once a day. You can update individual packages more quickly if the need for verification is urgent. We recommend that you have SELinux enabled and set to Enforcing mode (this is the default configuration, but many people disable SELinux after installing Fedora) for the purpose of testing.

General tests

The system must be able to shut down and reboot.
The user must be able to log in to the desktop.
The user must be able to access the network.

Testing applications

If the package is an application, run the application and check that basic operations work.

Testing libraries and shared components

If it is a library or other shared component, run an application which uses the component and ensure that it works.

Feedback procedures

In general, proven testers should follow the standard update feedback guidelines. Proven testers should pay special attention to critical path updates, and make a particular effort to provide feedback on these; usually, it is acceptable to post positive feedback on a critical path update so long as critical path functionality continues to work with the update in place.

Proven tester mentoring

Proven tester mentors accept requests from prospective proven tester members, and check that the applicants have read and understood the proven tester instructions before approving their membership. This process is not intended to be onerous, and we should expect to accept all applications unless they have obviously been made in error, seem malicious in intent or the applicant fails to affirm that they have read the instructions for the process.

Becoming a mentor

Any proven tester can become a mentor. Simply let any existing mentor or group administrator - those listed as administrator or sponsor in the group member list - know you would like to become a mentor, and they will upgrade you to sponsor level, which will allow you to accept applications to the group.

Mentoring process

Hiatus
As noted at the top of the page, the proven testers group is on hiatus. It is probably not necessary to perform any mentoring at this time.

You can find membership applications in Trac. You can also subscribe to an RSS feed of membership applications. To accept an application, assign it to yourself. Now ensure that the applicant has...

applied to the FAS group
read the instructions at Proven_tester
knows how to use the updates-testing repository and fedora-easy-karma.

If the applicant provides links to some feedback they have already posted, read these to check that they are in line with the process. If all of this is in order, sponsor the applicant into the proventesters FAS group, and close the application ticket. You can see an example completed application ticket here.

If a tester applies to the FAS group, but does not submit a corresponding TRAC ticket, reach out to the applicant directly by email using the email associated with their FAS account. Welcome their application and request that they also file a ticket for proper tracking.

@@ Line 1: / Line 1: @@
-== Overview ==
+{{autolang|base=yes}}
-A '''proven tester''' ensures that updates to [[Critical_Path_Packages_Proposal|critical path packages]] do not violate the [[Fedora Release Criteria]] and do not break any [[Critical_Path_Packages_Proposal#What_is_the_critical_path_of_actions.3F|critical path actions]]. Proven testers provide feedback through [http://bodhi.fedoraproject.org Bodhi] and [http://redhat.bugzilla.com Bugzilla] in the form of karma and bug reports. This feedback minimizes the risk of broken updates as their stability is assured before they are pushed to the community at large.
-== Responsibilities ==
+{{admon/important|Proven testers on hiatus|As of 2013-02-05, the proven testers system is currently 'on hiatus'. Contrary to what's written below, proven tester feedback is at present treated no differently from any other feedback for the purposes of approving Fedora updates. This means that, at present, proven tester status has no consequence. The proven tester group continues to exist and proven tester feedback will still have (proventester) when listed in Bodhi, but please be aware there is no actual reason to do so under the current updates policy. We are keeping the group and documentation alive in the expectation that future improvements to Bodhi will allow us to make the group 'active' once more. We are not currently accepting applications to join the proventesters group: if it becomes active again, we will open up the application process again.}}
-Proven testers are responsible for installing, testing, and providing feedback to updates for [[Critical Path Packages|critical path packages]].
-Feel free to vary the details as suits you, but please bear in mind the advice on what types of feedback to leave in what cases.
+A '''proven tester''', also known as a '''critical path wrangler''', verifies and reports on the stability of test updates to [[critical path package]]s. They retrieve their updates from the [[QA:Updates_Testing]] repository and report their findings as karma using [[Bodhi]]. Positive karma from a proven tester is required for each critical path update before it can be pushed to the stable repository.
-=== Find and install package updates for testing ===
+A proven tester is a member of the [https://admin.fedoraproject.org/accounts/group/view/proventesters proventesters] group. Individuals who wish to join this group must first be mentored and approved.
-Enable the [[QA/Updates_Testing|updates-testing]] repository, if you haven't already, and do a full system update each day. It is of course fine to quickly install one specific critical path update and test it in isolation, if you are particularly interested in ensuring the update is tested promptly, or the package maintainer sends out a request for testing.
+<!--
+== Joining the proven testers ==
+# Sign up for the [https://admin.fedoraproject.org/accounts/login Fedora Account System] (if you haven't already) and apply to join the ''proventesters'' group
+# File a ticket in [https://fedorahosted.org/fedora-qa/ Fedora QA Trac], with type ''proventester request'' and component ''Proventester Mentor Request'', asking to join the proven testers and requesting a mentor
+# Wait for a mentor to accept your request, and follow the instructions they provide
-=== Check for major regressions ===
+To speed up the process, you can get ahead with learning to be a proven tester by following the instructions further down this page, and filing some feedback while you wait for the mentoring process. Post in your mentor request to state that you have read the instructions on this page, and that you understand how to install test updates, test them, and post feedback. You can also link to some of the feedback you post, so your mentor can check to make sure you've got it right!
-Perform a quick functionality test to ensure the updated package is working as intended. If the package is an application, run the application and check that basic operations work. If it is a library or other shared component, run an application which uses the component and ensure that that works.
+-->
+== Testing process ==
+Proven testers verify a basic level of stability before releasing an update to the public. They do not need to test for total correctness or ensure complete test coverage. Some tests will vary depending on the type of the package, while others must pass for all updates. Generally speaking, an update should successfully execute all applicable [[critical path action]]s. The [[Fedora release criteria]] is another useful guide for minimum testing criteria.
-* After doing the update, reboot the system and log in to the desktop (assuming you can). If you observe any problems or changed behavior, take a detailed note and try to identify the package responsible for the change.
+Proven testers verify updates by installing them from the updates-testing repository. For instructions on using this repository, see [[QA:Updates_Testing|this page]]. To ensure rapid detection of regressions, you should perform a full system update from this repository at least once a day. You can update individual packages more quickly if the need for verification is urgent. We recommend that you have SELinux enabled and set to Enforcing mode (this is the default configuration, but many people disable SELinux after installing Fedora) for the purpose of testing.
-* Ensure your network access is working as usual. If not, try to identify the cause.
-* Check that the update application will run, although you now likely will have no updates remaining to test immediately whether it works. If not, try to identify the cause.
-=== Investigate and provide feedback ===
+=== General tests ===
-If the component works as expected, post positive feedback. If you identify a major problem, post negative feedback. If you identify a problem which is minor in nature and does not impede the actual critical path functionality, please do not post negative feedback. Post neutral or positive feedback with a note of the issue encountered (and a link to a bug report if appropriate).
+# The system must be able to shut down and reboot.
+# The user must be able to log in to the desktop.
+# The user must be able to access the network.
-* Use {{package|fedora-easy-karma}} to list all installed packages from the updates-testing repository and allow to file feedback on each one at a time.
+=== Testing applications ===
-* Pay particular attention to updates whose description notes that they are critical path updates. If you identified any serious problems in earlier testing and were able to identify the package responsible, post negative feedback for that package. If possible, please file a bug report on the problem and link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behavior change and the cause, if you were able to determine it.
+If the package is an application, run the application and check that basic operations work.
-* If you are not sure what the component does or how to test it, do not post positive or negative feedback. If the above general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed.
-* If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their problem, and post negative feedback if you are now able to confirm it. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is a mistake.
-== Typical Scenarios ==
+=== Testing libraries and shared components ===
-Since a proven tester's karma determines whether an update is allowed to be promoted, they follow special procedures based on the severity of regressions they encounter. A bug or regression is considered severe if it violates either the release criteria or breaks a critical path action.
+If it is a library or other shared component, run an application which uses the component and ensure that it works.
-* Major bug - Report, vote down.
-* Minor bug - Report, vote up/neutral
-* Previously reported bug - Confirm, vote accordingly
-== Unusual Scenarios ==
+== Feedback procedures ==
-* Unreproducible bug
+In general, proven testers should follow the standard [[QA:Update feedback guidelines|update feedback guidelines]]. Proven testers should pay special attention to critical path updates, and make a particular effort to provide feedback on these; usually, it is acceptable to post positive feedback on a critical path update so long as critical path functionality continues to work with the update in place.
-* Unfamiliar package
+== Proven tester mentoring ==
+Proven tester mentors accept requests from prospective proven tester members, and check that the applicants have read and understood the proven tester instructions before approving their membership. This process is not intended to be onerous, and we should expect to accept all applications unless they have obviously been made in error, seem malicious in intent or the applicant fails to affirm that they have read the instructions for the process.
+=== Becoming a mentor ===
+Any proven tester can become a mentor. Simply let any existing mentor or group administrator - those listed as ''administrator'' or ''sponsor'' in the [https://admin.fedoraproject.org/accounts/group/members/proventesters group member list] - know you would like to become a mentor, and they will upgrade you to ''sponsor'' level, which will allow you to accept applications to the group.
+=== Mentoring process ===
+{{admon/important|Hiatus|As noted at the top of the page, the proven testers group is on hiatus. It is probably not necessary to perform any mentoring at this time.}}
+You can [https://fedorahosted.org/fedora-qa/query?component=Proventester+Mentor+Request find membership applications] in Trac.  You can also subscribe to an [https://fedorahosted.org/fedora-qa/query?status=new&status=reopened&format=rss&component=Proventester+Mentor+Request&order=priority RSS feed of membership applications]. To accept an application, assign it to yourself. Now ensure that the applicant has...
+# applied to the FAS group
+# read the instructions at [[Proven_tester]]
+# knows how to use the updates-testing repository and {{command|fedora-easy-karma}}.
+If the applicant provides links to some feedback they have already posted, read these to check that they are in line with the process. If all of this is in order, sponsor the applicant into the ''proventesters'' FAS group, and close the application ticket.  You can see an example completed application ticket [https://fedorahosted.org/fedora-qa/ticket/67 here].
+If a tester applies to the FAS group, but does not submit a corresponding [https://fedorahosted.org/fedora-qa TRAC ticket], reach out to the applicant directly by email using the email associated with their FAS account.  Welcome their application and request that they also file a ticket for proper tracking.
+== External Links ==
+* [http://bodhi.fedoraproject.org Bodhi]
+* [http://bugzilla.redhat.com Bugzilla]

Search