|
|
| (40 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| This is a copy of [[QA:Rawhide_Acceptance_Test_Plan]] but uses [[Template:QA/Test_Plan]] instead.
| | {{header|qa}} |
| | {{draft}} |
|
| |
|
| {{QA/Test_Plan
| | This page will outline criteria for success for a Fedora AMI release |
| |history=
| |
| * First draft: [[User:Wwoods|WillWoods]] 20:54, 17 June 2009 (UTC)
| |
|
| |
|
| |introduction=
| | = Release Criteria = |
| This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.
| |
|
| |
|
| In short, this is how we decide if Rawhide is broken or not. | | In addition to the existing [[Fedora Release Criteria]], the following criteria apply to EC2 Fedora images. A <span style="color:green; font-weight:bold;">good</span> AMI '''must'''... |
| |strategy=
| | # allow non-root user login |
| There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.
| | # have SELinux enabled and ''enforcing'' by default |
| | # not contain any package dependency problems |
| | # not include any packages built outside of Fedora infrastructure |
|
| |
|
| |priority=
| | = How to Test = |
| TBD. No clear priorities exist yet.
| |
|
| |
|
| |scope= | | To ''create'' an EC2 image, one needs... |
| This plan seeks to answer three basic questions:
| | # an amazon account |
| | # the {{package|python-boto}} libraries installed |
| | # a python script to tell Amazon to build an instance |
| | # the ability to SSH into a machine to run tests, etc. |
|
| |
|
| # Can current Rawhide users update their systems using this repo?
| | = Open Questions = |
| # Can this Rawhide tree be installed? | | # Who is responsible for creating and building EC2 images? |
| # Does the basic system (a subset of the [[Critical_Path_Packages_Proposal|"critical path"]]) work as expected for simple testing? | | #: |
|
| |
|
| It is ''not'' intended to be an exhaustive test of any part of the system.
| | = Additional Reading = |
| | | * [[Publishing_image_to_EC2]] |
| |criteria=
| | * [[Cloud_SIG]] |
| Rawhide will be considered ''Good'' for each purpose if all of the underlying conditions are met:
| | * [[Cloud_SIG/EC2_Images]] |
| == Repo Sanity ==
| |
| * Contains valid yum metadata (repodata)
| |
| * Contains key packages (kernel, glibc, coreutils)
| |
| * No unresolved dependencies in critical packages
| |
| * comps.xml exists and is valid
| |
| | |
| == Installability ==
| |
| * installer images (kernel, initrd, install.img, and boot.iso) exist
| |
| * kernel boots on ''most'' machines of the primary architectures
| |
| * initrd is able to find stage2 (install.img) by at least one method (network, local CD)
| |
| * stage2 is able to detect the presence of disks attached to most common controllers
| |
| * installer can write packages to disk storage
| |
| * installer can setup up bootloader correctly for common boot systems
| |
| | |
| == Basic Functionality ==
| |
| * Kernel/X is able to set up common display configurations | |
| ** At least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) must be functioning
| |
| * Kernel/X properly handle input from standard USB keyboard/mouse
| |
| * System is able to establish a wired network connection
| |
| ** As with video, this should work for ''most'' common network drivers. Individual chipset failures may be waived.
| |
| * {{package|yum}} is able to install simple updates
| |
| | |
| |deliverables=
| |
| This test plan should produce:
| |
| * A summary report on whether Rawhide is broken or not
| |
| * Bug reports for broken dependencies / missing files / etc.
| |
| * A list of test cases used to verify the expected results. (see below)
| |
| | |
| |testcases=
| |
| ''TODO'': Automate these test cases. See [https://fedorahosted.org/autoqa/query?status=new&status=assigned&status=reopened&component=tests&summary=~Automate&milestone=israwhidebroken.com&order=priority tickets for each test here].
| |
| | |
| == Repo Sanity ==
| |
| # [[QA:Repodata validity test case|Repodata validity]]
| |
| # [[QA:Comps Validity Test Case|comps.xml validity]]
| |
| # [[QA:Core package dependency closure test case|Core package dependency closure]]
| |
| # [[QA:Core package existence test case|Core package existence]]
| |
| | |
| == Installability ==
| |
| # [[QA:Installer image presence test case|Installer image existence]]
| |
| # [[QA:Kernel simple boot test case|Kernel boot]]
| |
| # [[QA:Anaconda stage2 fetch test case|Anaconda loader fetching stage2]]
| |
| # [[QA:Anaconda storage probe test case|Anaconda stage2 disk probe]]
| |
| # [[QA:Anaconda package install test case|Anaconda package install]]
| |
| # [[QA:Anaconda bootloader setup test case|Anaconda bootloader setup]]
| |
| | |
| == Basic Functionality ==
| |
| # [[QA:X basic display test case|X startup/basic display configuration]]
| |
| # [[QA:X basic input handling test case|X basic input handling]]
| |
| # [[QA:Network basic test case|Basic network connectivity]]
| |
| # [[QA:Yum simple update test case|yum update functionality]]
| |
| | |
| |environment=
| |
| * The tests will be run on a host which has the Rawhide tree accessible by HTTP or FTP.
| |
| ** The tests will download a lot of data, so ideally the host should have a very fast network link to the HTTP/FTP mirror.
| |
| * The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
| |
| * The test host will have at least 1GB free disk space:
| |
| ** critical path packages (~300MB)
| |
| ** their associated metadata (~200MB)
| |
| ** all of the boot images (~300MB)
| |
| ** plus some extra scratch space for rebuilding images etc.
| |
| | |
| |resposibility=
| |
| Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
| |
| | |
| |schedule=
| |
| Ideally this test plan should be run for ''every new Rawhide tree'' - i.e. daily.
| |
| | |
| |risks=
| |
| | |
| * Testing kernel boot and X startup is currently tricky to automate.
| |
| * [[Critical_Path_Packages_Proposal]] - while exact list of packages may change, it is possible that no agreement on the exact list can be made. In this event, a temporary subset of critical packages will be used | |
| | |
| |reviewedby=
| |
| <!-- Add a line like this:
| |
| * ~~~~
| |
| This will automatically be expanded to your username and timestamp. Preview to check it. -->
| |
| * [[User:Jlaska|jlaska]] 11:13, 19 June 2009 (UTC) <!-- jlaska --> | |
| * [[User:Johannbg|viking-ice]] 16:14, 24 June 2009 (UTC)
| |
| | |
| |references=
| |
| * [[JohnPoelstra/ImproveRawhideF10#Defining_GOOD]] - A previous discussion of Rawhide requirements
| |
| * [[Critical Path Packages Proposal]] - Proposal to define the most critical packages - required for some repo sanity test cases
| |
| * [[Israwhidebroken.com Proposal]] - This test plan is a central part of this proposal.
| |
| <references/>
| |
| }}
| |
python-boto