Is anaconda broken roadmap: Difference between revisions

Latest revision as of 07:02, 1 June 2012

This page provides a high-level roadmap for implementing the Is_anaconda_broken_proposal project. More detailed tasks can be found in autoqa TRAC roadmap. We follow these steps to define the methods by which we initiate testing

First, in order to provide a consistent and documented test approach, the existing Fedora Install test plan ^[1] will be revisited. The test plan will be adjusted to ensure proper test coverage for the failure scenarios listed above. Existing test cases will be reviewed for accuracy. New test cases will be created using the Template:QA/Test_Case template. Finally, the test plan will be adjusted to match the improved Fedora Release Criteria ^[2]. This includes adjusting the test case priority to match milestone criteria.

Next, in order to reduce the setup/execution time, improve efficiency and to provide test results on a more consistent basis, a subset of test cases will be chosen for automation. Tests will be written in python and will be developed and executed on a system supporting KVM virtualization. Test scripts will be responsible for preparing a virtual install environment, initiating a kickstart install and validating the results. Once an initial batch of tests exist, they will be formally integrated into the AutoQA project.

Last, a method will be developed for collecting test results into a single test result matrix. Results may be posted to the wiki directly, or a custom turbogears application may be needed to display results ^[3]. The results will be easily accessible for testers and the installer development team.

The project will be divided into several phases.

Phase#1 - proof of concept

pass Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above
pass Select a small, but representative, subset of test cases from the install test plan to automate
The following test cases are selected:
- QA:Rawhide_Acceptance_Test_Plan
- DVD.iso Installation
- Boot.iso/Netinst.iso Installation
- Live.iso Installation
- upgrade an exiting system
- system with basic video driver
- Rescue installed system
- QA:Testcase_Memtest86
pass Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results
- pass Check the virtualization environment system sanity
- pass virt-install tree and ISO media.
- pass Print out test results
pass Create python scripts to parse differenet parameters
- pass parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive, QA:Testcase_Ftp_Repository, QA:Testcase_Nfs_Repository
- pass parse different kickstart delivery: http, file, hard drive, nfs QA/TestCases/KickstartKsHttpServerKsCfg, QA/TestCases/KickstartKsFilePathKsCfg, QA/TestCases/KickstartKsHdDevicePathKsCfg, QA/TestCases/KickstartKsNfsServerPathKsCfg
Investigate methods for leveraging GUI automation to aid in automating applicable test cases
- pass Open Virt Viewer of the guest
- pass Input text into the GUI
- Close the Virt Viewer of the guest

Phase#2 - implementation

Implement the selected test cases

pass QA:Rawhide_Acceptance_Test_Plan
- pass system sanity
- pass parse arguments
- pass Tree Compose Sanity
  - pass QA:Repodata_validity_test_case
  - pass QA:Comps_Validity_Test_Case
  - pass QA:Core_package_dependency_closure_test_case
  - pass QA:Core_package_existence_test_case
- pass QA:Installer_image_presence_test_case
- pass QA:Kernel_simple_boot_test_case
- pass QA:Anaconda_stage2_fetch_test_case
- pass QA:Anaconda_storage_probe_test_case
- pass QA:Anaconda_package_install_test_case
inprogress RATS Installation
- pass system sanity
- pass parse arguments
- pass prepare kickstart
  - pass QA/TestCases/KickstartKsHttpServerKsCfg
  - pass QA/TestCases/KickstartKsFilePathKsCfg
  - none QA/TestCases/KickstartKsHdDevicePathKsCfg
  - none
- pass Tree Compose Sanity
  - pass QA:Repodata_validity_test_case
  - pass QA:Comps_Validity_Test_Case
  - pass QA:Core_package_dependency_closure_test_case
  - none QA:Core_package_existence_test_case
- pass QA:Installer_image_presence_test_case
- pass QA:Kernel_simple_boot_test_case
- pass QA:Anaconda_stage2_fetch_test_case
- pass QA:Anaconda_storage_probe_test_case
- pass QA:Anaconda_package_install_test_case
- pass Iterate different ksfile,eg. fedora,updates,updates-testing
- none Close virtio serial port
inprogress DVD.iso Installation
- pass system sanity
- pass parse arguments, （kickstart, repo, architecture are parsed）
- pass prepare kickstart
  - pass create kickstart disk on guest if kick start is local
  - pass kickstart on server (ks=http|ftp|nfs)
  - none kickstart ks=file://path
    - none compose kickstart to initrd
  - none kicstart on hard drive (ks=hd:)
  - none
- inprogress prepare repo
  - pass repo on http, ftp nfs, nfsiso server
  - none repo on cdrom
  - none repo on hd
- none download remote media to local, since checksums and repoclosure does not work with remote media
- pass QA:Testcase_Mediakit_ISO_Size
- pass QA:Testcase_Mediakit_ISO_Checksums
- pass QA:Testcase_Mediakit_Repoclosure
- pass QA:Testcase_Mediakit_FileConflicts
- create a function to report mediakit sanity status
- none QA/TestCases/BootMethodsDvd
- none QA/TestCases/InstallSourceDvd
- inprogress Iterate all ksfiles
- none Close virtio serial port
none Hard Drive Installation
- none system sanity
- none parse arguments, （kickstart, repo, architecture, image）
- none prepare kickstart
  - none create physical disk on guest to store image
  - none create kickstart disk on guest if kick start is local
  - none kickstart on server (ks=http|ftp|nfs)
  - none kickstart ks=file://path
    - none compose kickstart to initrd
  - none kicstart on hard drive (ks=hd:)
  - none
- none prepare repo
  - none repo on http, ftp nfs, nfsiso server
  - none repo on cdrom
  - none repo on hd
- none QA:Testcase_Mediakit_ISO_Size
- none QA:Testcase_Mediakit_ISO_Checksums
- none QA:Testcase_Mediakit_Repoclosure
- none QA:Testcase_Mediakit_FileConflicts
- create a function to report mediakit sanity status
- none QA/TestCases/BootMethodsDvd
- none QA/TestCases/InstallSourceDvd
- none Close virtio serial port
Boot.iso/Netinst.iso Installation
Live.iso Installation
- system sanity
- QA:Testcase_Mediakit_ISO_Size
- QA:Testcase_Mediakit_ISO_Checksums
upgrade an exiting system
- perform a default installation of the previous release
- install the current release
QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver
QA:Testcase_Anaconda_rescue_mode
Memory test - memtest86

Automate remainder of test cases from the install test plan

Rawhide Acceptance Test Plan
DVD installation
Boot.iso/netinst.iso installation
- QA:Testcase_Http_Repository
Live.iso installation
- QA:TestCases/Install_Source_Live_Image

Implement the general test cases

preupgrade
- QA:Testcase_Preupgrade
- QA:Testcase_Preupgrade_from_older_release

Create kick start database to cover test cases that can be covered with kick start

Phase#3 - integration

Identify test event triggers which will be used to automatically initiate testing
Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see Writing_AutoQA_Tests)
Develop or update AutoQA test event hooks to accommodate new test events (see Writing_AutoQA_Hooks)
Implement initial test result dashboard intended to eventually replace the wiki test matrix. The dashboard will also support FAS user test result submission. This will likely rely on

References

↑ QA:Fedora_13_Install_Test_Plan
↑ Fedora Release Criteria
↑ For a similar project see screenshot of is rawhide broken and source code and the QA:Rawhide_Acceptance_Test_Plan.

[1] QA:Fedora_13_Install_Test_Plan

[2] Fedora Release Criteria

[3] For a similar project see screenshot of is rawhide broken and source code and the QA:Rawhide_Acceptance_Test_Plan.

[1]

[2]

[3]

@@ Line 17: / Line 17: @@
 * {{result|pass|Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above}}
 * {{result|pass|Select a small, but representative, subset of test cases from the install test plan to automate}}The following test cases are selected:
-** Rawhide Acceptance Test Plan [[https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan]]
+** [[QA:Rawhide_Acceptance_Test_Plan]]
 ** DVD.iso Installation
 ** Boot.iso/Netinst.iso Installation
 ** Live.iso Installation
-** URL Installation
 ** upgrade an exiting system
 ** system with basic video driver
 ** Rescue installed system
-** Memory test [[https://fedoraproject.org/wiki/QA:Testcase_Memtest86]]
+** [[QA:Testcase_Memtest86]]
 * {{result|pass|Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results}}
-* {{result|pass|Investigate methods for leveraging GUI automation to aid in automating applicable test cases}}
+** {{result|pass|Check the virtualization environment system sanity}}
+** {{result|pass|virt-install tree and ISO media.}}
+** {{result|pass|Print out test results}}
+* {{result|pass| Create python scripts to parse differenet parameters}}
+** {{result|pass|parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive, [[QA:Testcase_Ftp_Repository]], [[QA:Testcase_Nfs_Repository]]}}
+** {{result|pass|parse different kickstart delivery: http, file, hard drive, nfs [[QA/TestCases/KickstartKsHttpServerKsCfg]], [[QA/TestCases/KickstartKsFilePathKsCfg]], [[QA/TestCases/KickstartKsHdDevicePathKsCfg]], [[QA/TestCases/KickstartKsNfsServerPathKsCfg]]}}
+* Investigate methods for leveraging GUI automation to aid in automating applicable test cases
+** {{result|pass| Open Virt Viewer of the guest}}
+** {{result|pass| Input text into the GUI}}
+** Close the Virt Viewer of the guest
 ==Phase#2 - implementation==
-*Implement the selected test cases
+===Implement the selected test cases===
-** {{result|pass| Rawhide Acceptance Test Plan [[https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan]]}}
+* {{result|pass|[[QA:Rawhide_Acceptance_Test_Plan]]}}
-** {{result|inprogress| DVD.iso Installation}}
+** {{result|pass|system sanity}}
-** {{result|inprogress|URL Installation}}
+** {{result|pass|parse arguments}}
-** Boot.iso/Netinst.iso Installation
+** {{result|pass|Tree Compose Sanity}}
-** Live.iso Installation
+*** {{result|pass|[[QA:Repodata_validity_test_case]]}}
-** upgrade an exiting system
+*** {{result|pass|[[QA:Comps_Validity_Test_Case]]}}
-** system with basic video driver
+*** {{result|pass|[[QA:Core_package_dependency_closure_test_case]]}}
-** Rescue installed system
+*** {{result|pass|[[QA:Core_package_existence_test_case]]}}
-** Memory test [[https://fedoraproject.org/wiki/QA:Testcase_Memtest86]]
+** {{result|pass|[[QA:Installer_image_presence_test_case]]}}
-*Identify and automate remainder of test cases from the install test plan
+** {{result|pass|[[QA:Kernel_simple_boot_test_case]]}}
-** Rawhide Acceptance Test Plan
+** {{result|pass|[[QA:Anaconda_stage2_fetch_test_case]]}}
-*** repodata validity
+** {{result|pass|[[QA:Anaconda_storage_probe_test_case]]}}
-*** comps.xml validity
+** {{result|pass|[[QA:Anaconda_package_install_test_case]]}}
-*** Core package dependency closure
+* {{result|inprogress|RATS Installation}}
-*** Core package existence
+** {{result|pass|system sanity}}
-*** installer image existence
+** {{result|pass|parse arguments}}
-*** Kernel boot
+** {{result|pass|prepare kickstart}}
-*** Anaconda loader fetching stage2
+*** {{result|pass|[[QA/TestCases/KickstartKsHttpServerKsCfg]]}}
-*** Anaconda stage2 disk probe
+*** {{result|pass|[[QA/TestCases/KickstartKsFilePathKsCfg]]}}
-*** Anaconda package install
+*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg]]}}
-*** Anaconda bootloader setup
+*** {{result|none|kickstart on cdrom (ks=cdrom)}}
-*** X startup/basic display configuration
+** {{result|pass|Tree Compose Sanity}}
-*** X basic input handing
+*** {{result|pass|[[QA:Repodata_validity_test_case]]}}
-*** basic network connectivity
+*** {{result|pass|[[QA:Comps_Validity_Test_Case]] }}
-*** yum update functionality
+*** {{result|pass|[[QA:Core_package_dependency_closure_test_case]]}}
-**  DVD installation
+*** {{result|none|[[QA:Core_package_existence_test_case]]}}
-*** mediakit_ISO size
+** {{result|pass|[[QA:Installer_image_presence_test_case]]}}
-*** mediakit_ISO checksums
+** {{result|pass|[[QA:Kernel_simple_boot_test_case]]}}
-*** mediakit repoclosure
+** {{result|pass|[[QA:Anaconda_stage2_fetch_test_case]]}}
-*** mediakit file conflicts
+** {{result|pass|[[QA:Anaconda_storage_probe_test_case]]}}
-*** boot methods
+** {{result|pass|[[QA:Anaconda_package_install_test_case]]}}
-*** install source
+** {{result|pass|Iterate different ksfile,eg. fedora,updates,updates-testing}}
-*** additional http repository
+**{{result|none|Close virtio serial port}}
-*** additional ftp  repository
+* {{result|inprogress| DVD.iso Installation}}
-*** additional mirrorlist repository
+** {{result|pass|system sanity}}
-*** additional nfs repository
+** {{result|pass|parse arguments, （kickstart, repo, architecture are parsed）}}
-** Boot.iso/netinst.iso installation
+** {{result|pass|prepare kickstart}}
-*** mediakit ISO size
+*** {{result|pass|create kickstart disk on guest if kick start is local}}
-*** mediakit ISO checksums
+*** {{result|pass|[[QA/TestCases/KickstartKsHttpServerKsCfg|kickstart on server (ks=http|ftp|nfs)]]}}
-*** boot methods
+*** {{result|none|[[QA/TestCases/KickstartKsFilePathKsCfg|kickstart ks=file://path]]}}
-*** install source
+**** {{result|none|compose kickstart to initrd}}
-*** http repository
+*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg|kicstart on hard drive (ks=hd:)]]}}
-** Live.iso installation
+*** {{result|none|kickstart on cdrom (ks=cdrom)}}
-*** media ISO size
+** {{result|inprogress|prepare repo}}
-*** mediakit ISO checksums
+***{{result|pass|repo on http, ftp nfs, nfsiso server}}
-*** install source live image
+***{{result|none|repo on cdrom}}
-** URL installation
+***{{result|none|repo on hd}}
-*** environment check
+** {{result|none|download remote media to local, since checksums and repoclosure does not work with remote media}}
-*** sanity check
+** {{result|pass|[[QA:Testcase_Mediakit_ISO_Size]]}}
-*** installation
+** {{result|pass|[[QA:Testcase_Mediakit_ISO_Checksums]]}}
-* Implement the general test cases
+** {{result|pass|[[QA:Testcase_Mediakit_Repoclosure]]}}
-** Anaconda user interface graphical  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Graphical]]
+** {{result|pass|[[QA:Testcase_Mediakit_FileConflicts]]}}
-** Anaconda user interface basic video driver[[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver]]
+** create a function to report mediakit sanity status
-** Anaconda user interface text  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text]]
+** {{result|none|[[QA/TestCases/BootMethodsDvd]]}}
-** Anaconda user interface VNC  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC]]
+** {{result|none|[[QA/TestCases/InstallSourceDvd]]}}
-** Anaconda user interface cmdline  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Cmdline]]
+** {{result|inprogress|Iterate all ksfiles}}
-** parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive
+**{{result|none|Close virtio serial port}}
-** parse different kickstart delivery: http, file, hard drive, nfs
+* {{result|none|Hard Drive Installation}}
-** different packages selections: default, minimal
+** {{result|none|system sanity}}
-** different partation: autopart, autopart encrypted, autopart shrink install, autopart use free space, ext4 on native device, ext3 on native device, no swap, software raid
+** {{result|none|parse arguments, （kickstart, repo, architecture, image）}}
-** rescue mode
+** {{result|none|prepare kickstart}}
-*** update.img via url installation source  local media
+*** {{result|none|create physical disk on guest to store image}}
-*** Anaconda save traceback to remote system/ bugzilla/ disk /debug mode
+*** {{result|none|create kickstart disk on guest if kick start is local}}
-** upgrade
+*** {{result|none|[[QA/TestCases/KickstartKsHttpServerKsCfg|kickstart on server (ks=http|ftp|nfs)]]}}
-*** new boot loader
+*** {{result|none|[[QA/TestCases/KickstartKsFilePathKsCfg|kickstart ks=file://path]]}}
-*** skip boot loader
+**** {{result|none|compose kickstart to initrd}}
-*** updte boot loader
+*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg|kicstart on hard drive (ks=hd:)]]}}
-*** encrypted root
+*** {{result|none|kickstart on cdrom (ks=cdrom)}}
-*** skip boot loader text mode
+** {{result|none|prepare repo}}
-*** update boot loader text mode
+***{{result|none|repo on http, ftp nfs, nfsiso server}}
+***{{result|none|repo on cdrom}}
+***{{result|none|repo on hd}}
+** {{result|none|[[QA:Testcase_Mediakit_ISO_Size]]}}
+** {{result|none|[[QA:Testcase_Mediakit_ISO_Checksums]]}}
+** {{result|none|[[QA:Testcase_Mediakit_Repoclosure]]}}
+** {{result|none|[[QA:Testcase_Mediakit_FileConflicts]]}}
+** create a function to report mediakit sanity status
+** {{result|none|[[QA/TestCases/BootMethodsDvd]]}}
+** {{result|none|[[QA/TestCases/InstallSourceDvd]]}}
+**{{result|none|Close virtio serial port}}
+* Boot.iso/Netinst.iso Installation
+** system sanity
+** [[QA:Testcase_Mediakit_ISO_Size]]
+** [[QA:Testcase_Mediakit_ISO_Checksums]]
+** [[QA/TestCases/BootMethodsBootIso]]
+** [[QA/TestCases/InstallSourceBootIso]]
+* Live.iso Installation
+** system sanity
+** [[QA:Testcase_Mediakit_ISO_Size]]
+** [[QA:Testcase_Mediakit_ISO_Checksums]]
+* [[QA:Testcase_Anaconda_Upgrade_New_Bootloader|upgrade an exiting system]]
+** perform a default installation of the previous release
+** install the current release
+* [[QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver]]
+* [[QA:Testcase_Anaconda_rescue_mode]]
+* [[QA:Testcase_Memtest86|Memory test - memtest86]]
+===Automate remainder of test cases from the install test plan===
+* Rawhide Acceptance Test Plan
+** [[QA:Anaconda_bootloader_setup_test_case]]
+** [[QA:X_basic_display_test_case]]
+** [[QA:X_basic_input_handling_test_case]]
+** [[QA:Network_basic_test_case]]
+** [[QA:Yum_simple_update_test_case]]
+*  DVD installation
+** [[QA:Testcase_Additional_Http_Repository]]
+** [[QA:Testcase_Additional_Ftp_Repository]]
+** [[QA:Testcase_Additional_Mirrorlist_Repository]]
+** [[QA:Testcase_Additional_NFS_Repository]]
+* Boot.iso/netinst.iso installation
+** [[QA:Testcase_Http_Repository]]
+* Live.iso installation
+** [[QA:TestCases/Install_Source_Live_Image]]
+===Implement the general test cases===
+* [[QA:Testcase_Anaconda_User_Interface_Cmdline]]
+* [[QA:Testcase_Anaconda_updates.img_via_URL]], [[QA:Testcase_Anaconda_updates.img_via_installation_source]], [[QA:Testcase_Anaconda_updates.img_via_local_media]]
+* [[QA:Testcase_Anaconda_save_traceback_to_remote_system]], [[QA:Testcase_Anaconda_save_traceback_to_bugzilla]], [[QA:Testcase_Anaconda_save_traceback_to_disk]], [[QA:Testcase_Anaconda_traceback_debug_mode]]
+* upgrade
+** [[QA:Testcase_Anaconda_Upgrade_New_Bootloader]]
+** [[QA:Testcase_Anaconda_Upgrade_Skip_Bootloader]]
+** [[QA:Testcase_Anaconda_Upgrade_Update_Bootloader]]
+** [[QA:Testcase_Anaconda_Upgrade_Encrypted_Root]]
+** [[QA:Testcase_Anaconda_Upgrade_Skip_Bootloader_Text_Mode]]
+** [[QA:Testcase_Anaconda_Upgrade_Update_Bootloader_Text_Mode]]
-** preupgrade
+* preupgrade
-*** preupgrade
+** [[QA:Testcase_Preupgrade]]
-*** preupgrade from older release
+** [[QA:Testcase_Preupgrade_from_older_release]]
-*Identify and integrate anaconda built-in unit tests into tests
+=== Create kick start database to cover test cases that can be covered with kick start===
-*Identify test event triggers which will be used to automatically initiate testing.
+* [[QA:Testcase_Anaconda_User_Interface_Graphical]]
-*Identify test cases where GUI automation will be required
+* [[QA:Testcase_Anaconda_User_Interface_Text]]
+* [[QA:Testcase_Anaconda_User_Interface_VNC]]
+* different packages selections: [[QA/TestCases/PackageSetsDefaultPackageInstall]], [[QA/TestCases/PackageSetsMinimalPackageInstall]]
+* different partition: [[QA:Testcase_Anaconda_autopart_install]], [[QA:Testcase_Anaconda_autopart_(encrypted)_install]], [[QA:Testcase_Anaconda_autopart_(shrink)_install]], [[QA:Testcase_Anaconda_autopart_(use_free_space)_install]], [[QA/TestCases/PartitioningExt4OnNativeDevice]], [[QA/TestCases/PartitioningExt3OnNativeDevice]], [[QA/TestCases/PartitioningXFSOnNativeDevice]], [[QA/TestCases/PartitioningBtrFSOnNativeDevice]], [[QA/TestCases/PartitioningNoSwap]], [[QA:Testcase_Partitioning_On_Software_RAID]]
+* [[QA:Testcase_Anaconda_rescue_mode]]
 ==Phase#3 - integration==
+*Identify test event triggers which will be used to automatically initiate testing
 * Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see [[Writing_AutoQA_Tests]])
 * Develop or update [[AutoQA]] test event hooks to accommodate new test events (see [[Writing_AutoQA_Hooks]])
 * Implement initial test result dashboard intended to eventually replace the wiki test matrix.  The dashboard will also support FAS user test result submission.  This will likely rely on
+== References ==
+<references/>

Search