From Fedora Project Wiki

 
(69 intermediate revisions by 2 users not shown)
Line 17: Line 17:
* {{result|pass|Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above}}
* {{result|pass|Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above}}
* {{result|pass|Select a small, but representative, subset of test cases from the install test plan to automate}}The following test cases are selected:
* {{result|pass|Select a small, but representative, subset of test cases from the install test plan to automate}}The following test cases are selected:
** Rawhide Acceptance Test Plan [[https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan]]
** [[QA:Rawhide_Acceptance_Test_Plan]]
** DVD.iso Installation
** DVD.iso Installation
** Boot.iso/Netinst.iso Installation
** Boot.iso/Netinst.iso Installation
** Live.iso Installation
** Live.iso Installation
** URL Installation
** upgrade an exiting system
** upgrade an exiting system
** system with basic video driver
** system with basic video driver
** Rescue installed system
** Rescue installed system
** Memory test [[https://fedoraproject.org/wiki/QA:Testcase_Memtest86]]
** [[QA:Testcase_Memtest86]]
* {{result|pass|Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results}}
* {{result|pass|Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results}}
* {{result|pass|Investigate methods for leveraging GUI automation to aid in automating applicable test cases}}
** {{result|pass|Check the virtualization environment system sanity}}
** {{result|pass|virt-install tree and ISO media.}}
** {{result|pass|Print out test results}}
* {{result|pass| Create python scripts to parse differenet parameters}}
** {{result|pass|parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive, [[QA:Testcase_Ftp_Repository]], [[QA:Testcase_Nfs_Repository]]}}
** {{result|pass|parse different kickstart delivery: http, file, hard drive, nfs [[QA/TestCases/KickstartKsHttpServerKsCfg]], [[QA/TestCases/KickstartKsFilePathKsCfg]], [[QA/TestCases/KickstartKsHdDevicePathKsCfg]], [[QA/TestCases/KickstartKsNfsServerPathKsCfg]]}}
* Investigate methods for leveraging GUI automation to aid in automating applicable test cases
** {{result|pass| Open Virt Viewer of the guest}}
** {{result|pass| Input text into the GUI}}
** Close the Virt Viewer of the guest


==Phase#2 - implementation==
==Phase#2 - implementation==
*Implement the selected test cases
===Implement the selected test cases===
** {{result|pass| Rawhide Acceptance Test Plan [[https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan]]}}
* {{result|pass|[[QA:Rawhide_Acceptance_Test_Plan]]}}
*** {{result|pass|repodata validity[[https://fedoraproject.org/wiki/QA:Repodata_validity_test_case]]}}
** {{result|pass|system sanity}}
*** {{result|pass|comps.xml validity[[https://fedoraproject.org/wiki/QA:Comps_Validity_Test_Case]]}}
** {{result|pass|parse arguments}}
*** {{result|pass|Core package dependency closure[[https://fedoraproject.org/wiki/QA:Core_package_dependency_closure_test_case]]}}
** {{result|pass|Tree Compose Sanity}}
*** {{result|pass|Core package existence[[https://fedoraproject.org/wiki/QA:Core_package_existence_test_case]]}}
*** {{result|pass|[[QA:Repodata_validity_test_case]]}}
*** {{result|pass|installer image existence[[https://fedoraproject.org/wiki/QA:Installer_image_presence_test_case]]}}
*** {{result|pass|[[QA:Comps_Validity_Test_Case]]}}
*** {{result|pass|Kernel boot[[https://fedoraproject.org/wiki/QA:Kernel_simple_boot_test_case]]}}
*** {{result|pass|[[QA:Core_package_dependency_closure_test_case]]}}
*** {{result|pass|Anaconda loader fetching stage2[[https://fedoraproject.org/wiki/QA:Anaconda_stage2_fetch_test_case]]}}
*** {{result|pass|[[QA:Core_package_existence_test_case]]}}
*** {{result|pass|Anaconda stage2 disk probe[[https://fedoraproject.org/wiki/QA:Anaconda_storage_probe_test_case]]}}
** {{result|pass|[[QA:Installer_image_presence_test_case]]}}
*** {{result|pass|Anaconda package install[[https://fedoraproject.org/wiki/QA:Anaconda_package_install_test_case]]}}
** {{result|pass|[[QA:Kernel_simple_boot_test_case]]}}
** {{result|inprogress| DVD.iso Installation}}
** {{result|pass|[[QA:Anaconda_stage2_fetch_test_case]]}}
*** {{result|inprogress|mediakit_ISO size[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Size]]}}
** {{result|pass|[[QA:Anaconda_storage_probe_test_case]]}}
*** {{result|inprogress|mediakit_ISO checksums[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Checksums]]}}
** {{result|pass|[[QA:Anaconda_package_install_test_case]]}}
*** {{result|inprogress|mediakit repoclosure[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_Repoclosure]]}}
* {{result|inprogress|RATS Installation}}
*** {{result|inprogress|mediakit file conflicts[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_FileConflicts]]}}
** {{result|pass|system sanity}}
*** {{result|inprogress|boot methods[[https://fedoraproject.org/wiki/QA/TestCases/BootMethodsDvd]]}}
** {{result|pass|parse arguments}}
*** {{result|inprogress|install source[[https://fedoraproject.org/wiki/QA/TestCases/InstallSourceDvd]]}}
** {{result|pass|prepare kickstart}}
** {{result|inprogress|URL Installation}}
*** {{result|pass|[[QA/TestCases/KickstartKsHttpServerKsCfg]]}}
*** repodata validity[[https://fedoraproject.org/wiki/QA:Repodata_validity_test_case]]
*** {{result|pass|[[QA/TestCases/KickstartKsFilePathKsCfg]]}}
*** comps.xml validity[[https://fedoraproject.org/wiki/QA:Comps_Validity_Test_Case]]}}
*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg]]}}
*** Core package dependency closure[[https://fedoraproject.org/wiki/QA:Core_package_dependency_closure_test_case]]
*** {{result|none|kickstart on cdrom (ks=cdrom)}}
*** Core package existence[[https://fedoraproject.org/wiki/QA:Core_package_existence_test_case]]
** {{result|pass|Tree Compose Sanity}}
*** installer image existence[[https://fedoraproject.org/wiki/QA:Installer_image_presence_test_case]]
*** {{result|pass|[[QA:Repodata_validity_test_case]]}}
*** Kernel boot[[https://fedoraproject.org/wiki/QA:Kernel_simple_boot_test_case]]
*** {{result|pass|[[QA:Comps_Validity_Test_Case]] }}
*** Anaconda loader fetching stage2[[https://fedoraproject.org/wiki/QA:Anaconda_stage2_fetch_test_case]]
*** {{result|pass|[[QA:Core_package_dependency_closure_test_case]]}}
*** Anaconda stage2 disk probe[[https://fedoraproject.org/wiki/QA:Anaconda_storage_probe_test_case]]
*** {{result|none|[[QA:Core_package_existence_test_case]]}}
*** Anaconda package install[[https://fedoraproject.org/wiki/QA:Anaconda_package_install_test_case]]
** {{result|pass|[[QA:Installer_image_presence_test_case]]}}
** Boot.iso/Netinst.iso Installation
** {{result|pass|[[QA:Kernel_simple_boot_test_case]]}}
*** mediakit ISO size[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Size]]
** {{result|pass|[[QA:Anaconda_stage2_fetch_test_case]]}}
*** mediakit ISO checksums[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Checksums]]
** {{result|pass|[[QA:Anaconda_storage_probe_test_case]]}}
*** boot methods[[https://fedoraproject.org/wiki/QA/TestCases/BootMethodsBootIso]]
** {{result|pass|[[QA:Anaconda_package_install_test_case]]}}
*** install source[[https://fedoraproject.org/wiki/QA/TestCases/InstallSourceBootIso]]
** {{result|pass|Iterate different ksfile,eg. fedora,updates,updates-testing}}
** Live.iso Installation
**{{result|none|Close virtio serial port}}
*** media ISO size[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Size]]
* {{result|inprogress| DVD.iso Installation}}
*** mediakit ISO checksums[[https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Checksums]]
** {{result|pass|system sanity}}
** upgrade an exiting system
** {{result|pass|parse arguments, (kickstart, repo, architecture are parsed)}}
** system with basic video driver
** {{result|pass|prepare kickstart}}
** Rescue installed system
*** {{result|pass|create kickstart disk on guest if kick start is local}}
** Memory test [[https://fedoraproject.org/wiki/QA:Testcase_Memtest86]]
*** {{result|pass|[[QA/TestCases/KickstartKsHttpServerKsCfg|kickstart on server (ks=http|ftp|nfs)]]}}
*Identify and automate remainder of test cases from the install test plan
*** {{result|none|[[QA/TestCases/KickstartKsFilePathKsCfg|kickstart ks=file://path]]}}
** Rawhide Acceptance Test Plan
**** {{result|none|compose kickstart to initrd}}
*** Anaconda bootloader setup[[https://fedoraproject.org/wiki/QA:Anaconda_bootloader_setup_test_case]]
*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg|kicstart on hard drive (ks=hd:)]]}}
*** X startup/basic display configuration[[https://fedoraproject.org/wiki/QA:X_basic_display_test_case]]
*** {{result|none|kickstart on cdrom (ks=cdrom)}}
*** X basic input handing[[https://fedoraproject.org/wiki/QA:X_basic_input_handling_test_case]]
** {{result|inprogress|prepare repo}}
*** basic network connectivity[[https://fedoraproject.org/wiki/QA:Network_basic_test_case]]
***{{result|pass|repo on http, ftp nfs, nfsiso server}}
*** yum update functionality[[https://fedoraproject.org/wiki/QA:Yum_simple_update_test_case]]
***{{result|none|repo on cdrom}}
** DVD installation
***{{result|none|repo on hd}}
*** additional http repository[[https://fedoraproject.org/wiki/QA:Testcase_Additional_Http_Repository]]
** {{result|none|download remote media to local, since checksums and repoclosure does not work with remote media}}
*** additional ftp  repository[[https://fedoraproject.org/wiki/QA:Testcase_Additional_Ftp_Repository]]
** {{result|pass|[[QA:Testcase_Mediakit_ISO_Size]]}}
*** additional mirrorlist repository[[https://fedoraproject.org/wiki/QA:Testcase_Additional_Mirrorlist_Repository]]
** {{result|pass|[[QA:Testcase_Mediakit_ISO_Checksums]]}}
*** additional nfs repository[[https://fedoraproject.org/wiki/QA:Testcase_Additional_NFS_Repository]]
** {{result|pass|[[QA:Testcase_Mediakit_Repoclosure]]}}
** Boot.iso/netinst.iso installation
** {{result|pass|[[QA:Testcase_Mediakit_FileConflicts]]}}
*** http repository
** create a function to report mediakit sanity status
** Live.iso installation
** {{result|none|[[QA/TestCases/BootMethodsDvd]]}}
*** install source live image[[https://fedoraproject.org/wiki/QA:TestCases/Install_Source_Live_Image]]
** {{result|none|[[QA/TestCases/InstallSourceDvd]]}}
** URL installation
** {{result|inprogress|Iterate all ksfiles}}
*** Anaconda bootloader setup[[https://fedoraproject.org/wiki/QA:Anaconda_bootloader_setup_test_case]]
**{{result|none|Close virtio serial port}}
*** X startup/basic display configuration[[https://fedoraproject.org/wiki/QA:X_basic_display_test_case]]
* {{result|none|Hard Drive Installation}}
*** X basic input handing[[https://fedoraproject.org/wiki/QA:X_basic_input_handling_test_case]]
** {{result|none|system sanity}}
*** basic network connectivity[[https://fedoraproject.org/wiki/QA:Network_basic_test_case]]
** {{result|none|parse arguments, (kickstart, repo, architecture, image)}}
*** yum update functionality[[https://fedoraproject.org/wiki/QA:Yum_simple_update_test_case]]
** {{result|none|prepare kickstart}}
* Implement the general test cases
*** {{result|none|create physical disk on guest to store image}}
** Anaconda user interface graphical  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Graphical]]
*** {{result|none|create kickstart disk on guest if kick start is local}}
** Anaconda user interface basic video driver[[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver]]
*** {{result|none|[[QA/TestCases/KickstartKsHttpServerKsCfg|kickstart on server (ks=http|ftp|nfs)]]}}
** Anaconda user interface text  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text]]
*** {{result|none|[[QA/TestCases/KickstartKsFilePathKsCfg|kickstart ks=file://path]]}}
** Anaconda user interface VNC  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC]]
**** {{result|none|compose kickstart to initrd}}
** Anaconda user interface cmdline  [[https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Cmdline]]
*** {{result|none|[[QA/TestCases/KickstartKsHdDevicePathKsCfg|kicstart on hard drive (ks=hd:)]]}}
** parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive
*** {{result|none|kickstart on cdrom (ks=cdrom)}}
** parse different kickstart delivery: http, file, hard drive, nfs
** {{result|none|prepare repo}}
** different packages selections: default, minimal
***{{result|none|repo on http, ftp nfs, nfsiso server}}
** different partation: autopart, autopart encrypted, autopart shrink install, autopart use free space, ext4 on native device, ext3 on native device, no swap, software raid
***{{result|none|repo on cdrom}}
** rescue mode
***{{result|none|repo on hd}}
*** update.img via url installation source  local media
** {{result|none|[[QA:Testcase_Mediakit_ISO_Size]]}}
*** Anaconda save traceback to remote system/ bugzilla/ disk /debug mode
** {{result|none|[[QA:Testcase_Mediakit_ISO_Checksums]]}}
** upgrade
** {{result|none|[[QA:Testcase_Mediakit_Repoclosure]]}}
*** new boot loader
** {{result|none|[[QA:Testcase_Mediakit_FileConflicts]]}}
*** skip boot loader
** create a function to report mediakit sanity status
*** updte boot loader
** {{result|none|[[QA/TestCases/BootMethodsDvd]]}}
*** encrypted root
** {{result|none|[[QA/TestCases/InstallSourceDvd]]}}
*** skip boot loader text mode
**{{result|none|Close virtio serial port}}
*** update boot loader text mode
* Boot.iso/Netinst.iso Installation
** system sanity
** [[QA:Testcase_Mediakit_ISO_Size]]
** [[QA:Testcase_Mediakit_ISO_Checksums]]
** [[QA/TestCases/BootMethodsBootIso]]
** [[QA/TestCases/InstallSourceBootIso]]
* Live.iso Installation
** system sanity
** [[QA:Testcase_Mediakit_ISO_Size]]
** [[QA:Testcase_Mediakit_ISO_Checksums]]
* [[QA:Testcase_Anaconda_Upgrade_New_Bootloader|upgrade an exiting system]]
** perform a default installation of the previous release
** install the current release
* [[QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver]]
* [[QA:Testcase_Anaconda_rescue_mode]]
* [[QA:Testcase_Memtest86|Memory test - memtest86]]
 
===Automate remainder of test cases from the install test plan===
* Rawhide Acceptance Test Plan
** [[QA:Anaconda_bootloader_setup_test_case]]
** [[QA:X_basic_display_test_case]]
** [[QA:X_basic_input_handling_test_case]]
** [[QA:Network_basic_test_case]]
** [[QA:Yum_simple_update_test_case]]
*  DVD installation
** [[QA:Testcase_Additional_Http_Repository]]
** [[QA:Testcase_Additional_Ftp_Repository]]
** [[QA:Testcase_Additional_Mirrorlist_Repository]]
** [[QA:Testcase_Additional_NFS_Repository]]
* Boot.iso/netinst.iso installation
** [[QA:Testcase_Http_Repository]]
* Live.iso installation
** [[QA:TestCases/Install_Source_Live_Image]]
 
===Implement the general test cases===
* [[QA:Testcase_Anaconda_User_Interface_Cmdline]]
* [[QA:Testcase_Anaconda_updates.img_via_URL]], [[QA:Testcase_Anaconda_updates.img_via_installation_source]], [[QA:Testcase_Anaconda_updates.img_via_local_media]]
* [[QA:Testcase_Anaconda_save_traceback_to_remote_system]], [[QA:Testcase_Anaconda_save_traceback_to_bugzilla]], [[QA:Testcase_Anaconda_save_traceback_to_disk]], [[QA:Testcase_Anaconda_traceback_debug_mode]]
* upgrade
** [[QA:Testcase_Anaconda_Upgrade_New_Bootloader]]
** [[QA:Testcase_Anaconda_Upgrade_Skip_Bootloader]]
** [[QA:Testcase_Anaconda_Upgrade_Update_Bootloader]]
** [[QA:Testcase_Anaconda_Upgrade_Encrypted_Root]]
** [[QA:Testcase_Anaconda_Upgrade_Skip_Bootloader_Text_Mode]]
** [[QA:Testcase_Anaconda_Upgrade_Update_Bootloader_Text_Mode]]


** preupgrade
* preupgrade
*** preupgrade
** [[QA:Testcase_Preupgrade]]
*** preupgrade from older release
** [[QA:Testcase_Preupgrade_from_older_release]]


*Identify and integrate anaconda built-in unit tests into tests
=== Create kick start database to cover test cases that can be covered with kick start===
*Identify test event triggers which will be used to automatically initiate testing.
* [[QA:Testcase_Anaconda_User_Interface_Graphical]]
*Identify test cases where GUI automation will be required
* [[QA:Testcase_Anaconda_User_Interface_Text]]
* [[QA:Testcase_Anaconda_User_Interface_VNC]]
* different packages selections: [[QA/TestCases/PackageSetsDefaultPackageInstall]], [[QA/TestCases/PackageSetsMinimalPackageInstall]]
* different partition: [[QA:Testcase_Anaconda_autopart_install]], [[QA:Testcase_Anaconda_autopart_(encrypted)_install]], [[QA:Testcase_Anaconda_autopart_(shrink)_install]], [[QA:Testcase_Anaconda_autopart_(use_free_space)_install]], [[QA/TestCases/PartitioningExt4OnNativeDevice]], [[QA/TestCases/PartitioningExt3OnNativeDevice]], [[QA/TestCases/PartitioningXFSOnNativeDevice]], [[QA/TestCases/PartitioningBtrFSOnNativeDevice]], [[QA/TestCases/PartitioningNoSwap]], [[QA:Testcase_Partitioning_On_Software_RAID]]
* [[QA:Testcase_Anaconda_rescue_mode]]


==Phase#3 - integration==
==Phase#3 - integration==
*Identify test event triggers which will be used to automatically initiate testing
* Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see [[Writing_AutoQA_Tests]])
* Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see [[Writing_AutoQA_Tests]])
* Develop or update [[AutoQA]] test event hooks to accommodate new test events (see [[Writing_AutoQA_Hooks]])
* Develop or update [[AutoQA]] test event hooks to accommodate new test events (see [[Writing_AutoQA_Hooks]])
* Implement initial test result dashboard intended to eventually replace the wiki test matrix.  The dashboard will also support FAS user test result submission.  This will likely rely on
* Implement initial test result dashboard intended to eventually replace the wiki test matrix.  The dashboard will also support FAS user test result submission.  This will likely rely on
== References ==
<references/>

Latest revision as of 07:02, 1 June 2012

QA.png


This page provides a high-level roadmap for implementing the Is_anaconda_broken_proposal project. More detailed tasks can be found in autoqa TRAC roadmap. We follow these steps to define the methods by which we initiate testing


First, in order to provide a consistent and documented test approach, the existing Fedora Install test plan [1] will be revisited. The test plan will be adjusted to ensure proper test coverage for the failure scenarios listed above. Existing test cases will be reviewed for accuracy. New test cases will be created using the Template:QA/Test_Case template. Finally, the test plan will be adjusted to match the improved Fedora Release Criteria [2]. This includes adjusting the test case priority to match milestone criteria.

Next, in order to reduce the setup/execution time, improve efficiency and to provide test results on a more consistent basis, a subset of test cases will be chosen for automation. Tests will be written in python and will be developed and executed on a system supporting KVM virtualization. Test scripts will be responsible for preparing a virtual install environment, initiating a kickstart install and validating the results. Once an initial batch of tests exist, they will be formally integrated into the AutoQA project.

Last, a method will be developed for collecting test results into a single test result matrix. Results may be posted to the wiki directly, or a custom turbogears application may be needed to display results [3]. The results will be easily accessible for testers and the installer development team.


The project will be divided into several phases.

Phase#1 - proof of concept

  • Pass pass Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above
  • Pass pass Select a small, but representative, subset of test cases from the install test plan to automate
    The following test cases are selected:
  • Pass pass Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results
    • Pass pass Check the virtualization environment system sanity
    • Pass pass virt-install tree and ISO media.
    • Pass pass Print out test results
  • Pass pass Create python scripts to parse differenet parameters
  • Investigate methods for leveraging GUI automation to aid in automating applicable test cases
    • Pass pass Open Virt Viewer of the guest
    • Pass pass Input text into the GUI
    • Close the Virt Viewer of the guest

Phase#2 - implementation

Implement the selected test cases

Automate remainder of test cases from the install test plan

Implement the general test cases

Create kick start database to cover test cases that can be covered with kick start

Phase#3 - integration

  • Identify test event triggers which will be used to automatically initiate testing
  • Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see Writing_AutoQA_Tests)
  • Develop or update AutoQA test event hooks to accommodate new test events (see Writing_AutoQA_Hooks)
  • Implement initial test result dashboard intended to eventually replace the wiki test matrix. The dashboard will also support FAS user test result submission. This will likely rely on

References