From Fedora Project Wiki

Line 50: Line 50:
** {{result|pass|Anaconda stage2 disk probe[[]]}}
** {{result|pass|Anaconda stage2 disk probe[[]]}}
** {{result|pass|Anaconda package install[[]]}}
** {{result|pass|Anaconda package install[[]]}}
* {{result|inprogress| URL Installation}}
** {{result|inprogress|system sanity}}
** {{result|inprogress|repodata validity[[]]}}
** {{result|inprogress|comps.xml validity[[]]}}
** {{result|inprogress|Core package dependency closure[[]]}}
** {{result|inprogress|Core package existence[[]]}}
** {{result|inprogress|installer image existence[[]]}}
** {{result|inprogress|Kernel boot[[]]}}
** {{result|inprogress|Anaconda loader fetching stage2[[]]}}
** {{result|inprogress|Anaconda stage2 disk probe[[]]}}
** {{result|inprogress|Anaconda package install[[]]}}
* {{result|inprogress| DVD.iso Installation}}
* {{result|inprogress| DVD.iso Installation}}
** {{result|inprogress|system sanity}}
** {{result|inprogress|system sanity}}

Revision as of 05:19, 9 May 2011


This page provides a high-level roadmap for implementing the Is_anaconda_broken_proposal project. More detailed tasks can be found in autoqa TRAC roadmap. We follow these steps to define the methods by which we initiate testing

First, in order to provide a consistent and documented test approach, the existing Fedora Install test plan [1] will be revisited. The test plan will be adjusted to ensure proper test coverage for the failure scenarios listed above. Existing test cases will be reviewed for accuracy. New test cases will be created using the Template:QA/Test_Case template. Finally, the test plan will be adjusted to match the improved Fedora Release Criteria [2]. This includes adjusting the test case priority to match milestone criteria.

Next, in order to reduce the setup/execution time, improve efficiency and to provide test results on a more consistent basis, a subset of test cases will be chosen for automation. Tests will be written in python and will be developed and executed on a system supporting KVM virtualization. Test scripts will be responsible for preparing a virtual install environment, initiating a kickstart install and validating the results. Once an initial batch of tests exist, they will be formally integrated into the AutoQA project.

Last, a method will be developed for collecting test results into a single test result matrix. Results may be posted to the wiki directly, or a custom turbogears application may be needed to display results [3]. The results will be easily accessible for testers and the installer development team.

The project will be divided into several phases.

Phase#1 - proof of concept

  • Pass pass Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above
  • Pass pass Select a small, but representative, subset of test cases from the install test plan to automate
    The following test cases are selected:
    • Rawhide Acceptance Test Plan [[1]]
    • DVD.iso Installation
    • Boot.iso/Netinst.iso Installation
    • Live.iso Installation
    • upgrade an exiting system
    • system with basic video driver
    • Rescue installed system
    • Memory test [[2]]
  • Pass pass Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results
    • Pass pass Check the virtualization environment system sanity
    • Pass pass virt-install tree and ISO media.
    • Pass pass Print out test results
  • Pass pass Create python scripts to parse differenet parameters
    • Pass pass parse different repository: cdrom, http,ftp(anonymout, non anonymus), nfs, nfsiso, hard drive[[3]][[4]]
    • Pass pass parse different kickstart delivery: http, file, hard drive, nfs [[5]] [[6]][[7]][[8]]
  • Investigate methods for leveraging GUI automation to aid in automating applicable test cases
    • Pass pass Open Virt Viewer of the guest
    • Pass pass Input text into the GUI
    • Close the Virt Viewer of the guest

Phase#2 - implementation

Implement the selected test cases

  • Pass pass Rawhide Acceptance Test Plan [[9]]
    • Pass pass system sanity
    • Pass pass repodata validity[[10]]
    • Pass pass comps.xml validity[[11]]
    • Pass pass Core package dependency closure[[12]]
    • Pass pass Core package existence[[13]]
    • Pass pass installer image existence[[14]]
    • Pass pass Kernel boot[[15]]
    • Pass pass Anaconda loader fetching stage2[[16]]
    • Pass pass Anaconda stage2 disk probe[[17]]
    • Pass pass Anaconda package install[[18]]
  • Inprogress inprogress URL Installation
    • Inprogress inprogress system sanity
    • Inprogress inprogress repodata validity[[19]]
    • Inprogress inprogress comps.xml validity[[20]]
    • Inprogress inprogress Core package dependency closure[[21]]
    • Inprogress inprogress Core package existence[[22]]
    • Inprogress inprogress installer image existence[[23]]
    • Inprogress inprogress Kernel boot[[24]]
    • Inprogress inprogress Anaconda loader fetching stage2[[25]]
    • Inprogress inprogress Anaconda stage2 disk probe[[26]]
    • Inprogress inprogress Anaconda package install[[27]]
  • Inprogress inprogress DVD.iso Installation
    • Inprogress inprogress system sanity
    • Inprogress inprogress mediakit_ISO size[[28]]
    • Inprogress inprogress mediakit_ISO checksums[[29]]
    • Inprogress inprogress mediakit repoclosure[[30]]
    • Inprogress inprogress mediakit file conflicts[[31]]
    • Inprogress inprogress boot methods[[32]]
    • Inprogress inprogress install source[[33]]
  • Boot.iso/Netinst.iso Installation
    • system sanity
    • mediakit ISO size[[34]]
    • mediakit ISO checksums[[35]]
    • boot methods[[36]]
    • install source[[37]]
  • Live.iso Installation
    • system sanity
    • media ISO size[[38]]
    • mediakit ISO checksums[[39]]
  • upgrade an exiting system[[40]]
    • perform a default installation of the previous release
    • install the current release
  • system with basic video driver[[41]]
  • Rescue installed system[[42]]
  • Memory test [[43]]

Automate remainder of test cases from the install test plan

  • Rawhide Acceptance Test Plan
    • Anaconda bootloader setup[[44]]
    • X startup/basic display configuration[[45]]
    • X basic input handing[[46]]
    • basic network connectivity[[47]]
    • yum update functionality[[48]]
  • DVD installation
    • additional http repository[[49]]
    • additional ftp repository[[50]]
    • additional mirrorlist repository[[51]]
    • additional nfs repository[[52]]
  • Boot.iso/netinst.iso installation
    • http repository[[53]]
  • Live.iso installation
    • install source live image[[54]]

Implement the general test cases

  • Anaconda user interface cmdline [[55]]
  • update.img via url installation source local media[[56]] [[57]][[58]]
  • Anaconda save traceback to remote system/ bugzilla/ disk /debug mode [[59]][[60]][[61]][[62]]
  • upgrade
    • new boot loader[[63]]
    • skip boot loader[[64]]
    • update boot loader[[65]]
    • encrypted root[[66]]
    • skip boot loader text mode[[67]]
    • update boot loader text mode[[68]]
  • preupgrade
    • preupgrade[[69]]
    • preupgrade from older release[[70]]

Create kick start database to cover test cases that can be covered with kick start

  • Anaconda user interface graphical [[71]]
  • Anaconda user interface text [[72]]
  • Anaconda user interface VNC [[73]]
  • different packages selections: default, minimal [[74]][[75]]
  • different partation: autopart, autopart encrypted, autopart shrink install, autopart use free space, ext4 on native device, ext3 on native device, no swap, software raid
  • rescue mode

Phase#3 - integration

  • Identify test event triggers which will be used to automatically initiate testing
  • Create appropriate control files and test wrappers to allow for scheduling tests through AutoQA (see Writing_AutoQA_Tests)
  • Develop or update AutoQA test event hooks to accommodate new test events (see Writing_AutoQA_Hooks)
  • Implement initial test result dashboard intended to eventually replace the wiki test matrix. The dashboard will also support FAS user test result submission. This will likely rely on