From Fedora Project Wiki
(Updated Ansible staging and invocation instructions: Using Ansible variables, and vary names of playbooks based on test subjects. Discussed with Pingou Ari, and Martin.)
Line 11: Line 11:


This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow Fedora packagers to locally and manually invoke a ''test suite''.
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow Fedora packagers to locally and manually invoke a ''test suite''.
'''First see the [https://fedoraproject.org/wiki/Changes/InvokingTests#Terminology Terminogy] division of [https://fedoraproject.org/wiki/Changes/InvokingTests#Terminology Responsibilities] and [https://fedoraproject.org/wiki/Changes/InvokingTests#Requirements Requirements]'''


=== Staging ===
=== Staging ===


Tests files will be added into the ''tests'' folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there should be a ''run_tests.yml'' playbook at the top level of the tests folder to set up and run all the tests.
Tests files will be added into the <code>tests/</code> folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the <code>tests/</code> folder that can be invoked to run the test suites.


# The ''testing system'' should stage the tests on Fedora operating system appropriate for the branch name of the dist-git repository containing the tests.
# The ''testing system'' SHOULD stage the tests on Fedora operating system appropriate for the branch name of the dist-git repository containing the tests.
# The ''testing system'' must have installed the following packages:
# The ''testing system'' MUST stage the following packages:
## <code>ansible python2-dnf libselinux-python</code>
## <code>ansible python2-dnf libselinux-python</code>
# The ''testing system'' clones the dist-git repository for the test, and checks out the appropriate branch.
# The ''testing system'' MUST clone the dist-git repository for the test, and checks out the appropriate branch.
# The contents of /etc/yum.repos.d is replaced with repository information that reflects the known good Fedora packages the test may require.
# The contents of <code>/etc/yum.repos.d</code> on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
## The ''testing system'' may use multiple repositories, including ''updates'' or ''updates-testing'' to ensure this.
## The ''testing system'' MAY use multiple repositories, including ''updates'' or ''updates-testing'' to ensure this.


=== Invocation ===
=== Invocation ===


The test can be invoke simply by calling <code>ansible-playbook run_tests.yml</code> on the ''run_tests.yml'' playbook of interest.
The ''testing system'' MUST select a playbooks in the <code>tests/</code> folder depending on the type of ''test subject'' it would like to test. The filename of each of these playbooks start with the <code>test_</code> prefix and ends with a <code>.yml</code> extension. The following well known playbooks correspond to common ''test subjects''. Additional playbooks will be added to this list as additional ''test subjects'' become common:
 
{|
! Playbook invoked !! Test subject
|-
| test_rpm.yml    || A string containing a space separated list of rpm filenames
|-
| test_repo.yml   || A string containing a space separated list of repo filenames appropriate for <code>/etc/yum.repos.d</code>
|-
| test_cloud.yml  || A string containing the filename of one virtual machine disk image bootable with cloud-init
|-
| test_oci.yml || A string containing the filename of one OCI container image filesystem bundle
|-
| test_local.yml  || An empty string. No test subject or installation.
|}
 
If a playbook for a given ''test subject'' is not present in a dist-git repository, the ''testing system'' SHOULD treat the test as having been "skipped". That is, the invocation SHOULD neither pass nor fail.


To invoke the test suite, one would:
The <code>test_local.yml</code> SHOULD test a booted system where the test suite, its framework, and test subject are already installed. This playbook is usually invoked by the other playbooks. Additional playbooks may be present in the <code>tests/</code> folder, and these MAY represent multiple test suites. The testing system is not expected to be aware of these additional playbooks.


# Create a folder and place the ''test subject(s)'' in that folder.
To invoke the selected playbook, the ''testing system'':
# Set an enivonment variable <code>$TEST_SUBJECT_FOLDER</code> to the path of the folder.
 
# Execute the following command <code>ansible-playbook run-tests.yml</code>
# MUST execute the playbook locally with <code>ansible_connection=local</code> and host <code>localhost</code>
## The command is invoked as root, and the roles/modules may drop privileges as desired
# MUST execute the playbook with the following variables.
## Examine the exit code of each ansible-playbook command. Zero exit code is a successful ''test result'', non-zero is failure.
## <code>subjects</code>: The ''test subjects'' string as described above
## The ansible playbook must put the output of the tests into a file named <code>output</code> in the current directory.
## <code>artifacts</code>: The full path of an empty folder for ''test artifacts''
# Tests can put any additional ''test artifacts'' like screenshots, core dumps, into the current directory.
# MUST execute the playbook as root.
# MUST examine the exit code of the playbook. A zero exit code is successful ''test result'', non-zero is failure.
# MUST treat the file <code>test.log</code> in the <code>artifacts</code> folder as the main readable output of the test.
# SHOULD place the textual stdout/stderr of the <code>ansible-playbook</code> command in the <code>ansible.log</code> file in the <code>artifacts</code> folder.
# SHOULD treat the contents of the <code>artifacts</code> folder as the ''test artifacts''.
 
The playbook and its ''test suite'' or ''test framework'':
 
# SHOULD drop privileges appropriately if the ''test suite'' should be run as non-root.
# MUST install any requirements of its ''test suite'' or ''test framework'' and MUST fail if this is not possible.
# MUST provision the ''test subject'' listed in the <code>subjects</code> variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
# MUST place the main readable output of the ''test suite'' into a <code>test.log</code> file in the <code>artifacts</code> variable folder. This MUST happen even if some of the test suites fail.
# SHOULD place additional ''test artifacts'' in the folder defined in the <code>artifacts</code> variable.


=== Discovery ===
=== Discovery ===


A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.
A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.


Use repoquery, basically I propose we rely on the dependency chain of the
Use repoquery, basically I propose we rely on the dependency chain of the
Line 186: Line 215:
         │  │  └── vars
         │  │  └── vars
         └── configure.yml
         └── configure.yml
         └── run_tests.yml
         └── test_rpm.yml
   
   


Tests will live under tests directory in a dist-git repo.  The playbooks directory will define the roles for configuration and execution of the tests.
Tests will live under tests directory in a dist-git repo.  The playbooks directory will define the roles for configuration and execution of the tests.
The run_tests.yml will call roles necessary and dependencies of other roles can be defined there or in the meta of another role. (Well documented on writing ansible playbooks)
The test_rpm.yml will call roles necessary and dependencies of other roles can be defined there or in the meta of another role. (Well documented on writing ansible playbooks)
I put the config as a place holder for configuration files needed or for provisioning (thinking of linch-pin https://github.com/CentOS-PaaS-SIG/linch-pin)
I put the config as a place holder for configuration files needed or for provisioning (thinking of linch-pin https://github.com/CentOS-PaaS-SIG/linch-pin)
''Note :This does not mean all these role sub-directories are required this just shows a full example case''
''Note :This does not mean all these role sub-directories are required this just shows a full example case''

Revision as of 08:11, 19 April 2017

Ansible: Standard Discovery, Staging, Invocation of Integration Tests

Warning.png
This is a proposal
Feedback is more than welcome. There's a discussion tab above.

First see the Terminogy division of Responsibilities and Requirements

Invoking-tests-standard-interface.png

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.

First see the Terminogy division of Responsibilities and Requirements

Staging

Tests files will be added into the tests/ folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the tests/ folder that can be invoked to run the test suites.

  1. The testing system SHOULD stage the tests on Fedora operating system appropriate for the branch name of the dist-git repository containing the tests.
  2. The testing system MUST stage the following packages:
    1. ansible python2-dnf libselinux-python
  3. The testing system MUST clone the dist-git repository for the test, and checks out the appropriate branch.
  4. The contents of /etc/yum.repos.d on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
    1. The testing system MAY use multiple repositories, including updates or updates-testing to ensure this.

Invocation

The testing system MUST select a playbooks in the tests/ folder depending on the type of test subject it would like to test. The filename of each of these playbooks start with the test_ prefix and ends with a .yml extension. The following well known playbooks correspond to common test subjects. Additional playbooks will be added to this list as additional test subjects become common:

Playbook invoked Test subject
test_rpm.yml A string containing a space separated list of rpm filenames
test_repo.yml A string containing a space separated list of repo filenames appropriate for /etc/yum.repos.d
test_cloud.yml A string containing the filename of one virtual machine disk image bootable with cloud-init
test_oci.yml A string containing the filename of one OCI container image filesystem bundle
test_local.yml An empty string. No test subject or installation.

If a playbook for a given test subject is not present in a dist-git repository, the testing system SHOULD treat the test as having been "skipped". That is, the invocation SHOULD neither pass nor fail.

The test_local.yml SHOULD test a booted system where the test suite, its framework, and test subject are already installed. This playbook is usually invoked by the other playbooks. Additional playbooks may be present in the tests/ folder, and these MAY represent multiple test suites. The testing system is not expected to be aware of these additional playbooks.

To invoke the selected playbook, the testing system:

  1. MUST execute the playbook locally with ansible_connection=local and host localhost
  2. MUST execute the playbook with the following variables.
    1. subjects: The test subjects string as described above
    2. artifacts: The full path of an empty folder for test artifacts
  3. MUST execute the playbook as root.
  4. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.
  5. MUST treat the file test.log in the artifacts folder as the main readable output of the test.
  6. SHOULD place the textual stdout/stderr of the ansible-playbook command in the ansible.log file in the artifacts folder.
  7. SHOULD treat the contents of the artifacts folder as the test artifacts.

The playbook and its test suite or test framework:

  1. SHOULD drop privileges appropriately if the test suite should be run as non-root.
  2. MUST install any requirements of its test suite or test framework and MUST fail if this is not possible.
  3. MUST provision the test subject listed in the subjects variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
  4. MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
  5. SHOULD place additional test artifacts in the folder defined in the artifacts variable.

Discovery

A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.

Use repoquery, basically I propose we rely on the dependency chain of the RPMs itself instead of trying to replicate it differently.

repoquery --whatrequires or an equivalent relying on mdapi: https://apps.fedoraproject.org/mdapi/ (which I need to adjust to support back walking (ie find which packages requires "foo" instead of what packages "foo" requires which we currently have) and we should be able to build a list of dependencies.

In addition, a test suite can be uniquely identified using the git hash of the commit of the git repo.

Scope

Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.

Only the testing system will need to be taught to install the requirements and run the playbooks.

User Experience

A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.

  • This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
  • Ansible is being more and more popular, thus making it easier for people to contribute new tests
  • Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.

Upgrade/compatibility impact

There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.


Examples

What follows are examples of writing and/or packaging existing tests to this standard.

TODO: Put general example notes here.

Example: Simple in-situ test

A simple downstream integration test for gzip can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip

This is how the folder structure looks like:

 .
 ├── files
 │   └── test-simple
 └── run_tests.yml

And the content of run_tests.yml is:

---
- hosts: localhost
  remote_user: root
  tasks:
  - name: Install the requirements
    package: 
      name: "{{item}}" 
      state: latest
    with_items:
      - coreutils
      - /sbin/install-info
      - gzip
 
  - name: Create the folder where we will store the tests
    file: 
      state: directory
      path: "{{ item }}"
      owner: root 
      group: root
    with_items:
     - /usr/libexec/tests/gzip/

  - name: Install the test files
    copy: 
      src: "{{ item.file }}"
      dest: "/usr/libexec/tests/gzip/{{ item.dest }}"
      mode: 0755
    with_items:
     - {file: test-simple, dest: test-simple }

  - name: Execute the tests
    shell: /usr/libexec/tests/gzip/test-simple 2>&1 | tee output

Example: GNOME style "Installed Tests"

A downstream integration test running in gnome installed tests can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip full example structure: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip/playbooks

Example: Tests run in Docker Container

An integration test running tests in a docker container can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/glib2 full example structure: https://pagure.io/ansible_based_tests/blob/master/f/tests/glib2/playbooks

Example: Modularity testing Framework

TODO: Port an example

Example: Ansible with Atomic Host

TODO: Port an existing test

Example: Beakerlib based test

Beakerlib test of 'setup' package:

Beakerlib test of 'coreutils' package:

Beakerlib tests of 'sed' package:

Example: Full Structure

 .
 └── tests
    └── test-case
    └── config
    └── playbooks
        └── group_vars
        └── roles
        │   └── configure
        │   │   └── defaults
        │   │   └── files
        │   │   └── handlers
        │   │   └── meta
        │   │   └── tasks
        │   │   └── templates
        │   │   └── vars
        │   └── run_tests
        │   │   └── defaults
        │   │   └── files
        │   │   └── handlers
        │   │   └── meta
        │   │   └── tasks
        │   │   └── templates
        │   │   └── vars
        └── configure.yml
        └── test_rpm.yml

Tests will live under tests directory in a dist-git repo. The playbooks directory will define the roles for configuration and execution of the tests. The test_rpm.yml will call roles necessary and dependencies of other roles can be defined there or in the meta of another role. (Well documented on writing ansible playbooks) I put the config as a place holder for configuration files needed or for provisioning (thinking of linch-pin https://github.com/CentOS-PaaS-SIG/linch-pin) Note :This does not mean all these role sub-directories are required this just shows a full example case

Evaluation

Instructions: Copy the block below, sign your name and fill in each section with your evaluation of that aspect. Add additional bullet points with overall summary or notes.

Full Name -- SignAture

  • Summary: ...
  • Staging: ...
  • Invocation: ...
  • Discovery: ...

Stef Walter -- Stefw

  • Summary:
    • PRO: Ansible is readable and approachable
    • PRO: Tests are stored in same repo as tests
    • PRO: Inclusion of upstream tests seems to require packaging them as RPMs.
    • CON: Ansible is another technology (in addition to RPM spec files, etc.) that packager is required to learn in order to maintain a package in dist-git.
    • CON: If tests become a core Fedora concept (which we hope), Ansible becomes a core technology that Fedora requires and is built upon.
    • CON: Most Ansible modules require Python 2.x while the distro is trying to move to Python 3.x
    • CON: No standard mechanism for passing a test subject to a test suite implementing the standard test interface
    • CON: No standard mechanism for reporting test log, or test artifacts from standard interface
    • CON: No way to describe whether tests are compatible with or conflict with specific NVR of test subjects.
  • Staging:
    • No mechanism for passing a test subject (eg: a built package, a module, or a container) to the test suite to operate on.
    • No guidance on what Ansible modules should be used to install test dependencies
    • No mechanism for a test system to control which repo of known-good packages to pull test or test suite dependencies from.
    • Requires sudo, dnf, git, ansible, python2-dnf, libselinux-python as well known staging dependencies
  • Invocation:
    • Seems that zero exit code from sudo means success, non-zero exit code means failure? Not described explicitly in standard.
    • The use of sudo seems to imply invocation should happen as a non-root user. Is this correct?
    • Does the standard assume sudo is guaranteed to work? Should the sudo part just be dropped and require invocation as root?
    • No mechanism for reporting logs, or test artifacts has been described.
  • Discovery:
    • Mechanism is simple, but no concrete description of how exactly this works. How does a testing system find tests given a test subject such as an RPM or NVR?
    • MDAPI link is broken: https://apps.fedoraproject.org/mdapi/
      • This has been fixed --Pingou (talk) 08:03, 12 April 2017 (UTC)

Martin Pitt -- mpitt

  • Summary:
    • I agree to what Stef said above, so I just add my "delta" review.
    • PRO: I prefer keeping tests in the sources (like in this proposal) over packaging tests, as it's much less overhead for the packager and avoids having to create a new kind of package archive.
    • CON: My main concern is that the Ansible format/tool might be replaced with something else in a few years, but the test format should be stable for a long time to avoid having to port hundreds/thousands of tests.
    • CON: The ansible format is relatively verbose and too procedural for my taste; I prefer a purely declarative syntax and avoiding boilerplate for installing test deps and invoking the tests.
  • Staging:
    • Not supporting test subjects is a major gap in the prototype - this is one of the core requirements here!
    • Installing the actual tests is unnecessary overhead in the playbook, and clutters the host system with files in /usr that don't belong to a package; this can be rectified though with dropping the "Create folder"/"Install" tasks and replacing the run part with
- name: Execute the tests
  script: files/test-simple
  • Invocation:
    • Getting live logs from the test and also saving it as an artifact is crucial, this is a major gap in the prototype. Can ansible do this somehow?
  • Discovery:
    • Checking out and inspecting hundreds/thousands of dist-gits whether they contain tests does not meet "able to efficiently answer the question..."; this needs a new service which regularly indexes all dist-gits and creates list of source packages that have tests.


Pierre-Yves Chibon -- pingou

  • Disclaimer: I am one of the owners above.
  • Summary:
    • PRO: Ansible is a well-know technology for sys-admin making it easier for them to contribute tests
    • CON: While being well-know for some people, it will be new for others
    • PRO: Very flexible it gives the packagers all the flexibility to install/configure/run their tests as they wish
    • PRO: We could use --tag to allow running just a part of the test suite at certain time (-t PR to run on pull-request -t updates to run on bodhi updates...)
    • CON: We may need to "regulate" the flexibility to suggest a set of standard/gold practices to be used in the test system (using different tags or playbook if we want)
  • Staging:
    • PRO: its flexibility makes it easy to test anything
    • CON: we will need to write policies/guidelines on how to test the different subject (RPM, container, images...)
  • Invocation:
    • PRO: easy to run locally
    • PRO: easy to run as root and switch to a local user or vice-versa
    • PRO: easy to couple with something like vagrant to allow running locally destructive tests
    • CON: May require policy to set expectations and document how to move from one to the other
    • CON: Inter-package dependencies is a challenge that can be overcome with a custom ansible module allowing to git clone other dist-git repo and while allowing us to block other network accesses (to avoid downloading random things from the internet that may be gone tomorrow and thus kill the reproducibility aspect).
  • Discovery:
    • Git hash uniquely identifies a test suite
      • Meaning the identifier may change while the test suite itself hasn't
    • PRO: Relies on the same dependency chain as the artefacts themselves
    • QUESTION: What is the aim here? Do we really want to run all the tests of every perl module for every change made to the perl package? If so, good luck, otherwise repoquery --whatrequires <pkg> should do what we want.
      • MartinPitt: That's what Debian/Ubuntu do, and indeed that triggers thousands of tests (times 5 architectures). This allows landing new Perl versions with confidence and points out modules that need to be adjusted (and believe me, pretty much every new Perl version breaks some module or two!). That said, it should be possible to discover tests for that reason - I don't expect our infra to be scalable and fast enough right from the start to actually do testing at that depth.


Tim Flink -- Tflink

  • Disclaimer: I am one of the owners of this proposal
  • Summary:
    • PRO: Storing tests in this way decouples them from the build process
    • PRO: Ansible has better docs and more examples than Fedora packages or RPM do
    • PRO: non-packager testers don't have to learn RPM syntax
    • PRO: Able to provide a lot more in the way of convenience functions to the test author - galaxy, roles/modules that we provide
    • PRO: easy to change tests during devel, does not require a dedicated path in the filesystem
    • PRO/CON: More easily extendable
    • CON: Adds ansible et. al as a dependency for the test process - what happens if ansible changes or if it becomes unattractive 5-10 years from now?
    • CON: Adds additional thing that packagers have to learn
    • CON: We would have no control over when/how ansible changes
    • It's not incredibly clear what all would be distributed (ansible modules, plugins) or how those would be distributed (galaxy-ish, package, etc.)
  • Staging:
    • There is no obvious way to say what NVR is under test other than looking at what's installed or what's locally available pre-build
  • Invocation:
    • Not sure sudo is required, it would likely be easier to have a plugin (if required) that ran things in a temp dir kind of how we do with libtaskotron today
  • Discovery:
    • While arguably more complex than the -tests package proposal, the additional complexity in terms of code to be written doesn't seem to be much more complex
    • There are systems already doing some parts of this discovery and could likely be re-used to a certain extent (Taskotron's trigger)

Dennis Gilmore -- Ausil

  • Summary:
    • PRO: we could have unique git repos for collections, gnome-desktop, KDE, Atomic Host, Server, etc
    • PRO: Docs are good as is support ofr the format across platforms
    • PRO: Branching could be separate from package branching, simplifying workflows
      • I believe the idea is to store the tests in dist-git next to the spec files and patches, so branching would be at the same time --Pingou (talk) 08:03, 12 April 2017 (UTC)
    • PRO: should be simple to write validation testing of tests, making sure that people are in compliance.
    • CON: Not clear how we should store tests for same package with different git namespaces. for example Cockpit rpm and cockpit container
      • If they are stored in dist-git the tests for the rpm would be stored next to the spec file and the tests for the container next to the Dockerfile or equivalent --Pingou (talk) 08:03, 12 April 2017 (UTC)
    • CON: getting started with Ansible for those who do not now it is a steep learning curve
    • CON: can not reuse tools like rpmlint, rpmdiff etc
      • Could you expand on why? I don't see anything preventing using these tools. --Pingou (talk) 08:03, 12 April 2017 (UTC)
    • PRO: seems like we should be able to easily setup a template for a tests repo
    • PRO: We should be able to easily put a web interface for adding and editing tests for people not familiar with git
  • Staging:
    • Using VM's and containers seems to have a much clearer path than the -tests package proposal
  • Invocation:
    • use of sudo seems very suboptimal.
  • Discovery:
    • indexing, searching and mapping of tests seems uncovered. Likely we will need to write some tooling to make it useful and easy to find and get for people.


Micah Abbott -- miabbott

  • Summary:
    • Disclaimer: I was pulled into this evaluation later in the game and may be missing some context/pieces of the larger effort.
    • PRO: Ansible feels easier to read/understand/learn
    • PRO: Ansible appears to give more flexibility and options to packagers
    • CON: New requirement on Ansible; not a standard install option like rpm/yum/dnf
    • CON: Easy to do bad things with Ansible + root user
  • Staging:
    • Using Ansible here seems to better support the in-situ and outside-in test approaches. There may still be the issue of multiple, conflicting provisioning solutions.
  • Invocation:
    • Using root has risks, although widely used when running Ansible playbooks.
  • Discovery:
    • Using reqoquery seems reasonable enough, although I'd like to see a more concrete example of the whole process.


Dusty Mabe -- dustymabe

  • Summary: ...
    • I think ansible gives a balance of simple & sophisticated tooling to enable us to write simple tests or write complex tests. If a user is not familiar with ansible then they can use an example yaml file to just execute a shell script. More advanced users can ramp up to ansible's potential.
  • Staging:
    • PRO: storing tests in git and not needing to repacking them into an RPM.
  • Invocation:
    • PRO: simple: can invoke test by cloning repo and running run_tests.yml
  • Discovery:
    • CON: not quite sure how this discovery is going to work. are we baking in the rpm some meta about where the tests live?


Nick Coghlan -- Ncoghlan

  • Summary:
    • Ansible offers a lot more flexibility than RPM in managing complex test resources (VMs, users, etc), as well as installing test dependencies that aren't themselves packaged as RPMs
    • However, it's likely to be overkill for simple projects that just need to re-run their standard tests on a fully installed system
    • Regardless of which option is chosen, a standard shim should be provided to bootstrap the other (so if using packaged tests, have a boilerplate *-tests subpackage definition that bootstraps an Ansible based test)
    • With RPM, two boilerplate templates could be provided: one for running a shell script from the source package, one for running an Ansible playbook from dist-git
  • Staging:
    • CON: An Ansible-only approach introduces additional complexity in running non-intrusive test suites directly on the current system
  • Invocation:
    • CON: Spec file helpers can assist in defining test package definitions, but they'd need to be dist-git aware to help define out-of-band test cases
  • Discovery:
  • CON: Requiring additional metadata outside the RPM database for integration test discovery makes it more difficult to share tests across distributions