The Test Contract

As I've said elsewhere in the blog, regression tests are by definition deterministic.  Automated regression tests are just regression tests whose algorithms have been automated.  Their most fundamental attribute is that they must be easy to execute.  Accordingly, they should follow a standard execution synopsis, a 'Test Contract'.  We establish these rules because we know we will want to execute our tests automatically without custom plumbing.

If we can't do this, we haven't accomplished that goal:

for test in *
do
    $test
done

I recommend these rules for CLI based automated testing with Perl:

I. Inputs

You must resist the temptation to give-in to the automation developer's desire to add arguments to the standard synopsis.  Fully automated regression tests don't have variables.  They have specifications.  Variables are an indication that you don't have a complete test specification.  Here's the synopsis I recommend:

  test.pl -cfg
  test.pl -help
 
The testbed configuration file specifies the testbed on which the test is to be executed.  It tells the test the names of the hosts it should talk to.  Again, refrain from polluting the configuration file with unnecessary attributes.  Here's an example of what this should look like, in Perl.  Do yourself a favor, write this in eval-able Perl like this:

$Testbed = [
    node => {
        role => 'master',
        type => 'physical',
        ostype => 'linux',
        osversion => 'Centos 5.2',
        interface => {
            eth0 => '10.10.23.223',
            eth1 => '10.10.3.8'
        }
    }
];

II. Outputs

Print anything you want to be logged to standard out.

III. Script Exit Code

  0 = PASS
  1 = FAIL
 
Perl's "die" produces exit code 255.  This should be used when a test wants to abort.  That is, when it makes no sense for the test to continue running.  Don't make the mistake of calling that FAIL.  A test that doesn't run isn't a product test failure.

IV. Setup Assumptions

These are the general assumptions that each automated test script can make.  Each of these is a separate piece of automation used to setup the testbed in preparation for test execution.  There is room for debate about where to draw the line between these, but what is important is that a test writer knows what to assume has already been done.
 
1) Hosts have been rebooted as necessary and have the required operating system and platform packages installed
2) The product software has been installed
3) The product software has been initially configured

V. Cleanup Assumption

Tests will leave behind some persistent configuration.  We need the means to cleanup between tests without a full reset of the testbed.  This is done before each test execution, rather than after.  This is because we can't rely on automated tests to exit normally, so cleanup may not be conducted and because we may need to leave the systems their failed state for bug triage.

1) The testbed is returned to the state equivalent to initial setup (IV, above)

VI. Automatic Execution

Regression tests should be written with automated execution in mind.  Assume the following algorithm for serially running a batch of test scripts.  Reference the above sections.

1) Initial Setup (Do this once, at the beginning of the batch)
     a. Reboot hosts and install OS and required packages (IV.1)
     b. Install product software (IV.2)
     c. Configure product software (IV.3)
 
2) Test Execution (Do this for each test script execution)
     a. Cleanup testbed (V.1)
     b. Execute test

This recipe has worked well for me in building a robust automation framework.  I hope you find it helpful.  The Test Contract is an important part of the foundation of reliable automated testing.


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt