Automation Framework

What's a test automation harness?  It's the means by which a batch of test scripts are serially executed.  It's fairly easy to build one in Perl and it can be leveraged in several ways once it's built to pay you back for the investment.

You must first define the Test Contract because the test harness will rely on it.  This is absolutely essential.  The Test Contract is the agreement between test input/output and the test harness.  Test Automators must follow the rules.  Perl is preferred for tests, but as long as the Test Contract's rules are followed, the programming language you use for tests is unimportant (although you should have a very good reason for your choice).

The harness will be designed to run from the command line and will eventually run from cron nightly and per checkin.  All the forensic data produced by the run will be plain text, so it can be viewed and parsed at the command line with grep and viewed in a browser.

Here are the requirements for the test harness script:

Arguments

• Take a software configuration file as an argument to specify the software versions for the test.
• Take a testbed configuration file as an argument specifying the testbed on which to execute the tests. 
• Take a batch file listing the test scripts to be run as an argument, with timeouts for each.
  

Setup

• Create a unique run ID
• Create a unique directory named after the run ID to contain run logs.
• Reboot hosts, install the software, and configure the testbed before running batch of scripts.
• Create a detailed log file containing all test output.
• Copy the batch file and testbed configuration file to the run directory.
• Create a meta data file listing the software version, hosts, and log locations.
• Start memory measurements at the beginning of the run.

Execution

• Capture standard out and standard error from each test execution to the detailed log.
• Timestamp the start and end time of each test script in the detailed log.
• Timestamp the start and end time of the entire batch run.
• Cleanup the testbed between each test script.
• Check for configuration issues between each test script.
  

Cleanup

• Create a summary log detailing the start/end time, duration, exact command executed, and results for each test.
• Run a triage script at the end of the run to look for problems with the testbed (eg cores, configuration disruption, reboots, memory leaks, etc...).
• Email results to concerned parties.
  

For extra credit, build a simple Perl CGI to allow the plain text test results to be viewed in the browser.  After you've built the framework and are running tests nightly from cron, your job becomes monitoring test results daily and writing automated tests to add to the nightly batch.  In my experience, 12 hours of nightly tests executed from a such a framework can deliver the work of twelve full time testers with a single testbed.  You'll find a tremendous number of bugs far in advance of formal qualification testing.


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt