Leveraging the Automation Harness

In previous posts (http://qa4software.blogspot.com/2009/10/automation-framework.html and http://qa4software.blogspot.com/2009/07/startup-automation-recipe.html), I detailed the requirements and benefits of a test execution harness.  After being initially developed, the test harness can be repeatedly leveraged to improve development quality and massively increase test execution frequency.  The test harness logs every detail of its activities and testbed configuration, makes the results browseable and linkable via a web server, and emails the user the test results.  Automated execution can easily be scheduled to run from cron for each codeline and platform to increase coverage at little cost.

Here are the ways in which the test harness can be leveraged to pay you back for the initial investment.

Nightly Functional Regression Tests

A fixed batch of regression tests can be scheduled to run nightly from cron with the latest build for each codeline and platform.  The product can be installed on virtual machines or simulators.

Nightly Upgrade/Revert Tests

A batch of upgrade/revert pairs can be scheduled to run nightly from cron for each codeline and platform.  Upgrade from each fixed release version to the latest build of each platform.  Revert from the latest build of each platform to a fixed release version.  Run sanity tests after each upgrade or revert.  The released versions are the major versions of the software that are already being run by customers and supported by the company.  The product can be installed on virutal machines or simulators.

Nightly Performance Tests

A batch of performance regression tests can be scheduled to run nightly from cron for each codeline and platform.  The performance tests will establish performance baselines for each platform compare results to that baseline.  The product must installed on hardware.

Test Automation Verifier Web Tool

A simple Perl CGI can be written to allow an automation developer to execute new tests in the harness.  This is intended to assure that newly written automated tests can be added to the nightly tests safely.  Obviously, the execution testbed configuration should match the regression testbeds.  The developer supplies his automated test.  The CGI launches the harness to install standard product builds and runs the new test repeatedly.  The product can be installed on virtual machines or simulators.

Pre-Checkin Test Web Tool

A simple Perl CGI can be written to allow a product developer to execute tests of his private builds before checking in his code.  The developer supplies his privately built executable and chooses the tests he wants to run.  The CGI launches the harness to install the developer's build and executes the tests.  The product can be installed on virtual machines or simulators.

Software Install/Uninstall Web Tool

This is just a scaled down version of the pre-checkin test tool above.  A simple Perl CGI can be written to allow anyone to reboot hosts, install, configure, uninstall, and/or unconfigure the product software.  The user supplies the build executable, either privately or automatically built, and chooses the operations he wants to run.  The CGI launches the harness to execute the chosen operations.  The product can be installed on virtual machines or simulators.

Per-Build Tests

A daemon or cron can be written to monitor each codeline and run a series of quick smoke and integration tests via the harness for each product source code checkin.  The product can be installed on virtual machines or simulators.

A well designed test execution harness has many uses and can do the work of literally dozens of test engineers.  It shifts your attention from manual testing to developing automated tests and monitoring test results - a higher order, more sophisticated mode of operation that saves you money.

In short, the test execution harness is just something you simply can't live without.


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt