Test utilities are scripts or compiled programs that can be run from the command line that automate routine tasks associated with development and testing. They have several use cases and hence pay us back continuously for the initial investment of creating them. They have the dual advantage of benefiting both manual testers and automation. They can be strapped in to CGI GUIs as well to allow the utilities and tests to be executed from a browser. They can be used in configuration management to maintain the readiness of lab equipment. Here are some of the utilities I've found to be of general use:
Install - Newly install the software on a testbed.
Upgrade - The software is already installed on the system. Upgrade the software to a newer version while maintaining data, state, configuration, and (possibly) service continuity.
Revert - Like with upgrade, the software is already installed on the system. Revert the software to an older version. The data, state, configuration, and service continuity as possible should remain intact, net of new features that didn't exist in the version to which we're reverting.
Configure - Set the standard software configuration for our lab after the install script is run. This can include extra configuration specific to testing our software.
Monitor - aka 'Alert Daemon'. Monitor the testbed and tell us when the product is unhealthy. Alert criteria can include things like cores, missing services/processes, or unplanned reboots. The monitor can be designed to run snarf when it sees problems, assuring we don't lose important forensic data or bugs.
Snarf - Copy forensic data off the testbed to somewhere safe, typically a NAS share. Copied data can include logs, config files, binaries, and cores for each node in the testbed.
Reset - Return the nodes in the testbed to its pre-install state, leaving it ready for the install script to be run.