Clever Frameworks Aren't

Clever, that is.

In QA, we test interfaces.  We always test interfaces, whether we're testing a CLI, a GUI, an API, a web service, or an individual subroutine.  Our job is to understand the interface's specifications, then create tests cases to assure the interface performs to those specifications.  That is most effectively done by operating the interface directly.

The last thing we want to do is to interject a translation layer between our test and the interface we're testing.  Such a layer will inevitably include limitations.

Here's an example of what not to do.  A large software company made a product that had a CLI.  The QA engineers wrote an object oriented framework that automatically builds a class hierarchy of the available CLI commands from the appropriate version of the product source code.  So you could do stuff like this (where setIpAddress() is translated to a CLI operation).  Very clever stuff.

CLI cli = new CLI;
cli.setProductVersion( "1.0" );
bool r = cli.setIpAddress( "10.10.22.123" );

Clever, yes, but unwise.  Only correct commands and arguments were permitted by the framework, because only those commands were generated from the source code.  Think about it.  The cutesy framework made it illegal to do negative testing.  The layer's "help" prevented QA from doing its job.

I think you'll find that wherever such framework layers exist, testing limitations exist.  It's therefore a mistake create frameworks that insulate you from the object under test.  It isn't that clever after all.


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt