Test Specifications Don't Have Variables

Specifications are, um, specific.

In the network storage industry, I've seen hundreds test cases that read something like, "STEP 1: Create a volume between 100 and 500 gigabytes in size".

Well, which?

Assuming the system doesn't take a size range for an argument, eventually we have to decide how big to make it.  We're not getting out of the decision by being indecisive in the specification.

Decisions like this should be made during test planning, because:

1) The test planners are the most qualified to make them.
2) By documenting decisions, you can get input during test plan review.  You can get by-in from the stakeholders on test coverage to avoid surprises and second guessing.
3) You can evaluate your testbeds to assure you have the required equipment.

By listing a range, the test planner is saying, "I want somebody else to make the decision".  The implication is that the person executing the test is more qualified to fill in the blanks in the specification.  That shouldn't be the case.  The test designers should be more knowledgeable than the test executors.  They therefore shouldn't leave these decisions to the executors.  Test designers should nail down the test case specification to assure it becomes a reliable, deterministic, and repeatable test of quality and to do that they must remove all ambiguity.

An open specification undermines regression testing.  For instance, during release 1.0 testing, the tester uses a 100 gig volume and it passes.  During release 1.1 testing, the tester uses a 500 gig volume for the same test case and it fails.  Is it a regression, or not?  This sort of mismatch is very common even if the same person executes the test in both releases.  This variability is even more likely when the test is executed on different testbeds by different people.  Further, when we eventually automate the test, we will certainly have to specify the volume size anyway.  The automation requires that the specifications details be known.

Regression test specifications don't have variables.  They are deterministic.  They have to be.  You'll have to make up your mind eventually, so why not do it in the specification?


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt