What is a test?

As my first post, I want to define the basic terminology I'll be using to establish the foundation for future posts.

What is a test?
A test is the evaluation of an assertion.

To test our software, we evaluate its quality by comparing actual results to expected results. If the actual result equals the expected result, the test PASSes. If it doesn't, it FAILs. The expected results are specified in a test specification, which is derived from a functional specification. That is, we must know how our software is supposed to behave before we test it. A specification is specific. It is unambiguous. There are no variables in a test specification and there are no test results other than 'PASS' or 'FAIL'.
What isn't a test?
Simply operating something is not a test. You drive your car to work. That doesn't mean you're testing your car.
A test of your car's top speed, for instance, requires that:

1. the product specification includes its top speed,
2. the top speed specification provides the testbed specification (straight road, flat, asphalt, etc...),
3. the test specification specifies the assertion and PASS/FAIL criterion
   
4. the test specification is executed to completion, and
5. the actual top speed is evaluated against the expected top speed, resulting in an unambiguous test result: PASS or FAIL.

Demonstrating the software lacks a feature, isn't a test failure.

If you drive your car into the ocean and it sinks, you can't say the car failed the float test. The product specification doesn't say the car can float. We test to specifications. The fundamental responsibility of QA is to demonstrate the product meets its specifications. If QA doesn't know how the software is supposed to behave, they're can't assure its quality.

Often, QA will be engaged in what I call, discovery. In the absence of a specification, they're trying to 'discover' how the product works (performs, scales...whatever). This may be important work, but is not QA testing.

As the quality evaluator, QA is responsible for knowing the characteristics of the software.

Your teacher gives you a piece of paper with questions on it. You answer them and return the paper. You ask, "What's my score?". She says, "I don't know. I don't know what the correct answers are." If that's the case, how can you call that a test? It's her job to know the correct answers, then evaluate your answers against the correct answers, and finally calculate your score.
I'll address the many types of tests in another post.



Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt