Test Data Types

There are two kinds of test data generated by testing.  As I've said elsewhere in this blog, it's the Release Candidate that separates them.  If we want to run an efficient testing organization we need to understand the ways in which we use data to decide how we store and summarize it in managing our software projects.

Pre-Release Candidate Testing

Testing we do prior to having a completed product doesn't count toward qualifying the product for shipping to customers.  We may do a lot of testing during development.  We may test every integration build or every check-in.  That produces a lot of test results data.  Testing during this phase focuses on maintaining product quality during development, so we're trying to find bugs.  We're managing by exception.  We want to know about regressions as soon as possible.  We're very interested in the failures and not at all in tests that pass.  We file bugs for the failures.  The passing tests we ignore.

The Data - We need to keep test failure forensic data (cores, logs, config, test logs, etc...) for debugging and associate it with the bugs we file.  But we have no use for the test results data from the tests beyond that.  After filing bugs the data not associated with the bugs can be discarded.

Release Candidate Testing

By contrast, testing we do on a Release Candidate is official.  We very much care about all the test results and we want to see them all pass.  We want to manage execution of a Test Plan through to completion.  After all, the goal of this testing is to ship the product.  We should have already found the bugs.  We're hoping to ship the product and we need all the test results to prove that it's ready.

The Data - We need to keep the test results data forever.  We're going to use it for comparison in future regression testing.  These are the results that customers and partners will want to see.  Of course, we keep the forensic data with bugs as usual but unlike the test results generated by Pre-Release Candidate testing, we need to keep all of the results data too.

Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt

Quality - What is it?

"God is in the details" - Ludwig Mies van der Rohe

Mercedes, Apple, Google...why is it we think of their products as exceptionally high quality?

Let's take the example of Mercedes.  Is it enough for them to try to deliver a car that never breaks down on the freeway?  Do you think that's the end to which they aspire?  Is that enough to separate them from, say, Volkswagon?  No, that can't be it.  VWs don't break down of the freeway either.  In fact, isn't that just meeting the bare minimum customer expectation?  Isn't it (rightly) assumed that your car won't break down?

If we should be able to expect any car to never break down, then robustness can't be what characterizes high quality.

Can you guess why I love this commercial?

http://www.youtube.com/watch?v=6sr3Rh7Yjnc

Look at the focused attention to detail.  There is a product specification that details the width of that gap.  It specifies how wide it can be, how narrow it can be, and the permissible variance in width.  There are test cases about that gap.  Mr. Labcoat is going to send that car back for adjustment and possible redesign if the tests fail and the product doesn't meet those specifications.

"God is in the details."

So is quality.


Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt