Thursday, August 19, 2010

Should I "Break Stuff"?

"We want people in QA who are good at breaking stuff." - Generic Manager

Sorry, but that's just patently incorrect.  I've known a lot of QA engineers who think that's their job.  As a consequence, they feel unproductive when they're not finding (creating?) bugs.  That feeling is often validated by their manager's focus on measuring their performance by their bug rate.  This misunderstanding results in searching for and finding a lot of non-bugs and the consequent wasted QA and development cycles.  They drive the car into the lake and say, "Look!  It doesn't float!  That's a bug!"  But it's only a bug if the product specifications say the car should float.

Contrast the above attitude with the correct understanding of QA's mission: Our job is to demonstrate the product conforms to its specifications.  Of course, among other things, the specifications should include robustness goals.  And demonstrating we meet those robustness goals is a legitimate QA function.  Demonstrating the product behaves correctly in negative test cases is also legitimate.  The reason so many QA engineers and managers focus on "breaking the product", is that they often don't have specifications from which to plan their tests.  They don't know whether the car is supposed to float or not.  Trouble is, it's their job to know and if they don't, to find out.

QA's value, then, in the correct paradigm is in providing the information necessary to decide whether the product is fit to ship.  This results in a different assessment of QA's value that doesn't depend on the bug find rate.  It means the tests that pass are every bit as valuable as the tests that fail, a dramatically different mindset from the "breaking stuff" mentality.  The QA engineer who believes he must, "break the product" will see no value in tests that pass.  He will also see no value in issues he considers trivial, like whether the product is easy to use or whether the documentation is correct.  As the product's quality improves with time, he will view his value as diminishing as it becomes increasingly difficult to find robustness bugs.

Can you imagine the QA engineer at the end of the car assembly line banging on the car with a sledge hammer?  Why is that absurd?  He could easily beat holes into it and claim to have found a defect.  If you agree with me that that's not his job, what exactly is his value in the production process?

Todd Shoenfelt

No comments:

Post a Comment