Development and Testing

There are two distinct phases to QA:

1) Testing we do before Code Freeze
2) Testing we do after Code Freeze

The testing we do after Code Freeze is the only testing that "counts" toward qualifying the product.  Before that, our testing is designed to keep product quality high during development, but doesn't "count" because we don't have a completed product.  Here's a graph showing how I see the development and testing milestones in the production process.

Branch Milestone

The line at the left of the graph is the point at which we branch our new release.  At that point, its content is exactly equal to the previous release.  There have been no check-ins, neither new content nor bug fixes.  Consequently, all the tests in the previous release still apply, assuming no features were deleted.  We test a build of the product that may include special test features and privileges that won't ship with the product.  During this development phase, pre-checkin unit tests assure new features work.  Post-checkin nightly regression tests assure existing features don't break.  Weekly disruptive system testing keeps robustness high.  The goal of the testing is to assure bugs are identified with the check-ins that caused them, before reaching Code Freeze.  Defects found later are harder to diagnose and assign, expensive to fix, and can seriously delay the release.

Release Candidate Milestone

The next line is where the first Release Candidate (RC) is declared (for the purposes of simplicity and clarity I assume only one RC).  At that point all principle development of new features and bug fixes have been checked in, unit tested, and fully regression tested, including upgrade and revert testing.  All the previous release's tests still apply along with the new features' tests.  All the known bugs that would stop shipment have been fixed.  This is the starting point of our Test Plan execution and official product qualification.  Here, we're starting to test in order to decide whether we can ship the RC.  If show stopper bugs occur, we fix the bug(s), then produce a new RC and restart Test Plan execution.

Release Milestone

The last line is the product release.  We have a completed product and all the tests in the Test Plan have been executed on the RC.  Bugs found during qualification testing are not severe enough to warrant to stop shipment.

I'll be referring to the graph in future posts in describing how testing relates to the Development and Testing phases and how we can make our test automation count.

Todd Shoenfelt
http://www.linkedin.com/in/toddshoenfelt