Tuesday, May 18, 2010

Combinations and Permutations

Denial makes for poor quality software.

"If you choose not to decide, you still have made a choice." - Neil Peart

Software and hardware combinations quickly get out of hand.  The number can easily be in the millions.  Software companies respond to this in two ways, both of them bad:

1) Build billion dollar interoperability labs
2) Allow enormous testing gaps and keep customers in the dark

Here's a simple combinatorics computation:

Supported Hardware Options

* Hardware platforms: 3 (3)
* Memory sizes: 4 (3 x 4 = 12)
* Disk types and sizes: 6 (12 x 6 = 72)
* Network cards: 4 = (72 x 4 = 288)

Supported Software Options

* Operating Systems software versions: 6 (288 x 6 = 1,728)

To qualify our product software to run on hardware with that many options, we need 288 testbeds, assuming we have the ability and sophistication to install/uninstall the 6 operating systems on those testbeds on demand.  We must execute 1,728 instances of each test.

If we want to dedicate operating systems to testbeds, we need 1,728 testbeds.  And we still need to execute each test 1,728 times.

Every time another hardware platform or OS version is proffered by marketing, management must make the necessary calculations to estimate the impact on schedules, resources, and budget.  They must understand what they're signing up for.  In the above example, each additional O/S version means each test has to be executed another 288 times in order to be qualified across all the hardware.

Companies have to get a handle on this problem.  Denial is all too common, but it isn't an answer.  Your customer has a right to believe that you've tested what you say works.  Short of building a billion dollar lab, which isn't an option for most companies, the only real solution is carefully to limit the combinatoric variables.

Sadly, these calculations are never made.  Most companies unknowingly choose option 2 and choose to live in denial.  So my advice to software customers is to think about the enormity of this, and always buy the support contract.  ;^)

Todd Shoenfelt

No comments:

Post a Comment