Complex systems require complex tests. Remember 42? Douglas Adams’ conceit that the entire world was a computer built to answer a question and the answer was 42? Adams was, I believe, quite the technologist. He even wrote a computer game (which was viciously difficult), back when ‘computer game’ meant, typing in stuff, and reading cryptic descriptions. And he understood the nature of testing: any computer program very complex requires a very complex testbed.
Commercial companies do not test all of their code, for the reasons mentioned above: the time and expense are huge. Your most complex software (embedded systems) has 80% code coverage at best, often far less. By coverage, I mean, tests that pass through that percentage of the code’s logic paths, with 100% of the range of possible values at any point. Why is that?
When you start small, testing is easy. A widget that sorts a list is easy to test: write some lists, write the expected result, have the test engine run the inputs, and compare the outputs. Things get tougher when you have multiple widgets working together – you had to have pretty complex inputs, and the outputs have to be computed ahead of time. Put a lot of those widgets together and the testing challenge becomes exponentially harder.
Here’s an example of how things work in the real world. One of my first jobs was doing QA for a little software firm that did apps that valued complex financial instruments. Most of my ‘testing’ was just making sure the answers did not change from version to version. Fair enough, the current versions’ answers were deemed correct, and I just had to compare the new answers, and make sure they did not change.
But hold on here, how do we know the old answers were right? After all the financial instruments in question are so complex, the guys who write the algorithms are called ‘rocket scientists.’ (Yes, these were the much-maligned ‘derivatives’ you’ve heard about). But we knew the old code was right because our clients had been using it and their rocket scientists would complain if the answers were wrong. When they got a version with a new module, they’d check out answers given by our app against their existing spreadsheet models with some existing contract, make sure it was all good, then start entering new data. If they got a wrong answer – bug report.
Our customers check our work. That’s the dirty secret of the software industry.
Sure, we did some of our own modeling. I got to flex my young brain and write the test for the Black-Scholes algorithm. Basically, to validate our app, I had to replicate the logic, sort of a clean room test – I wasn’t looking at our code, just my old Econ 405 textbook. And I got some weird results, which our local rocket scientist, Jay B., would check out. Anyway, this was cool and fun and took weeks, for one new module. Quality = Time + $$$$. Easier and cheaper to let your clients test for you.
Taking it to the extremes of space
How are you going to test a spaceship? Something so complex that even the smallest adjustments are 1000 times more complex than old Black-Scholes? How will you test it for heat, cold, passing through radiation belts and being hit by solar flares? Each of the systems has to handle exceptions caused by other systems, all the APIs have to line up right — in other words, no bugs. It will be incredibly expensive, and the entity doing these tests will not be able to sustain a pace required by a competitive environment.