I feel most comfortable when I test from two directions – functional or acceptance testing from the user point of view and unit testing from the developer’s point of view. Testing from these two perspectives creates robust coverage and a really solid base for refactoring and maintenance work.

The functional direction really should express how clients use your software. The goal is to treat your component or app as a black box, fling real stuff at it (as real as you can get), and verify that the results are as expected. The ideal point to reach is where you can take data files (or db dumps or whatever, but files are best) and drop them somewhere and have a running functional test within 5 minutes. If you get to that point, then you are able to do a really fantastic level of support and maintenance because you can typically reproduce a reported problem, debug it, and turn around a solution very fast. You also have a great bed of real world data to check more complicated code base changes with.

This kind of testing is anecdotal, but covers how people really use the software, hence the coverage you get is broad (hits lots of features in your software) and hits them deeper in areas where they are more heavily used. So, the coverage profile should hit functionality proportional to usage. Indeed, an interesting experiment when using a coverage tool is to produce a coverage-enabled version of your product, and actually run typical user scenarios on it, then examine the profile. It may be surprising which parts of the code are hit the most and which are not hit at all.

The unit testing direction should treat your software as a white box where we know what’s inside and we attempt to probe every corner of it, verifying that it works as expected. This kind of testing should ideally be exhaustive, testing all possible inputs on every piece of functionality, tending towards tests on a micro scale (function level). Realistically, testing at 100% coverage is not worth the effort. My experience has been that 80% coverage is about the point where you hit diminishing returns – the additional coverage is not worth the time as you have covered the majority of the critical code. Typically, once you get to that point you’re just padding your coverage numbers, not using your time wisely. The coverage profile you should from unit tests should be uniformly deep but deeper in areas of high complexity or criticality.

These two kinds of tests reinforce each other and both are essential. With only unit tests, you tend to lose sight of interaction and integration issues. You also lose the ability to quickly move from a real world problem report to a reproducible test. With only functional tests, you are likely to miss many small things that may not exist in your set of functional tests but lie just outside the most common and happy path, like a mountain lion ready to pounce out of the tree and chomp down on your jugular. Okay, maybe that was a little overdramatic. But you’re pretty much guaranteed to see those as soon as you show something to a customer, or at least that’s how it seems.