Hardware take on software testing.

[Peter Hansen]

Donald 'Paddy' McCarthy wrote:
> 
> I guess one of the things I question about XP then is this notion that
> you only add good code to your program because of the quality of your
> tests; and that prudent re-factoring also gets rid of unnecessary code.
> If you are doing the above manually then I think that you could gain a
> lot by using coverage tools to help assess the quality of your tests not
> necessarily w.r.t. a spec, but in how well those tests cover your
> program. Coverage tools can be applied automatically for simple but
> useful results, or can give more detailed results with manual
> intervention. If someone is doing XP for a commercial project, why not
> invest in coverage tools? A coverage report could be part of every peer
> review, or generated as a matter of course during development if
> runtimes are not too long.

Well, XP certainly neither requires nor prevents you from using 
coverage tools, but merely suggests strongly that you should do
so only when the benefits outweigh the costs.  It also suggests 
not doing so "just because" but only when there is a specific driving
need.  I suggest that most XP groups find that their test coverage
is already excellent and they find little reason to add another 
layer to their process.

My own group, for example, has easy access to PyCover (or whatever
it's called) but hasn't bothered to implement it yet.  The areas
of code which have been driven with tests seem to have 100% 
coverage, and until we get bitten at least once, there's no particular
reason to check up on that theory.  I realize that will sound 
backwards to some people, but in practice it's an approach that works
rather well in the face of tight resource constraints.

> Random constrained generation wouldn't have to be seen as outside XP or
> TDD. If the language used for writing tests had good support for random
> generation, and software engineers were trained in the technique, it
> would just become another type of arrow in their quiver on their hunt
> for bugs on their quest for quality software. (Whoops, getting a little
> flowery towards the end there :-)

We have also considered using randomized test input but we haven't
seen a test case where we felt that would add much value (for the higher
cost of implementing it properly).  Again, XP neither requires nor 
prevents a group from using such a thing, it merely doesn't discuss it.
This is the case with dozens of specific tools that other processes 
might mention and require.  XP tries to be lightweight and avoid 
forcing specific tools or very fine-grained processes on people.

To paraphrase Yogi Berra, then, on randomized tests: if people
don't want to use them, XP isn't stopping them! :-)

-Peter