Code correctness, and testing strategies

[Paul Rubin]

aahz at pythoncraft.com (Aahz) writes:
> You must have poor project management/tracking.  You WILL pay the cost
> of testing, the only question is when.  The when does have an impact on
> other aspects of the development process.

Well, let's say you used TDD and your program has 5000 tests.  One
might reasonably ask: why 5000 test?  Why not 10000?  Why not 20000?
No number of tests can give you mathematical certainty that your code
is error-free.  The only sensible answer I can think of to "why 5000"
is that 5000 empirically seemed to be enough to make the program
reliable in practice.  Maybe if you used a more error-prone coding
process, or wrote in assembly language instead of Python, you would
have needed 10000 or 20000 tests instead of 5000 to get reliable code.

But then one might reasonably ask again: why 5000 tests?  Why not 2000
or 1000?  Was there something wrong with the coding process, that
couldn't produce reliable code with fewer tests?

So, I think you have to consider the total development cycle and not
treat test development as if it were free.

I also haven't yet seem an example of a real program written in this
test-driven style that people keep touting.  I use doctest when
writing purely computational code, and maybe it helps some, but for
more typical code involving (e.g.) network operations, writing
automatic tests (with "mock objects" and all that nonsense) is a heck
of a lot more work than testing manually in the interactive shell, and
doesn't seem to help reliability much.  I'd be interested in seeing
examples of complex, interactive or network-intensive programs with
automatic tests.