Hardware take on software testing.

[Michele Simionato]

Peter Hansen <peter at engcorp.com> wrote in message news:<3EE14F0A.3D1E1887 at engcorp.com>...
> Bob Martin wrote "The act of writing a unit test is more an act of 
> design than of verification.  It is also more an act of documentation 
> than of verification.  The act of writing a unit test closes a remarkable 
> number of feedback loops, the least of which is the one pertaining to 
> verification of function."

Good points. See also a recent thread where I also advocate the merging of
documentation+testing.

> Let me go back to the "graph the number of bugs" thing you mention above.
> If you are working in a world where that concept holds much meaning, 
> you might have to change gears to picture this: with XP and TDD, you
> generally expect to have *no* bugs to graph.  Think about that for a
> moment.  You write a test.  You write a bit of code, and get the test
> to pass.  You write another test.  You write some code, only enough
> to pass the test, and run both tests.  If the code broke something in
> the first test, you immediately go and fix what you did wrong.

> Now fast-forward to months later, when you have literally hundreds of
> little tests, each one having driven the development of a few lines of
> code.  You have effectively 100% code coverage.  In fact, you probably
> have tests which overlap, but that's a good thing here.  Now you make
> a tiny mistake, which traditionally would not be noticed until "test 
> time", way down at the end of the project when the QA/verification people 
> get their hands on your code.  Instead of being noticed, perhaps, 
> months later just before shipping, you immediately fail one or two
> tests (or a whole pile) and fix the problem.

> Or, in spite of the fact that you actually *drove the development of
> the code with the tests*, and that therefore there is really no code
> that doesn't need to be there to pass the tests, you manage to let
> a bug get through.  Maybe it was more of an error in interpretation
> of the functional requirements.  In other words, almost certainly one
> of your tests is actually wrong.  Alternatively, the tests are all fine
> but you're in the unfortunate (but fortunately rare when you do it this
> way) position of having an actual, real _bug_ in spite of all those tests.

> What do you do?  Add it to the bug database and see the graph go up?
> No, you don't even *have* a bug database!  There are no bugs to go in it,
> except this one.  What's the best next step?  Write a test!

> The new test fails in the presence of the bug, and now you modify the
> code to pass the test (and to keep passing all those other tests) and
> check in the change.  Problem solved.  No bug, no bug database, no graph.

> Maybe this sounds goofy or unrealistic to some who haven't tried it.
> Personally I thought it was novel enough to warrant an experiment when
> I first encountered it, but it didn't take long before I was convinced
> that this approach was fundamentally different and more powerful than
> the previous approaches I'd tried over twenty plus years of coding.
> It may not feel right to some people, but since it's pretty darn easy
> to read up on the approach and experiment for a few hours or days to
> get a feel for it, "don't knock it if you haven't tried it".  :-)

> To answer the original question of "how do you know when you're done?"
> I would say that TDD itself doesn't really say, but in XP you have
> what are called "acceptance tests", which are similar to the unit tests
> in that they are a fully automated suite of tests that verify the 
> high level functionality of the entire program.  When your code
> passes all the units tests you have written to drive it's development,
> *and* all the acceptance tests, then you're done.  (That's another one
> of the "things of beauty" in XP: the tests aggressively control scope
> since you don't have to build anything for which you don't have a test.)

-Peter

Dear Peter,

I like the fact that you are so enthusiastic about TDD and eventually, having
read tons of your posts on TDD, I tried it. It was a worthwhile experience
and I must say it worked better then I expected. I am also convinced that  
TDD is the most effective technique we have at our disposal today (together 
with pair programming). No doubt also about the relevance of test in 
refactoring code, they really shine in this arena.

Nevertheless, I think you are a little bit too enthusiastic. 
I am referring to statements like "with XP and TDD, you generally 
expect to have *no* bugs" and "you have effectively 100% code coverage".

My concern is that it is effectively *impossible* to write a comprehensive
sets of tests. You cannot imagine and tests all the possible usages of
a piece of code you write, unless it is trivial. You cannot imagine and
tests all the possible abuse your future user could perform. You cannot
never be 100% sure of the correctness of your programs in all situations.

If it was so, Python (which is developed using LOTS of tests) would be
bug free. This is not the case, because of the subtle bugs you cannot prevent 
at writing time and that are discovered after months or years of usage. Unit 
testing is extremely effective againsts simple bugs (similarly to the compiler 
step in a compiled language, but much more effective), but it is powerless
(actually in some case it helps, but I am considering the really hard bugs
here) to prevent subtle bugs, exactly because they are subtle, i.e. you
haven't thought of a test able to discover them when you first wrote the
program. Of course, once the subtle bug has been discovered (maybe by your
users), tests helps a LOT in tracking it and fixing it.

Let me take a concrete example, i.e. my recipe to solve the metaclass
conflict I posted recently. Here the script is less than 20 line long.
However, I cannot simply write a comprehensive test suite, since there
are TONS of possibilities and it is extremely difficult for me to imagine
all the use cases. I can write a big test suite of hundreds of lines,
but still cannot be completely sure. I cannot spend weeks in writing the
test suite for 20 lines of code!

In general, the problem with test is that they are as effective as the 
time you spend in writing them. Since, because of constraint of money, 
deadlines and other real life issues, you cannot really write down a 
comprehensive test suite, you cannot say "with XP and TDD, you generally 
expect to have *no* bugs" and "you have effectively 100% code coverage".
These are really exaggerations in my view. But maybe it is good to
exaggerate a bit if this serves to convert more and more people ;)


                                                         Michele