[Python-3000] doctests vs. unittests (was Re: pimp; restructuring the standard library)

[Chris McDonough]

On Jun 28, 2007, at 11:46 PM, Barry Warsaw wrote:

> The thing that convinced me was the realization (assisted by my  
> colleagues) that doctests are first and foremost documentation.   
> They are testable documentation sure, but the unit tests are  
> secondary.  There's no question that for things like system  
> documentation, the narrative that weaves the testable bits together  
> in a well written doctest are much more valuable than the tests.

I suspect it would be even more valuable as documentation if it  
didn't give good coverage.


> Most unittest based tests have little or no comments, and nothing  
> approaching the narrative in a good doctest, so it's clear that  
> unittests are tests first and probably not documentation at all.

This probably isn't the place for this discussion but I'll give an  
explanation about why I think that's actually a good thing.

I find that I only get good test coverage when I have more test code  
for a component than the implementation of the component I'm trying  
to test.   At least that's been my experience.  I haven't been able  
to make the tests more succinct while still testing things adequately.

When coverage gets good, "documentation-ness" of tests suffers.  You  
can get good test coverage with any sort of tests.  But once you get  
good test coverage, whatever framework you've chose to write them in,  
the tests are no longer very good as narrative documentation because  
they're littered with bits of fixture code, edge case assertions, etc.

I don't mind doctest at all really (I just use unittest out of  
inertia and personal preference, I'd probably just as happy with nose  
or whatever).  I just don't like when folks advertise the same  
doctest as both a comprehensive set of tests and a component's only  
source of documentation, because I don't think it's possible for it  
to be both at the same time with any sort of quality in both  
directions simultaneously.

That said, having testable documentation is a good thing!  I'd just  
prefer that that documentation did not include lots of fixture noise.

> A good test suite can benefit from both doctests and unittests and  
> I don't think unittest will ever go away (nor should it), but in my  
> latest work I'm opting more and more for doctests.  That Tim Peters  
> is a smart guy after all I guess. :)

I miss "uncle Timmy". :-(

- C