[Python-ideas] doctest

[Barry Warsaw]

On Feb 17, 2012, at 02:57 PM, Mark Janssen wrote:

>I find myself wanting to use doctest for some test-driven development,
>and find myself slightly frustrated and wonder if others would be
>interested in seeing the following additional functionality in
>doctest:

FWIW, I think doctests are fantastic and I use them all the time.  There are
IMO a couple of things to keep in mind:

 - doctests are documentation first.  Specifically, they are testable
   documentation.  What better way to ensure that your documentation is
   accurate and up-to-date?  (And no, I do not generally find skew between the
   code and the separate-file documentation.)

 - I personally dislike docstring doctests, and much prefer separate reST
   documents.  These have several advantages, such as the ability to inject
   names into doctests globals (use with care though), and the ability to set
   up the execution context for doctests (see below).  The fact that it's so
   easy to turn these into documentation with Sphinx is a huge win.

Since so many people point this out, let me say that I completely agree that
doctests are not a *replacement* for unittests, but they are a fantastic
*complement* to unittests.  When I TDD, I always start writing the
(testable) documentation first, because if I cannot explain the component
under test in clearly intelligible English, then I probably don't really
understand what it is I'm trying to write.

My doctests usually describe mostly the good path through the API.
Occasionally I'll describe error modes if I think those are important for
understanding how to use the code.  However, for all those fuzzy corner cases,
weird behaviors, bug fixes, etc., unittests are much better suited because
ensuring you've fixed these problems and don't regress in the future doesn't
help the narrative very much.

>1. Execution context determined by outer-scope doctest defintions.

Can you explain this one?  For the separate-reST-document style I use, these
are almost always driven by a test_documentation.py which ostensibly fits into
the unittest framework.  It searches for .rst files and builds up
DocFileSuites around them.  Using this style it is very easy to clean up
resources, reset persistent state (e.g. reset the database after every
doctest), call setUp and tearDown methods, and even correctly fiddle the
__future__ state expected by doctests.

I usually put all this in an additional_tests() method, such as:

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/flufl/enum/tests/test_documentation.py

So setting up context is as easy as writing a setUp() method and passing that
to DocFileSuite.

One thing that bums me out about this is that I haven't really made the bulk
of additional_tests() very generic.  I usually cargo cult most of this code
into every package I write. :(

>2. Smart Comparisons that will detect output of a non-ordered type
>(dict/set), lift and recast it and do a real comparison.

I'm of mixed mind with these.  Yes, you must be careful with ordering, but I
find it less readable to just sort() some dictionary output for example.  What
I've found much more useful is to iterate over the sorted keys of a dictionary
and print the key/values pairs.  This general pattern has a few advantages,
such as the ability to add some filtering to the output if you don't care
about everything, and more importantly, the ability to print most string
values without their u'' prefix (for better py2/py3 compatibility from the
same code base without the use of 2to3).  Nested structures can be more
problematic, but I've often found that as the output gets uglier, the
narrative suffers, so that's a good time to re-evaluate your documentation!

>Without #1, "literate testing" becomes awash with re-defining re-used
>variables which, generally, also detracts from exact purpose of the
>test -- this creates testdoc noise and the docs become less useful.
>Without #2, "readable docs" nicely co-aligning with "testable docs"
>tends towards divergence.
>
>Perhaps not enough developers use doctest to care, but I find it one
>of the more enjoyable ways to develop python code -- I don't have to
>remember test cases nor go through the trouble of setting up
>unittests.   AND, it encourages agile development.  Another user wrote
>a while back of even having a built-in test() method.  Wouldn't that
>really encourage agile developement?  And you wouldn't have to muddy
>up your code with "if __name__ == "__main__": import doctest, yadda
>yadda".
>
>Anyway... of course patches welcome, yes...  ;^)

I've no doubt that doctests could be improved, but I actually find them quite
usable as is, with just a little bit of glue code to get it all hooked up.  As
I say though, I'm biased against docstring doctests.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120220/6f556bb9/attachment.pgp>