[Python-ideas] doctest (re-send to list)

[Mark Janssen]

On Mon, Feb 20, 2012 at 11:28 AM, Barry Warsaw <barry at python.org> wrote:
> On Feb 17, 2012, at 02:57 PM, Mark Janssen wrote:
> FWIW, I think doctests are fantastic and I use them all the time.  There are
> IMO a couple of things to keep in mind:
>
>  - doctests are documentation first.  Specifically, they are testable
>   documentation.  What better way to ensure that your documentation is
>   accurate and up-to-date?  (And no, I do not generally find skew between the
>   code and the separate-file documentation.)
>
>  - I personally dislike docstring doctests, and much prefer separate reST
>   documents.  These have several advantages, such as the ability to inject
>   names into doctests globals (use with care though), and the ability to set
>   up the execution context for doctests (see below).  The fact that it's so
>   easy to turn these into documentation with Sphinx is a huge win.
>
> Since so many people point this out, let me say that I completely agree that
> doctests are not a *replacement* for unittests, but they are a fantastic
> *complement* to unittests.  When I TDD, I always start writing the
> (testable) documentation first, because if I cannot explain the component
> under test in clearly intelligible English, then I probably don't really
> understand what it is I'm trying to write.
>
> My doctests usually describe mostly the good path through the API.
> Occasionally I'll describe error modes if I think those are important for
> understanding how to use the code.  However, for all those fuzzy corner cases,
> weird behaviors, bug fixes, etc., unittests are much better suited because
> ensuring you've fixed these problems and don't regress in the future doesn't
> help the narrative very much.

I think is an example of (mal)adapting to an incomplete module, rather
than fixing it.  I think doctest can handle all the points you're
making.  See clarification pointers below...

>>1. Execution context determined by outer-scope doctest defintions.
>
> Can you explain this one?

I gave an example in a prior message on this thread, dated Feb 17.  I
think it's clear there but let me know.

Basically, the idea is that since the class def can also have a
docstring, where better would setup and teardown code go to provide
the execution context of the inner method docstrings?

Now the question:  is it useful or appropriate to put setup and
teardown code in a classdef docstring?  Well, I think this requires a
committment on the behalf of the coder/documentor to concoct useful
(didactic) example that could go there.  For example, (as in the
prior-referenced message) I imagine putting example of defining a
variable of the classes type (">>> g = Graph({some complex,
interesting initialization})"), which might return a (testable) value
upon creation.

Now this could, logically, be put in the classes __init__ method, but
that doesn't make sense for defining an execution context, and *in
addition*, that can be saved for those complex corner cases you
mentioned earlier.

> I usually put all this in an additional_tests() method, such as:

Yes, I do the same for my modules with doctests.  A dummy function
which can catch all the non-interesting tests.  This, still superior,
in my opinion, than unittest.  It is easier syntactically, as well as
for casual users of your code (It has no leaning curve like
understanding unittest).

This superiority to unittest, by the way, is only realized if the
second suggestion (smart comparisons) is implemented into doctest.

>>2. Smart Comparisons that will detect output of a non-ordered type
>>(dict/set), lift and recast it and do a real comparison.
>
> I'm of mixed mind with these.  Yes, you must be careful with ordering, but I
> find it less readable to just sort() some dictionary output for example.  What
> I've found much more useful is to iterate over the sorted keys of a dictionary
> and print the key/values pairs.

Yes, but you see you're destroying the very intent and spirit of
doctest.  The point is to make literate documentation.  If you adapt
to it's incompleteness, you reduce the power of it.

>>Without #1, "literate testing" becomes awash with re-defining re-used
>>variables which, generally, also detracts from exact purpose of the
>>test -- this creates testdoc noise and the docs become less useful.
>>Without #2, "readable docs" nicely co-aligning with "testable docs"
>>tends towards divergence.
>
> I've no doubt that doctests could be improved, but I actually find them quite
> usable as is, with just a little bit of glue code to get it all hooked up.  As
> I say though, I'm biased against docstring doctests.

Well, hopefully, I've convinced you a little that the limitations in
doctests over unittests are almost, if not entirely due, to the
incompleteness of the module.  If the two items I mentioned were
implemented I think it would be far superior to unittest.  (Corner
cases, etc can all find a place, because every corner case should be
documented somewhere anyway!!)

Cheers!!

mark
santa fe, nm