[Python-ideas] Fwd: doctest

[Steven D'Aprano]

On Mon, Feb 27, 2012 at 12:28:17PM -0700, Mark Janssen wrote:
> On Mon, Feb 20, 2012 at 11:28 AM, Barry Warsaw <barry at python.org> wrote:
> > On Feb 17, 2012, at 02:57 PM, Mark Janssen wrote:
> > FWIW, I think doctests are fantastic and I use them all the time.  There are
> > IMO a couple of things to keep in mind:
> >
> >  - doctests are documentation first.  Specifically, they are testable
> >   documentation.  What better way to ensure that your documentation is
> >   accurate and up-to-date?  (And no, I do not generally find skew between the
> >   code and the separate-file documentation.)

I second what Barry says here. Doctests are for documentation, or at 
least, doctests in docstrings are for documentation, which means they 
should be simple and minimal, and only cover the most important parts of 
your function. Certainly they should only cover the interface, and never 
the implementation, so not used for regression testing or coverage of 
odd corner cases.

I have no problem with extensive doctests if they are put in an external 
document. I do this myself. But when I call help(func), I want to learn 
how to use func, not see seven pages of tests that don't help me 
understand how to use the function.


> > My doctests usually describe mostly the good path through the API.
> > Occasionally I'll describe error modes if I think those are important for
> > understanding how to use the code.  However, for all those fuzzy corner cases,
> > weird behaviors, bug fixes, etc., unittests are much better suited because
> > ensuring you've fixed these problems and don't regress in the future doesn't
> > help the narrative very much.

And again, +1 with what Barry says here, which means I disagree with 
your response:

> I think is an example of (mal)adapting to an incomplete module, rather
> than fixing it.  I think doctest can handle all the points you're
> making.  See clarification pointers below...

I don't accept this argument. Doctest is designed for including example 
code in documentation, and ensuring that the examples are correct. For 
that, it does a very good job. It makes a great hammer. Don't use it 
when you need a spanner.

It's not that doctest can't handle regression tests, but that regression 
tests shouldn't be put inside the the function docstring. Why should 
people see a test case for some bug that occurred three versions back 
in the documentation? Put it in a separate test suite, either unit 
tests, or a literate programming doc using doctest. Don't pollute the 
docstring with tests that aren't useful documentation.

When people read your docstring, you have to expect that they are 
reading it in isolation. They want to know "How do I use function 
spam?", and any example code should show them how to use function spam:

    >>> data = (23, 42, 'foo', 9)
    >>> collector = [5]
    >>> spam(data, collector)
    >>> collector
    [5, 28, 70, None, 79]

That works as documentation first, and as a test second. This does not:

    >>> spam(data, collector)  # data and collector defined elsewhere
    >>> collector
    [5, 28, 70, None, 79]

The fact that each docstring sees a fresh execution context is a 
good thing, not a bug.


> >>1. Execution context determined by outer-scope doctest defintions.
> >
> > Can you explain this one?
> 
> I gave an example in a prior message on this thread, dated Feb 17.  I
> think it's clear there but let me know.
> 
> Basically, the idea is that since the class def can also have a
> docstring, where better would setup and teardown code go to provide
> the execution context of the inner method docstrings?

Is that a trick question? I don't want docstrings to have automatic 
setup and teardown code at all, and if they did, I certainly don't want 
them to be in some other object's docstring (related or not).


> Now the question:  is it useful or appropriate to put setup and
> teardown code in a classdef docstring?

In my opinion, no, neither useful nor appropriate. It would be 
counter-productive, by reducing the value of documentation as 
documentation, while still being insufficiently powerful to replace unit 
tests.

A big -1 on this.


[...]
> Well, hopefully, I've convinced you a little that the limitations in
> doctests over unittests are almost, if not entirely due, to the
> incompleteness of the module.  If the two items I mentioned were
> implemented I think it would be far superior to unittest.

I already think that doctest is far superior to unittest, for testing 
executable examples in documentation. I don't think it is superior to 
unittest for unit testing, or regression testing. Nor is it inferior -- 
its just different.


> (Corner
> cases, etc can all find a place, because every corner case should be
> documented somewhere anyway!!)

I think you have a different idea of "corner case" than I do.

Corner cases, in my experience, refer to the implementation: does the 
function work correctly when the input is in the corner? Since this is 
testing the implementation, it shouldn't be in the documentation.

The classic example is, does this list-function work when the list is 
empty? So I would expect that the unit tests for, say, the sorted() 
built-in will include a test case for sorted([]). (This applies 
regardless of whether you use unittest, doctest, nose, or some other 
testing framework.)

But the documentation for sorted() don't need to explicitly state that 
sorting an empty list returns an empty list. That's a given from the 
accepted meaning of sorting -- if there's nothing to sort, you get 
nothing. Nor does it need to explicitly state that sorting a list with 
one item returns a list with one item. A single example of sorting a 
list of (say) four items is sufficient to document the purpose of 
sorted(), but it would be completely insufficient for unit testing 
purposes.


-- 
Steven