[Python-ideas] doctest

[Michael Foord]

On 18 February 2012 04:24, Ian Bicking <ianb at colorstudy.com> wrote:

> On Feb 17, 2012 4:12 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> >
> > On Sat, Feb 18, 2012 at 7:57 AM, Mark Janssen <dreamingforward at gmail.com>
> wrote:
> > > Anyway... of course patches welcome, yes...  ;^)
> >
> > Not really. doctest is for *testing code example in docs*. If you try
> > to use it for more than that, it's likely to drive you up the wall, so
> > proposals to make it more than it is usually don't get a great
> > reception (docs patches to make it's limitations clearer are generally
> > welcome, though). The stdib solution for test driven development is
> > unittest (the vast majority of our own regression suite is written
> > that way - only a small proportion uses doctest).
>
> This pessimistic attitude is why doctest is challenging to work with at
> times, not anything to do with doctest's actual model.  The constant
> criticisms of doctest keep contributors away, and keep its many resolvable
> problems from being resolved.
>

Personally I think there are several fundamental problems with doctest *as
a unit testing tool*. doctest is *awesome* for testing documentation
examples but in particular this one:

* Every line becomes an assertion - in a unit test you typically follow the
arrange -> act -> assert pattern. Only the results of the *assertion* are
relevant to the test. (Obviously unexpected exceptions at any stage are
relevant....). With doctest you have to take care to ensure that the exact
output of *every line* of your arrange and act steps also match, even if
they are irrelevant to your assertion. (The arrange and act steps will
often include lines where you are creating state, and their output is
irrelevant so long as they put the right things in place.)

The particular implementation of doctest means that there are additional,
potentially resolvable problems that are also a damn nuisance in a unit
testing fail:

Execution of an individual testing section continues after a failure. So a
single failure results in the *reporting* of potentially many failures.

The problem of being dependent on order of unorderable types (actually very
difficult to solve).

Things like shared fixtures and mocking become *harder* (although by no
means impossible) in a doctest environment.

Another thing I dislike is that it encourages a "test last" approach, as by
far the easiest way of generating doctests is to copy and paste from the
interactive interpreter. The alternative is lots of annoying typing of
'>>>' and '...', and as you're editing text and not code IDE support tends
to be worse (although this is a tooling issue and not a problem with
doctest itself).

So whilst I'm not against improving doctest, I don't promote it as a unit
testing tool and disagree that it is suited to that task.

All the best,

Michael Foord




> > An interesting third party alternative that has been created recently
> > is behave: http://crate.io/packages/behave/
>
> This style of test is why it's so sad that doctest is ignored and
> unmaintained.  It's based on testing patterns developed by people who care
> to promote what they are doing, but I'm of the strong opinion that they are
> inferior to doctest.
>
>   Ian
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120227/7771dd93/attachment.html>