[Python-ideas] doctest

[Michael Foord]

On 27 February 2012 20:35, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 18 February 2012 04:24, Ian Bicking <ianb at colorstudy.com> wrote:
>
>> On Feb 17, 2012 4:12 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>> >
>> > On Sat, Feb 18, 2012 at 7:57 AM, Mark Janssen <
>> dreamingforward at gmail.com> wrote:
>> > > Anyway... of course patches welcome, yes...  ;^)
>> >
>> > Not really. doctest is for *testing code example in docs*. If you try
>> > to use it for more than that, it's likely to drive you up the wall, so
>> > proposals to make it more than it is usually don't get a great
>> > reception (docs patches to make it's limitations clearer are generally
>> > welcome, though). The stdib solution for test driven development is
>> > unittest (the vast majority of our own regression suite is written
>> > that way - only a small proportion uses doctest).
>>
>> This pessimistic attitude is why doctest is challenging to work with at
>> times, not anything to do with doctest's actual model.  The constant
>> criticisms of doctest keep contributors away, and keep its many resolvable
>> problems from being resolved.
>>
>
> Personally I think there are several fundamental problems with doctest *as
> a unit testing tool*. doctest is *awesome* for testing documentation
> examples but in particular this one:
>
> * Every line becomes an assertion - in a unit test you typically follow
> the arrange -> act -> assert pattern. Only the results of the *assertion*
> are relevant to the test. (Obviously unexpected exceptions at any stage are
> relevant....). With doctest you have to take care to ensure that the exact
> output of *every line* of your arrange and act steps also match, even if
> they are irrelevant to your assertion. (The arrange and act steps will
> often include lines where you are creating state, and their output is
> irrelevant so long as they put the right things in place.)
>
> The particular implementation of doctest means that there are additional,
> potentially resolvable problems that are also a damn nuisance in a unit
> testing fail:
>

Jeepers, I changed direction mid-sentence there. It should have read
something along the lines of:

As well as fundamental problems, the particular implementation of doctest
suffers from these potentially resolvable problems:


>
> Execution of an individual testing section continues after a failure. So a
> single failure results in the *reporting* of potentially many failures.
>
> The problem of being dependent on order of unorderable types (actually
> very difficult to solve).
>
> Things like shared fixtures and mocking become *harder* (although by no
> means impossible) in a doctest environment.
>
> Another thing I dislike is that it encourages a "test last" approach, as
> by far the easiest way of generating doctests is to copy and paste from the
> interactive interpreter. The alternative is lots of annoying typing of
> '>>>' and '...', and as you're editing text and not code IDE support tends
> to be worse (although this is a tooling issue and not a problem with
> doctest itself).
>
>

More fundamental-ish problems:

    Putting debugging prints into a function can break a myriad of tests
(because they're output based).

    With multiple doctest blocks in a test file running an individual test
can be difficult (impossible?).

    I may be misremembering, but I think debugging support is also
problematic because of the stdout redirection.

So yeah. Not a huge fan.

All the best,

Michael


> So whilst I'm not against improving doctest, I don't promote it as a unit
> testing tool and disagree that it is suited to that task.
>
> All the best,
>
> Michael Foord
>
>
>
>
>>  > An interesting third party alternative that has been created recently
>> > is behave: http://crate.io/packages/behave/
>>
>> This style of test is why it's so sad that doctest is ignored and
>> unmaintained.  It's based on testing patterns developed by people who care
>> to promote what they are doing, but I'm of the strong opinion that they are
>> inferior to doctest.
>>
>>   Ian
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>>
>
>
> --
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
>
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120227/62db0ae2/attachment.html>