[Python-ideas] doctest

[Michael Foord]

On 27 February 2012 23:23, Mark Janssen <dreamingforward at gmail.com> wrote:

> On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord <fuzzyman at gmail.com> wrote:
>>
>> As well as fundamental problems, the particular implementation of doctest
>> suffers from these potentially resolvable problems:
>>
>>
>>> Execution of an individual testing section continues after a failure. So
>>> a single failure results in the *reporting* of potentially many failures.
>>>
>>> Hmm, perhaps I don't understand you.  doctest reports how many failures
> occur, without blocking on any single failure.
>


Right. But you  typically group a bunch of actions into a single "test". If
a doctest fails in an early action then every line after that will probably
fail - a single test failure will cause multiple *reported* failures.


>
>
>> The problem of being dependent on order of unorderable types (actually
>>> very difficult to solve).
>>>
>>
> Well, a crude solution is just to lift any output text that denotes an
> non-ordered type and pass it through an "eval" operation.
>


Not a general solution - not all reprs are reversible (in fact very few are
as a proportion of all objects).


>
>
>> Things like shared fixtures and mocking become *harder* (although by no
>>> means impossible) in a doctest environment.
>>>
>>>
> This, I think, what I was suggesting with doctest "scoping" where the
> execution environment is a matter of how nested the docstring is in
> relation to the "python semantic environment", with a final scope of
> "globs" that can be passed into the test environment, for anything with
> global scope.
>
>
>> Another thing I dislike is that it encourages a "test last" approach, as
>>> by far the easiest way of generating doctests is to copy and paste from the
>>> interactive interpreter. The alternative is lots of annoying typing of
>>> '>>>' and '...', and as you're editing text and not code IDE support tends
>>> to be worse (although this is a tooling issue and not a problem with
>>> doctest itself).
>>>
>>
> This is where I think the idea of having a test() built-in, like help(),
> would really be nice.  One could run test(myClass.mymethod) iterively while
> one codes, encouraging TDD and writing tests *along with* your code.   My
> TDD sense says it couldn't get any better.
>
>
>> More fundamental-ish problems:
>>
>>     Putting debugging prints into a function can break a myriad of tests
>> (because they're output based).
>>
>
> That's a good point.  But then it's a fairly simple matter of adding the
> output device:  'print >> stderr, 'here I am'", another possibility, if TDD
> were to become more of part of the language, is a special debug exception:
>  "raise Debug("Am at the test point, ", x)"  Such special exceptions could
> be caught and ignored by doctest.
>
>
>>     With multiple doctest blocks in a test file running an individual
>> test can be difficult (impossible?).
>>
>> This again solved with the test() built-in an making TDD something that
> is a feature of the language itself.
>

I don't fully follow you, but it shouldn't be hard to add this to doctest
and see if it is really useful.


>
>
>>     I may be misremembering, but I think debugging support is also
>> problematic because of the stdout redirection
>>
>
> Interesting, I try to pre-conceive tests well enough so I never need to
> invoke the debugger.
>

Heh. When I'm adding new features to existing code it is very common for me
to write a test that drops into the debugger after setting up some state -
and potentially using the test infrastructure (fixtures, django test client
perhaps, etc). So not being able to run a single test or drop into a
debugger puts the kybosh on that.

Michael


>
>
>> So yeah. Not a huge fan.
>>
>> That's good feedback.  Thanks.
>
> Mark
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120227/5b1e5d96/attachment.html>