New subject: doctest

Feb. 27, 2012


      On 27 February 2012 23:23, Mark Janssen <dreamingforward@gmail.com> wrote:
...
On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord <fuzzyman@gmail.com> wrote:
...
As well as fundamental problems, the particular implementation of doctest
suffers from these potentially resolvable problems:
...
Execution of an individual testing section continues after a failure. So
a single failure results in the *reporting* of potentially many failures.
Hmm, perhaps I don't understand you.  doctest reports how many failures
occur, without blocking on any single failure.
Right. But you  typically group a bunch of actions into a single "test". If
a doctest fails in an early action then every line after that will probably
fail - a single test failure will cause multiple *reported* failures.
...
...
The problem of being dependent on order of unorderable types (actually
...
very difficult to solve).
Well, a crude solution is just to lift any output text that denotes an
non-ordered type and pass it through an "eval" operation.
Not a general solution - not all reprs are reversible (in fact very few are
as a proportion of all objects).
...
...
Things like shared fixtures and mocking become *harder* (although by no
...
means impossible) in a doctest environment.
This, I think, what I was suggesting with doctest "scoping" where the
execution environment is a matter of how nested the docstring is in
relation to the "python semantic environment", with a final scope of
"globs" that can be passed into the test environment, for anything with
global scope.
...
Another thing I dislike is that it encourages a "test last" approach, as
...
by far the easiest way of generating doctests is to copy and paste from the
interactive interpreter. The alternative is lots of annoying typing of
'>>>' and '...', and as you're editing text and not code IDE support tends
to be worse (although this is a tooling issue and not a problem with
doctest itself).
This is where I think the idea of having a test() built-in, like help(),
would really be nice.  One could run test(myClass.mymethod) iterively while
one codes, encouraging TDD and writing tests *along with* your code.   My
TDD sense says it couldn't get any better.
...
More fundamental-ish problems:
Putting debugging prints into a function can break a myriad of tests
(because they're output based).
That's a good point.  But then it's a fairly simple matter of adding the
output device:  'print >> stderr, 'here I am'", another possibility, if TDD
were to become more of part of the language, is a special debug exception:
 "raise Debug("Am at the test point, ", x)"  Such special exceptions could
be caught and ignored by doctest.
...
With multiple doctest blocks in a test file running an individual
test can be difficult (impossible?).
This again solved with the test() built-in an making TDD something that
is a feature of the language itself.
I don't fully follow you, but it shouldn't be hard to add this to doctest
and see if it is really useful.
...
...
I may be misremembering, but I think debugging support is also
problematic because of the stdout redirection
Interesting, I try to pre-conceive tests well enough so I never need to
invoke the debugger.
Heh. When I'm adding new features to existing code it is very common for me
to write a test that drops into the debugger after setting up some state -
and potentially using the test infrastructure (fixtures, django test client
perhaps, etc). So not being able to run a single test or drop into a
debugger puts the kybosh on that.

Michael
...
...
So yeah. Not a huge fan.
That's good feedback.  Thanks.
Mark
-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

Re: [Python-ideas] doctest

Michael Foord

Ian Bicking

Nick Coghlan

Barry Warsaw

Ian Bicking

Nick Coghlan

Barry Warsaw

tags

participants (4)