On 27 February 2012 23:23, Mark Janssen
On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord
wrote: As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:
Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.
Hmm, perhaps I don't understand you. doctest reports how many failures
occur, without blocking on any single failure.
Right. But you typically group a bunch of actions into a single "test". If a doctest fails in an early action then every line after that will probably fail - a single test failure will cause multiple *reported* failures.
The problem of being dependent on order of unorderable types (actually
very difficult to solve).
Well, a crude solution is just to lift any output text that denotes an non-ordered type and pass it through an "eval" operation.
Not a general solution - not all reprs are reversible (in fact very few are as a proportion of all objects).
Things like shared fixtures and mocking become *harder* (although by no
means impossible) in a doctest environment.
This, I think, what I was suggesting with doctest "scoping" where the execution environment is a matter of how nested the docstring is in relation to the "python semantic environment", with a final scope of "globs" that can be passed into the test environment, for anything with global scope.
Another thing I dislike is that it encourages a "test last" approach, as
by far the easiest way of generating doctests is to copy and paste from the interactive interpreter. The alternative is lots of annoying typing of '>>>' and '...', and as you're editing text and not code IDE support tends to be worse (although this is a tooling issue and not a problem with doctest itself).
This is where I think the idea of having a test() built-in, like help(), would really be nice. One could run test(myClass.mymethod) iterively while one codes, encouraging TDD and writing tests *along with* your code. My TDD sense says it couldn't get any better.
More fundamental-ish problems:
Putting debugging prints into a function can break a myriad of tests (because they're output based).
That's a good point. But then it's a fairly simple matter of adding the output device: 'print >> stderr, 'here I am'", another possibility, if TDD were to become more of part of the language, is a special debug exception: "raise Debug("Am at the test point, ", x)" Such special exceptions could be caught and ignored by doctest.
With multiple doctest blocks in a test file running an individual test can be difficult (impossible?).
This again solved with the test() built-in an making TDD something that
is a feature of the language itself.
I don't fully follow you, but it shouldn't be hard to add this to doctest and see if it is really useful.
I may be misremembering, but I think debugging support is also problematic because of the stdout redirection
Interesting, I try to pre-conceive tests well enough so I never need to invoke the debugger.
Heh. When I'm adding new features to existing code it is very common for me to write a test that drops into the debugger after setting up some state - and potentially using the test infrastructure (fixtures, django test client perhaps, etc). So not being able to run a single test or drop into a debugger puts the kybosh on that. Michael
So yeah. Not a huge fan.
That's good feedback. Thanks.
Mark
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
participants (4)
-
Barry Warsaw
-
Ian Bicking
-
Michael Foord
-
Nick Coghlan