On 27 February 2012 23:23, Mark Janssen
On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord
wrote: As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:
Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.
Hmm, perhaps I don't understand you. doctest reports how many failures
occur, without blocking on any single failure.
Right. But you typically group a bunch of actions into a single "test". If a doctest fails in an early action then every line after that will probably fail - a single test failure will cause multiple *reported* failures.
The problem of being dependent on order of unorderable types (actually
very difficult to solve).
Well, a crude solution is just to lift any output text that denotes an non-ordered type and pass it through an "eval" operation.
Not a general solution - not all reprs are reversible (in fact very few are as a proportion of all objects).
Things like shared fixtures and mocking become *harder* (although by no
means impossible) in a doctest environment.
This, I think, what I was suggesting with doctest "scoping" where the execution environment is a matter of how nested the docstring is in relation to the "python semantic environment", with a final scope of "globs" that can be passed into the test environment, for anything with global scope.
Another thing I dislike is that it encourages a "test last" approach, as
by far the easiest way of generating doctests is to copy and paste from the interactive interpreter. The alternative is lots of annoying typing of '>>>' and '...', and as you're editing text and not code IDE support tends to be worse (although this is a tooling issue and not a problem with doctest itself).
This is where I think the idea of having a test() built-in, like help(), would really be nice. One could run test(myClass.mymethod) iterively while one codes, encouraging TDD and writing tests *along with* your code. My TDD sense says it couldn't get any better.
More fundamental-ish problems:
Putting debugging prints into a function can break a myriad of tests (because they're output based).
That's a good point. But then it's a fairly simple matter of adding the output device: 'print >> stderr, 'here I am'", another possibility, if TDD were to become more of part of the language, is a special debug exception: "raise Debug("Am at the test point, ", x)" Such special exceptions could be caught and ignored by doctest.
With multiple doctest blocks in a test file running an individual test can be difficult (impossible?).
This again solved with the test() built-in an making TDD something that
is a feature of the language itself.
I don't fully follow you, but it shouldn't be hard to add this to doctest and see if it is really useful.
I may be misremembering, but I think debugging support is also problematic because of the stdout redirection
Interesting, I try to pre-conceive tests well enough so I never need to invoke the debugger.
Heh. When I'm adding new features to existing code it is very common for me to write a test that drops into the debugger after setting up some state - and potentially using the test infrastructure (fixtures, django test client perhaps, etc). So not being able to run a single test or drop into a debugger puts the kybosh on that. Michael
So yeah. Not a huge fan.
That's good feedback. Thanks.
Mark
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mon, Feb 27, 2012 at 5:31 PM, Michael Foord
On 27 February 2012 23:23, Mark Janssen
wrote: On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord
wrote: As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:
Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.
Hmm, perhaps I don't understand you. doctest reports how many failures
occur, without blocking on any single failure.
Right. But you typically group a bunch of actions into a single "test". If a doctest fails in an early action then every line after that will probably fail - a single test failure will cause multiple *reported* failures.
The problem of being dependent on order of unorderable types (actually
very difficult to solve).
Well, a crude solution is just to lift any output text that denotes an non-ordered type and pass it through an "eval" operation.
Not a general solution - not all reprs are reversible (in fact very few are as a proportion of all objects).
Just an implementation suggestion - Guido's suggestion of using
sys.displayhook will work to change the repr of objects (I had never heard
of it until then, and had to test to convince myself). Doctest needs
reliable repr's more than reversable repr's, and you can create them using
that. You'll still get a lot of
On Tue, Feb 28, 2012 at 9:44 AM, Ian Bicking
Just an implementation suggestion - Guido's suggestion of using sys.displayhook will work to change the repr of objects (I had never heard of it until then, and had to test to convince myself). Doctest needs reliable repr's more than reversable repr's, and you can create them using that. You'll still get a lot of
strings, which suck... but if you are committed to doctest then maybe better to provide good __repr__ methods on your custom objects! For doctest.js (where I implemented a number of changes I would have wanted for doctest in Python) I have found this sort of thing sufficient, but Javascript objects tend to be a little more bare and there aren't existing conventions for repr/print/etc, so I have some more flexibility in my implementation.
You can actually do some pretty cool doctest hacks via displayhook and excepthook. I created a hacked together doctest variant [1] years ago that could run doctests from ODT files and also pay attention to sys.excepthook/displayhook before deciding that the test had failed. [1] http://svn.python.org/view/sandbox/trunk/userref/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Feb 27, 2012, at 05:44 PM, Ian Bicking wrote:
Doctest needs reliable repr's more than reversable repr's, and you can create them using that. You'll still get a lot of
strings, which suck... but if you are committed to doctest then maybe better to provide good __repr__ methods on your custom objects!
+1 even if you don't use doctests! I can't tell you how many times adding a useful repr has vastly improved debugging. I urge everyone to flesh out your reprs with a little bit of useful information so you can quickly identify your instances at a pdb prompt. Cheers, -Barry
participants (4)
-
Barry Warsaw
-
Ian Bicking
-
Michael Foord
-
Nick Coghlan