On 27 February 2012 20:35, Michael Foord <fuzzyman@gmail.com> wrote:
On 18 February 2012 04:24, Ian Bicking <ianb@colorstudy.com> wrote:
On Feb 17, 2012 4:12 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On Sat, Feb 18, 2012 at 7:57 AM, Mark Janssen <
dreamingforward@gmail.com> wrote:
Anyway... of course patches welcome, yes... ;^)
Not really. doctest is for *testing code example in docs*. If you try to use it for more than that, it's likely to drive you up the wall, so proposals to make it more than it is usually don't get a great reception (docs patches to make it's limitations clearer are generally welcome, though). The stdib solution for test driven development is unittest (the vast majority of our own regression suite is written that way - only a small proportion uses doctest).
This pessimistic attitude is why doctest is challenging to work with at times, not anything to do with doctest's actual model. The constant criticisms of doctest keep contributors away, and keep its many resolvable problems from being resolved.
Personally I think there are several fundamental problems with doctest *as a unit testing tool*. doctest is *awesome* for testing documentation examples but in particular this one:
* Every line becomes an assertion - in a unit test you typically follow the arrange -> act -> assert pattern. Only the results of the *assertion* are relevant to the test. (Obviously unexpected exceptions at any stage are relevant....). With doctest you have to take care to ensure that the exact output of *every line* of your arrange and act steps also match, even if they are irrelevant to your assertion. (The arrange and act steps will often include lines where you are creating state, and their output is irrelevant so long as they put the right things in place.)
The particular implementation of doctest means that there are additional, potentially resolvable problems that are also a damn nuisance in a unit testing fail:
Jeepers, I changed direction mid-sentence there. It should have read something along the lines of: As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:
Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.
The problem of being dependent on order of unorderable types (actually very difficult to solve).
Things like shared fixtures and mocking become *harder* (although by no means impossible) in a doctest environment.
Another thing I dislike is that it encourages a "test last" approach, as by far the easiest way of generating doctests is to copy and paste from the interactive interpreter. The alternative is lots of annoying typing of '>>>' and '...', and as you're editing text and not code IDE support tends to be worse (although this is a tooling issue and not a problem with doctest itself).
More fundamental-ish problems: Putting debugging prints into a function can break a myriad of tests (because they're output based). With multiple doctest blocks in a test file running an individual test can be difficult (impossible?). I may be misremembering, but I think debugging support is also problematic because of the stdout redirection. So yeah. Not a huge fan. All the best, Michael
So whilst I'm not against improving doctest, I don't promote it as a unit testing tool and disagree that it is suited to that task.
All the best,
Michael Foord
An interesting third party alternative that has been created recently is behave: http://crate.io/packages/behave/
This style of test is why it's so sad that doctest is ignored and unmaintained. It's based on testing patterns developed by people who care to promote what they are doing, but I'm of the strong opinion that they are inferior to doctest.
Ian
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
--
May you do good and not evil May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html