On 27 February 2012 20:35, Michael Foord wrote:

On 18 February 2012 04:24, Ian Bicking wrote:

...

On Feb 17, 2012 4:12 PM, "Nick Coghlan" wrote:

...

On Sat, Feb 18, 2012 at 7:57 AM, Mark Janssen <

dreamingforward@gmail.com> wrote:

...

...

Anyway... of course patches welcome, yes... ;^)

Not really. doctest is for *testing code example in docs*. If you try to use it for more than that, it's likely to drive you up the wall, so proposals to make it more than it is usually don't get a great reception (docs patches to make it's limitations clearer are generally welcome, though). The stdib solution for test driven development is unittest (the vast majority of our own regression suite is written that way - only a small proportion uses doctest).

This pessimistic attitude is why doctest is challenging to work with at times, not anything to do with doctest's actual model. The constant criticisms of doctest keep contributors away, and keep its many resolvable problems from being resolved.

Personally I think there are several fundamental problems with doctest *as a unit testing tool*. doctest is *awesome* for testing documentation examples but in particular this one:

* Every line becomes an assertion - in a unit test you typically follow the arrange -> act -> assert pattern. Only the results of the *assertion* are relevant to the test. (Obviously unexpected exceptions at any stage are relevant....). With doctest you have to take care to ensure that the exact output of *every line* of your arrange and act steps also match, even if they are irrelevant to your assertion. (The arrange and act steps will often include lines where you are creating state, and their output is irrelevant so long as they put the right things in place.)

The particular implementation of doctest means that there are additional, potentially resolvable problems that are also a damn nuisance in a unit testing fail:

Jeepers, I changed direction mid-sentence there. It should have read something along the lines of: As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:

Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.

The problem of being dependent on order of unorderable types (actually very difficult to solve).

Things like shared fixtures and mocking become *harder* (although by no means impossible) in a doctest environment.

Another thing I dislike is that it encourages a "test last" approach, as by far the easiest way of generating doctests is to copy and paste from the interactive interpreter. The alternative is lots of annoying typing of '>>>' and '...', and as you're editing text and not code IDE support tends to be worse (although this is a tooling issue and not a problem with doctest itself).

More fundamental-ish problems: Putting debugging prints into a function can break a myriad of tests (because they're output based). With multiple doctest blocks in a test file running an individual test can be difficult (impossible?). I may be misremembering, but I think debugging support is also problematic because of the stdout redirection. So yeah. Not a huge fan. All the best, Michael

So whilst I'm not against improving doctest, I don't promote it as a unit testing tool and disagree that it is suited to that task.

All the best,

Michael Foord

...

...

An interesting third party alternative that has been created recently is behave: http://crate.io/packages/behave/

This style of test is why it's so sad that doctest is ignored and unmaintained. It's based on testing patterns developed by people who care to promote what they are doing, but I'm of the strong opinion that they are inferior to doctest.

Ian

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

--

http://www.voidspace.org.uk/

May you do good and not evil May you find forgiveness for yourself and forgive others

May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord wrote:

On 27 February 2012 20:35, Michael Foord wrote:

...

Personally I think there are several fundamental problems with doctest *as a unit testing tool*. doctest is *awesome* for testing documentation examples but in particular this one:

* Every line becomes an assertion - in a unit test you typically follow the arrange -> act -> assert pattern. Only the results of the *assertion* are relevant to the test. (Obviously unexpected exceptions at any stage are relevant....). With doctest you have to take care to ensure that the exact output of *every line* of your arrange and act steps also match, even if they are irrelevant to your assertion. (The arrange and act steps will often include lines where you are creating state, and their output is irrelevant so long as they put the right things in place.)

The particular implementation of doctest means that there are additional, potentially resolvable problems that are also a damn nuisance in a unit testing fail:

Jeepers, I changed direction mid-sentence there. It should have read something along the lines of:

As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:

...

The problem of being dependent on order of unorderable types (actually very difficult to solve).

I just thought of something: isn't the obvious solution is for doctest

to test the type of an expression's output and if it's in the set of unordered types {set, dict} to run sort() on it? Then the docstring author can just put the (sorted) output of what's expected.... Perhaps I'm hashing a dead horse, but I really would like to see this added to the issue's tracker as a requested feature. I may code up the patch myself, but it helps my brain to have it "on the dev stack". mark

On Thu, Mar 1, 2012 at 8:49 PM, Devin Jeanpierre wrote:

On Thu, Mar 1, 2012 at 9:31 PM, Mark Janssen wrote:

...

I just thought of something: isn't the obvious solution is for doctest to test the type of an expression's output and if it's in the set of unordered types {set, dict} to run sort() on it? Then the docstring author can just put the (sorted) output of what's expected....

It's the solution people seem to think of first, so it's definitely fairly obvious. It's not a big improvement, though. The original problem is that dict order shouldn't matter, but in doctest it does, making dicts unusable in the normal doctest style. Making it a specific dict order be the prominent one lets you use dicts in doctest, but you have to sort the dicts and rewrite the doctest to take that into account.

Personally I never copy from the interactive prompt, but instead write my doctest and, if I'm in the mood to copy and paste, copy from the failed doctest error message (which is nicely indented just like my tests). So it would work fine if things were sorted.

And even so, some dicts cannot be represented this way. For example, the dict {1:2, "hello": world} cannot be sorted in Python 3, so it won't work in this scheme.

You could always use a heuristic sorting, e.g., sorted((str(key), key) for key in dict) To make this work you have to write a repr() replacement that is somewhat sophisticated. Though it still wouldn't save you from: class Something: def __repr__(self): return '<Something attr=%r>' % (self.attr) where attr is a dict or some other object with an awkward repr. That's the part I'm unsure of. Of course no eval will help you there either. I don't know if there's any way to really replace repr's implementation everywhere; I'm guessing there isn't. Ian

On Thu, Mar 1, 2012 at 6:31 PM, Mark Janssen wrote:

I just thought of something:  isn't the obvious solution is for doctest to test the type of an expression's output and if it's in the set of unordered types {set, dict} to run sort() on it?  Then the docstring author can just put the (sorted) output of what's expected....

What if the output is a list of dicts? -- --Guido van Rossum (python.org/~guido)

Mark Janssen wrote:

I just thought of something: isn't the obvious solution is for doctest to test the type of an expression's output and if it's in the set of unordered types {set, dict} to run sort() on it? Then the docstring author can just put the (sorted) output of what's expected....

{set, dict} is not the set of unordered types. The set of unordered types is without bound: anyone can create their own unordered types. Even if you limit yourself to the builtins, you forgot frozenset. And then there are non-builtins in the standard library, like OrderedDict, other types like dict_proxy. Sorting the output of an OrderedDict is the wrong thing to do, because the order is significant. So doctest would need to not just recognise mappings and sets, and sort them, but *also* recognise mappings and sets which should *not* be sorted. Remember too, that by the time doctest.OutputChecker sees the output, it only sees it as a string. I don't know how much work it would take to introduce actual type-checks into doctest, but I expect it would be a lot. And one last problem for you to consider: what happens if the output is unsortable? Try this dict for size: { 2+1j: None, 2-1j: None, float('NAN'): None}

Perhaps I'm hashing a dead horse, but I really would like to see this added to the issue's tracker as a requested feature. I may code up the patch myself, but it helps my brain to have it "on the dev stack".

Feel free to add it. For what it's worth, I am a very strong -1 on any suggestion to give doctest "Do What I Mean" powers when it comes to unordered objects. But I would support a "Do What I Say" doctest directive, like NORMALIZE_WHITESPACE, ELLIPSIS, IGNORE_EXCEPTION_DETAIL, e.g. a directive that tells doctest to split both the expected and actual output strings on whitespace, then lexicographically sort them before comparing. This approach doesn't try to be too clever: it's a dumb, understandable test which should fit in nicely with the other tests in doctest.OutputChecker.check_output, perhaps something like this: if optionflags & IGNORE_WORD_ORDER: if sorted(got.split()) == sorted(want.split()): return True It won't solve every doctest ordering problem, but doctest has other heuristics which can be fooled too. It is nice and simple, it solves the first 90% of the problem, and it is under the control of the coder. If you feel like submitting a patch, feel free to use my idea. -- Steven

It seems that the problem with any solution based on interpreting repr (especially when nothing in know about the object) is that there are just too many exceptions. Another approach might be to allow for a custom compare function to be defined on doctest. E.g., in the module to be tested: import doctest def _compare(got, expected): return (sorted(eval(got)) == sorted(eval(expected)) or doctest.compare(got, expected)) doctest.usercompare = _compare The compare function would only need to deal with the idiosyncrasies of types actually used in doctests in that module. I don't know how practical this idea is in terms of implementation - from my brief look through the code I think it should be fairly easy to slot this into OutputChecker, but it was a very brief look! David On Fri, Mar 2, 2012 at 7:36 AM, Steven D'Aprano wrote:

Steven D'Aprano wrote:

This approach doesn't try to be too clever: it's a dumb, understandable

...

test which should fit in nicely with the other tests in doctest.OutputChecker.check_**output, perhaps something like this:

if optionflags & IGNORE_WORD_ORDER: if sorted(got.split()) == sorted(want.split()): return True

Ah, buggarit, too simple. I neglected to take into account the delimiters.

Getting this right is harder than I thought, particularly with nested sets/dicts.

Still, I reckon a directive is the right approach.

-- Steven

______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideashttp://mail.python.org/mailman/listinfo/python-ideas

Le 02/03/2012 07:55, Devin Jeanpierre a écrit :

[...] dicts and sets are not idiosyncratic or in any way exceptional. doctest itself should handle them the way a naive user would expect.

This discussion seems to forget a core issue with doctest: The output lines can be *anything* that gets printed. eval-able reprs of Python objects are only a part of the possibilities. That’s why doctest cannot “just call sorted” on the output lines. Regards

On Fri, Mar 2, 2012 at 8:05 AM, Devin Jeanpierre wrote:

   This is totally fine        >>> {-1:1, -2:1} # doctest: +LITERAL_EVAL        {-1:1, -2:1}

Doing an ast.literal_eval seems like a great feature (and maybe even a sensible default after a string comparison fails). Having a real eval as an explicit doctest option would also make doctest more powerful. Mike

On Fri, Mar 2, 2012 at 6:07 AM, David Townshend wrote:

That was just a quick example of another approach to the problem. Sure, there are some issues to work out, but I don't believe this is an insurmountable problem.

Nor do I. I was attempting to offer constructive criticism on the basis that this is a serious suggestion, and deserves attention. Sorry that I gave the wrong impression.

Once again, I'm sure we could find a way around this. Perhaps it would also be acceptable to define the compare function inside the docstring, or in this case inside a rst comment.

I mentioned it earlier, but I think you missed it: I was actually thinking inside the doctest itself. Your system of global assignment works as-is if you do it inside the doctests themselves (except for threads, but who runs doctests in multiple threads? gah!)

So what about, say, a defaultdict or a WeakSet? What exactly would a naive user expect?

WeakSets shouldn't be tested like this, their contents are nondeterministic. Any expectations can be broken by unfortunate race conditions, and there is no way around this. Sometimes users might expect the impossible, but what they expect is irrelevant in such a case. So forget WeakSets. I think that a naive user would expect, w.r.t. all these things, that he copy-pasted a shell session, it would "just work". Where we can fix doctest to align with such lofty expectations, at least in the common cases -- such as with dicts and defaultdicts and so on -- is it really so bad? Or, for a different argument -- surely, if the natural way to write an example in a tutorial is to show a dict as a return value, then we should be able to test that with minimal fuss? Doctest is supposed to serve the documentation, not the other way around; and the harder things are, the higher the barrier to entry is, and the fewer people actually do it. Because testing is important, it's important that testing be as easy as reasonable and possible. -- Devin

Devin Jeanpierre wrote:

...

On Fri, Mar 2, 2012 at 7:36 AM, Steven D'Aprano wrote:

...

Still, I reckon a directive is the right approach.

Why? That's how I do it because I am/was paranoid about compatibility, but surely fixing dicts is important enough that, done right, nobody would object if the semantics of comparison change subtly to allow for unordered container comparison in the "natural" way?

I would. doctest doesn't compare dicts. If you want to compare dicts, write your doctest like this:

...

...

d = make_dict(...) d == {'a': 42, 'b': 23, 'c': None} True

The dicts will be compared directly, and doctest need only care that the result looks like "True". doctest compares *strings*. Under no circumstances do I want the default doctest comparison to try to be "clever" by guessing when I want strings to match using string equality and when I want strings to match using something else. doctest should Do What I Say, and not try to Do What It Guesses I Mean. [...]

Punting it to a user-defined function is nice for _really_ crazy situations, but dicts and sets are not idiosyncratic or in any way exceptional. doctest itself should handle them the way a naive user would expect.

No it shouldn't. doctest should handle them the way they actually are. A naive user might expect that {'None': 1} == {None: 1}. A naive user might expect that {2: 'spam'} == {2: 'SPAM'}. The problem with trying to satisfy naive users is that their expectations are often wrong or poorly thought-out. You, the author of the package being tested, is the best person to judge what your tests should be, not some arbitrary naive user who may know nothing about Python and even less about your package. -- Steven

Devin, You need to start writing real code rather than continue to tell us that the problems are minor and easily fixable, and the solutions are uncontroversial. To those who have tried and thought about it, the problems are *not* easy to solve , except for some superficial edge cases that you and other critics of doctest keep focusing on. And please don't propose that we change the behavior of dict or other data types itself, or add new APIs to objects just for the purpose of "fixing" doctest's issues. -- --Guido van Rossum (python.org/~guido)

On Fri, Mar 2, 2012 at 9:59 AM, Devin Jeanpierre wrote:

On Fri, Mar 2, 2012 at 12:30 PM, Guido van Rossum wrote:

...

Devin,

You need to start writing real code rather than continue to tell us that the problems are minor and easily fixable, and the solutions are uncontroversial. To those who have tried and thought about it, the problems are *not* easy to solve , except for some superficial edge cases that you and other critics of doctest keep focusing on.

I already did write real code. In the context of this discussion, I implemented a +LITERAL_EVAL flag.

Was there something else I was supposed to write, other than the solution I advocated? ;)

https://bitbucket.org/devin.jeanpierre/doctest2/src/e084a682ccbc/doctest2/co...

It's not a solution. It's a hack that only works in the simplest cases -- it requires the output to look like a Python expression (that can be evaluated in a limited environment). What if the output were something like ??? There's a dict in there but the whole thing is not parseable.

...

And please don't propose that we change the behavior of dict or other data types itself, or add new APIs to objects just for the purpose of "fixing" doctest's issues.

I would never dream of it. That's pretty obscene.

Good. I wasn't sure what you meant when you used the phrase "fix dict" -- I presume that was shorthand for "fix the problem that doctest has with dict". -- --Guido van Rossum (python.org/~guido)

Can I ask a possibly silly question? As I understand it, doctest takes a small snippet of code, runs it, and compares the resulting string with a string in the document. This thread seems to be centered around making comparisons of results with indeterminate ordering (dicts being the prime example) work properly. In fact, one proposal was to have doctest call sorted on the output to make sure it's right, which was shot down because that's not always the correct thing to do. So the question is - why isn't dealing with this the responsibility of the test writer? Yeah, it's not quite the spirit of documentation to turn a dictionary into a sorted list in the output, but neither is littering the documentation with +LITERAL_EVAL and the like. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

On Fri, Mar 2, 2012 at 1:27 PM, Mike Meyer wrote:

So the question is - why isn't dealing with this the responsibility of the test writer? Yeah, it's not quite the spirit of documentation to turn a dictionary into a sorted list in the output, but neither is littering the documentation with +LITERAL_EVAL and the like.

Currently the test-writer is not empowered to exercise this responsibility most effectively. Earlier, an example was presented that one should write

...

...

d = make_dict(...) d == {'a': 42, 'b': 23, 'c': None} True

rather than the implied

...

...

d = make_dict(...) {'a': 42, 'b': 23, 'c': None}

Using the former is certainly necessary today, but it's far from ideal. The documentor is stuck either writing the documentation clearly (the latter case) or testably (the former). The hope is to find a way to let people more easily write documentation that is both clear and testable. One thing that covers some--but far from all--cases is to ast.literal_eval or eval the output, since very often that output line is a repr of a Python expression. To be reasonable, this might require a directive in both cases and certainly requires one in the latter case. Though there are still tons of situations this cannot cover, it would allow people writing documentation to avoid ugly constructs like the first code snippet in a not-tiny set of cases. As for littering your code with "+doctest BLAH_BLAH", I don't think this is all that harmful. It allows the documentation writer get features she wants and will not display to the user in the processed documentation. There are already directives like this today and though ugly, they are conventional. Mike

On Fri, Mar 2, 2012 at 11:14 AM, Guido van Rossum wrote:

On Fri, Mar 2, 2012 at 9:59 AM, Devin Jeanpierre wrote:

...

On Fri, Mar 2, 2012 at 12:30 PM, Guido van Rossum wrote:> Was there something else I was supposed to write, other than the solution I advocated? ;)

https://bitbucket.org/devin.jeanpierre/doctest2/src/e084a682ccbc/doctest2/co...

It's not a solution. It's a hack that only works in the simplest cases

With all respect to Guido, who has mentioned probably the best solution so far (using sys.displayhook()) , on Devin's behalf, I must say that for those of us dedicated to TDD using doctest, we tend to be writing code in tandem to an ideal that doctest engenders; generally speaking: everything being written is so fine grained, that the problems that most people are speaking of, never arise. E. g., the tests are small and uncomplicated because they are written right to the core of the source, there is semi-rigid protocol for __repr__ in line with being eval'able, and code is broken apart so that when the tests gets too complicated, it's a sign that the code is not modular or fine-grained enough. Just as the way OOP eventually evolved into the ideal of the "abstract base class" as a way to match the notion of physical objects with programmatic ones. Python + doctest approach a different apex, catching two birds with one stone: documentation + TDD. Just my small input... mark

Barry Warsaw writes:

Actually, Emacs users should use rst-mode, which has no so bad support for separate file doctests. Of course, the mode is useful for reST documentation even if your documentation is untested <wink>.

Any idea where I should send bug reports for ‘rst-mode’? It's not clear to me who develops it. -- \ “Welchen Teil von ‘Gestalt’ verstehen Sie nicht? [What part of | `\ ‘gestalt’ don't you understand?]” —Karsten M. Self | _o__) | Ben Finney