[Python-ideas] doctest

[Steven D'Aprano]

Mark Janssen wrote:

> I just thought of something:  isn't the obvious solution is for doctest
> to test the type of an expression's output and if it's in the set of
> unordered types {set, dict} to run sort() on it?  Then the docstring author
> can just put the (sorted) output of what's expected....

{set, dict} is not the set of unordered types. The set of unordered types is 
without bound: anyone can create their own unordered types.

Even if you limit yourself to the builtins, you forgot frozenset. And then 
there are non-builtins in the standard library, like OrderedDict, other types 
like dict_proxy.

Sorting the output of an OrderedDict is the wrong thing to do, because the 
order is significant. So doctest would need to not just recognise mappings and 
sets, and sort them, but *also* recognise mappings and sets which should *not* 
be sorted.

Remember too, that by the time doctest.OutputChecker sees the output, it only 
sees it as a string. I don't know how much work it would take to introduce 
actual type-checks into doctest, but I expect it would be a lot.

And one last problem for you to consider: what happens if the output is 
unsortable? Try this dict for size:

{ 2+1j: None, 2-1j: None, float('NAN'): None}


> Perhaps I'm hashing a dead horse, but I really would like to see this added
> to the issue's tracker as a requested feature.  I may code up the patch
> myself, but it helps my brain to have it "on the dev stack".

Feel free to add it.

For what it's worth, I am a very strong -1 on any suggestion to give doctest 
"Do What I Mean" powers when it comes to unordered objects. But I would 
support a "Do What I Say" doctest directive, like NORMALIZE_WHITESPACE, 
ELLIPSIS, IGNORE_EXCEPTION_DETAIL, e.g. a directive that tells doctest to 
split both the expected and actual output strings on whitespace, then 
lexicographically sort them before comparing.

This approach doesn't try to be too clever: it's a dumb, understandable test 
which should fit in nicely with the other tests in 
doctest.OutputChecker.check_output, perhaps something like this:


if optionflags & IGNORE_WORD_ORDER:
     if sorted(got.split()) == sorted(want.split()):
         return True


It won't solve every doctest ordering problem, but doctest has other 
heuristics which can be fooled too. It is nice and simple, it solves the first 
90% of the problem, and it is under the control of the coder.

If you feel like submitting a patch, feel free to use my idea.



-- 
Steven