[Python-Dev] Purpose of Doctests [Was: Best practices for Enum]

Mon May 20 18:27:33 CEST 2013

Hi !
:)

I'll be replying some individual messages in this thread in spite of
putting my replies in the right context . Sorry if I repeat something
, or this makes the thread hard to read . Indeed , IMHO this is a
subject suitable to discuss in TiP ML .

On 5/19/13, Gregory P. Smith <greg at krypto.org> wrote:
> On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger <
> raymond.hettinger at gmail.com> wrote:
>
>>
>> On May 14, 2013, at 9:39 AM, Gregory P. Smith <greg at krypto.org> wrote:
>>
>> Bad: doctests.
>>
>>
>> I'm hoping that core developers don't get caught-up in the "doctests are
>> bad meme".
>>
>
> So long as doctests insist on comparing the repr of things being the number
> one practice that people use when writing them there is no other position I
> can hold on the matter.  reprs are not stable and never have been.
>  ordering changes, hashes change, ids change, pointer values change,
> wording and presentation of things change.  none of those side effect
> behaviors were ever part of the public API to be depended on.
>

«Bad doctests» slogan is not positive because the subliminal message
for new users is «there's something wrong with that ... let's better
not use it» . IMHO that's not true ; doctest is an incredibly awesome
testing framework for delta assertions and there is nothing wrong with
the philosophy behind that module and its intent .

This surfaces an issue I've noticed years ago wrt doctest module (so,
yes , it's obvious there's an issue ;) . The way I see it this is more
about the fact that module frontend does not offer the means to
benefit from all the possibilities of doctest classes in the backend
(e.g. output checkers , doctest runners, ...)

> That one can write doctests that don't depend on such things as the repr
> doesn't ultimately matter because the easiest thing to do, as encouraged by
> examples that are pasted from an interactive interpreter session into docs,
> is to have the interactive interpreter show the repr and not add code to
> check things in a accurate-for-testing manner that would otherwise make the
> documentation harder for a human to read.
>

This is something that could be easily mitigated by a custom output
checker . In the end , in docs there is no difference between output
messages like '<Spam object at 0xb7cf5d70>' or '<Spam object at
0x00001>' (i.e. some deterministic label like computed hex number or
anything else ...) . You might also avoid printing repr(s)

>> Instead, we should be clear about their primary purpose which is to test
>> the examples given in docstrings.   In many cases, there is a great deal
>> of benefit to docstrings that have worked-out examples (see the
>> docstrings
>> in the decimal module for example).  In such cases it is also worthwhile
>> to make sure those examples continue to match reality. Doctests are
>> a vehicle for such assurance.  In other words, doctests have a perfectly
>> legitimate use case.
>>
>
> I really do applaud the goal of keeping examples in documentation up to
> date.  But doctest as it is today is the wrong approach to that. A repr
> mismatch does not mean the example is out of date.
>

... and I confess I never use doctest «as it is today» in stdlib . So
, you are right .

> We should continue to encourage users to make thorough unit tests
>> and to leave doctests for documentation.  That said, it should be
>> recognized that some testing is better than no testing.  And doctests
>> may be attractive in that regard because it is almost effortless to
>> cut-and-paste a snippet from the interactive prompt.  That isn't a
>> best practice, but it isn't a worst practice either.
>>
>
> Not quite, they at least tested something (yay!) but it is uncomfortably
> close to a worst practice.
>

I disagree . IMO what is a bad practice is to spread the rumor that
«doctests are evil» rather than saying «doctest module has
limitations»

> It means someone else needs to come understand the body of code containing
> this doctest when they make an unrelated change that triggered a behavior
> change as a side effect that the doctested code may or may not actually
> depend on but does not actually declare its intent one way or another for
> the purposes of being a readable example rather than accurate test.
>

I see no problem in keeping both these aspects .

> bikeshed colors: If doctest were never called a test but instead were
> called docchecker to not imply any testing aspect

No way ! ( IMHO )

I just wrote dutest [1]_ framework , built on top of doctest and
unittest , that does the following (among other things) :

  1. Implements unittest loaders for doctests
  2. Allows for customizing output checkers , doctest runners , ...
      anything you might find in the backend
     * For instance , replacing default test runner and output checkers
       might be useful to write delta assertions for command-line scripts
  3. Tightly integrated with unittest (e.g. custom TestSuite(s) ...)
  4. Access to unittest test case in special __tc__ variable , so all
      known assertion methods are handy ootb
  5. Encapsulate doctest setup code (setUp , tearDown for doctests)
      e.g. to make doctests like
      the following work without actually writing in the docs all steps
      needed to load

"""Provided that «complex scenario» is prepared and satisfies some
preconditions (performed in hidden unittest setup methods)

>>> do_something()
>>> do_something_else()
>>> ...
"""

I can report usage of dutest module in practice to test non-trivial
web apps . I've even started to write a micro-framework on top of it
to test Trac plugins (based on 3. , 4. , 5. above + twill ) , and it's
possible to do the same thing for other web frameworks . Use cases
might not be restricted to web apps ; like I mentioned above , custom
output checkers + doctest runners will make it possible to test cli  .

... so , in a few words , delta assertions (like doctests) are about
testing and are much more than a doc checker.

> that might've helped (too
> late? the cat's out of the bag).

just to create more confusion ... afaict

> Or if it never compared anything but
> simply ran the example code to generate and update the doc examples from
> the statements with the current actual results of execution instead of
> doing string comparisons...  (ie: more of an documentation example "keep up
> to date" tool)
>

Considering what I mentioned above , I disagree ...

[...]
>
> In my earlier message I suggested that someone improve doctest to not do
> dumb string comparisons of reprs.

FWIW , that's possible with dutest [1]_ , indeed one of the main goals
it was created for .

> I still think that is a good goal if
> doctest is going to continue to be promoted. It would help alleviate many
> of the issues with doctests and bring them more in line with the issues
> many people's regular unittests have.
>
> As Tres already showed in an example,
> individual doctest using projects jump through hoops to do some of that
> today; centralizing saner repr comparisons for less false failures as an
> actual doctest feature just makes sense.
>

+1
Besides , IMHO doctest is very useful for APIs and lowers the barrier
for writing testing code for people not used to XUnit philosophy .

> Successful example: We added a bunch of new comparison methods to unittest
> in 2.7 that make it much easier to write tests that don't depend on
> implementation details such as ordering. Many users prefer to use those new
> features; even with older Python's via unittest2 on pypi. It doesn't mean
> users always write good tests, but a higher percentage of tests written are
> more future proof than they were before because it became easier.
>

all this is possible with dutest [1]_ . Assertion methods of the
underlying unittest test case are available in __tc__ variable e.g.

"""
>>> __tc__.assertEqual(x, y)
"""

so both approaches to testing may be combined ... and everybody
*should* be happy .

[...]

.. [1] dutest module @ PyPI
        (https://pypi.python.org/pypi/dutest)

-- 
Regards,

Olemis.

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/