[Python-Dev] Purpose of Doctests [Was: Best practices for Enum]

[Olemis Lang]

On 5/19/13, Steven D'Aprano <steve at pearwood.info> wrote:
> On 20/05/13 09:27, Gregory P. Smith wrote:
>> On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger <
>> raymond.hettinger at gmail.com> wrote:
>>
>>>
>>> On May 14, 2013, at 9:39 AM, Gregory P. Smith <greg at krypto.org> wrote:
>>>
>>> Bad: doctests.
>>>
>>>
>>> I'm hoping that core developers don't get caught-up in the "doctests are
>>> bad meme".
>>>
>>
>> So long as doctests insist on comparing the repr of things being the
>> number
>> one practice that people use when writing them there is no other position
>> I
>> can hold on the matter.  reprs are not stable and never have been.
>
> I think this *massively* exaggerates the "problem" with doc tests.

I agree , and it is a negative influence for beginners .

> I make
> heavy use of them, and have no problem writing doc tests that work in code
> running over multiple versions, including from 2.4 through 3.3. Objects that
> I write myself, I control the repr and can make it as stable as I wish. Many
> built-in types also have stable reprs. The repr for small ints is not going
> to change, the repr for floats like 0.5, 0.25, 0.125 etc. are stable and
> predictable, lists and tuples and strings all have stable well-defined
> reprs. Dicts are a conspicuous counter-example, but there are trivial
> work-arounds.
>

+1

> Doc tests are not limited to a simple-minded "compare the object's repr".

Yes

> You can write as much, or as little, scaffolding around the test as you
> need. If the scaffolding becomes too large, that's a sign that the test
> doesn't belong in documentation and should be moved out, perhaps into a unit
> test, or perhaps into a separate "literate testing" document that can be as
> big as necessary without overwhelming the doc string.
>

There is an alternate approach related to a feature of dutest [1]_ I
mentioned in a previous message (i.e. doctests setUp and tearDown
methods) . The main reason to desire to leave long doctests
scaffolding code out (e.g. loading a Trac environment, or setting up a
separate Python virtual environment , subversion repository , ... as
part of -unit, functional, ...- test setup ) is to focus on SUT / API
details , avoid repetition of some steps , and keep tests readable .
This code is moved to underlying unittest setUp method and it's still
possible to write readable doctests for the particular feature of the
SUT .

In general there's a need to find a balance to decide what should be
«hidden» in doctests fixture methods and what should be written in
doctests . Based on my experience there's no benefit in using unittest
over doctests

unittests :

  - are unreadable
  - require knowledge of XUnit , etc ...
  - Writing complex assertions might be hard and tedious

doctests:

  - are extremely readable
  - anybody familiar with the SUT could write tests
  - especially for modules that are meant to be used by persons
    who are not (professional / skilled) software developers
    encapsulating the use of a testing framework is a plus ;
    your test suite is «talking in users language»
    (/me not sure about stdlib ...)

>
>>   ordering changes, hashes change, ids change, pointer values change,
>> wording and presentation of things change.  none of those side effect
>> behaviors were ever part of the public API to be depended on.
>
> Then don't write doctests that depend on those things. It really is that
> simple. There's no rule that says doctests have to test the entire API.
> Doctests in docstrings are *documentation first*, so you write tests that
> make good documentation.
>

... but someone could do so , if it wasn't by the current limitations
of doctest frontend .
;)

> The fact that things that are not stable parts of the API can be tested is
> independent of the framework you use to do the testing. If I, as an ignorant
> and foolish developer, wrote a unit test like this:
>
> class MyDumbTest(unittest.TestCase):
>      def testSpamRepr(self):
>          x = Spam(arg)
>          self.assertEquals(repr(x), "<Spam object at 0x123ab>")
>
>
> we shouldn't conclude that "unit tests are bad", but that MyDumbTest is bad
> and needs to be fixed.

+1

[...]
> And that's great, it really is, I'm not being sarcastic. But unit testing is
> not in competition to doc testing, they are complimentary, not alternatives.
> If you're not using both, then you're probably missing out on something.
>

+1

PS: ... and well , this would be my last message about dutest and how
it improves upon what's offered by doctest module ...

Summarizing : «Bad doctests» is not a cool statement

.. [1] dutest @ PyPI
        (https://pypi.python.org/pypi/dutest)

-- 
Regards,

Olemis.

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article: