PEP: Procedure for Adding New Modules (please comment)

[Tim Peters]

[Paul Winkler]
> ...
> I also should not have implied that the existing library is not
> adequately tested; I have no basis for thinking that.

But I do:  it's not.  Huge steaming mounds of it are barely touched.  If you
take a look at test_sundry.py, you'll find about 100(!) library modules  are
"tested" there.  The test_sundry docstring reads:

"""Do a minimal test of all the modules that aren't otherwise tested."""

The "minimal test" test_sundry does is merely to import the modules, just to
verify that importing alone isn't enough to make them blow up.

This is an area where community contributions are almost non-existent.  If
anyone is looking for a low-commitment way to help Python out, pick your
favorite library module and contribute some good tests for it!  There's a
decent orientation Lib/test/REAMDE to help you get started.  The regrtest.py
framework is baroque and inadequately documented, but it can run both
unittest- and doctest-based tests in natural ways now.

> I based my suggestion (doctest at minimum, preferably unittest)
> on the note at the end of the documentation for doctest:
>
> """
> The first word in doctest is "doc", and that's why the author
> wrote doctest: to keep documentation up to date. It so happens that
> doctest makes a pleasant unit testing environment, but that's not
> its primary purpose.

I'm the author in question, and I stand by that.  OTOH, doctest is the
fastest and easiest way in existence to get a substantial test suite up and
running, especially when used as Fredrik suggested.

There's an interesting but unplanned experiment going on in Python 2.2:
Guido started the new test_iter.py (to test 2.2 iterators), using unittest,
and I later more than doubled its size by adding additional iterator tests.
unittest is easy enough to use for this, but it's always deliberate "extra"
work to write the tests.  Like:

    # Test list()'s use of iterators.
    def test_builtin_list(self):
        self.assertEqual(list(SequenceClass(5)), range(5))
        self.assertEqual(list(SequenceClass(0)), [])
        self.assertEqual(list(()), [])
        self.assertEqual(list(range(10, -1, -1)), range(10, -1, -1))

        d = {"one": 1, "two": 2, "three": 3}
        self.assertEqual(list(d), d.keys())

        self.assertRaises(TypeError, list, list)
        self.assertRaises(TypeError, list, 42)

        f = open(TESTFN, "w")
        try:
            for i in range(5):
                f.write("%d\n" % i)
        finally:
            f.close()
        f = open(TESTFN, "r")
        try:
            self.assertEqual(list(f), ["0\n", "1\n", "2\n", "3\n", "4\n"])
            f.seek(0, 0)
            self.assertEqual(list(f.xreadlines()),
                             ["0\n", "1\n", "2\n", "3\n", "4\n"])
        finally:
            f.close()
            try:
                unlink(TESTFN)
            except OSError:
                pass

OTOH, Neil Schemenauer started test_generators.py (to test 2.2 generators)
using doctest, and I followed his lead on that one too, until it's now
nearly 1,400 lines.  Here's what the tests look like in that:

"""
>From the Iterators list, about the types of these things.

>>> def g():
...     yield 1
...
>>> type(g)
<type 'function'>
>>> i = g()
>>> type(i)
<type 'generator'>
>>> dir(i)
['gi_frame', 'gi_running', 'next']
>>> print i.next.__doc__
next() -- get the next value, or raise StopIteration
>>> iter(i) is i
1
>>> import types
>>> isinstance(i, types.GeneratorType)
1

And more, added later.

>>> i.gi_running
0
>>> type(i.gi_frame)
<type 'frame'>
>>> i.gi_running = 42
Traceback (most recent call last):
  ...
TypeError: object has read-only attributes
>>> def g():
...     yield me.gi_running
>>> me = g()
>>> me.gi_running
0
>>> me.next()
1
>>> me.gi_running
0

A clever union-find implementation from c.l.py, due to David Eppstein.
Sent: Friday, June 29, 2001 12:16 PM
To: python-list at python.org
Subject: Re: PEP 255: Simple Generators

>>> class disjointSet:
...     def __init__(self, name):
...         self.name = name
...         self.parent = None
...         self.generator = self.generate()

[etc]
"""

IOW, every example posted to every mailing list, along with every example in
PEP 255, was simply *pasted* into a test string.  It was no additional work
to speak of; WYSIWYG; nothing was written *just* "to test"; and if you can
read straight Python, you can easily and directly see what each little piece
is testing.  WRT the latter, what do you think the earlier

        self.assertRaises(TypeError, list, list)

was testing?  I wrote it, but I couldn't remember after I copied the code
into this msg; I had to look up the source for unittest's assertRaises().
As a doctest, it would have read

"""
>>> list(list)
Traceback (most recent call last):
  ...
TypeError: iter() of non-sequence
"""

instead and then "it's obvious".

I still think doctest's *primary* purpose is for ensuring docstring examples
stay 100% accurate, but my extended recent experiences with test_iter.py vs
test_generators.py is making me wonder whether doctest should shift
emphasis.  But I'm more wary of Second System Syndrome than Guido, so I'm
holding back <wink>.

> So for more thorough testing of things that don't belong in the
> documentation, I thought unittest.py might be a worthy candidate.

Indeed it is.

> After all, that's what it's designed for, and it's already in the
> standard library.
>
> I should have said "Yes, testing is a good idea, let's have some
> standards for that", and then I should have shut my mouth.

What -- and deprive me of an opportunity to rant <wink>?

All new Python tests should be in unittest or doctest format; and we'd
gladly adopt conversions of existing tests to those formats too (all the
PythonLab'ers have converted some of them, but there's too much else that
needs doing too to expect us to finish all of them without help).

well-actually-you're-free-to-expect-anything-ly y'rs  - tim