[py-dev] Integrating doctests

[Ian Bicking]

OK, so I really want to integrate doctests, along the lines I mention in
this post:

http://blog.ianbicking.org/building-testability-into-web-applications.html

So, here's what I have so far.  There's a couple parts.  Here's a
function to test unit tests:

def unittest_tester(test):
     result = test.defaultTestResult()
     test.run(result)
     for flavor, lst in [('FAIL', result.failures),
                         ('ERROR', result.errors)]:
         for test, err in lst:
             print '%s: %s' % (flavor, test)
             print err
     if result.failures or result.errors:
         py.test.fail('Errors occurred')


This is a little lame, but I haven't gotten far enough into py.test to
figure out reporting.  Actually, I still feel totally mystified by
py.test, I just wander around poking things until they stop breaking.
Is there anyplace I should be starting at to figure out quite what is
going on?  It's kind of that Ravioli Code kind of feel; a description of
the role of all the classes would be really useful.  Though personally I
like descriptions that follow the code path chronologically, it's a good
compliment to the source which tends not to have many chronological hints.

Anyway, I needed that function because I'm using doctest's
TestSuite-building capability, though it'd probably be just as easy to
use doctest.DocTest directly.  Anyway, I really think unittest testing
is a useful feature that belong in there somewhere, in addition to
doctests, since I don't always have the ability to change other
projects' test structure.  I don't think the support has to be very
sophisticated; fix the errors in that example and it might be good
enough.  If it doesn't integrate particularly well into other UIs, like
Tkinter, I don't think that's a big deal.

After that I have this stuff that traverses all modules and gets
doctests from some of the modules:

class Directory(py.test.collect.Directory):
     def filefilter(self, path):
         return (path.check(fnmatch="*.py")
                 and path.purebasename != 'conftest')

class Module(py.test.collect.Module):

     def buildname2items(self):
         if (self.name.startswith('test_')
             or self.name.endswith('_test')):
             d = super(Module, self).buildname2items()
         else:
             d = {}
         try:
             test_suite = doctest.DocTestSuite(
                 self.obj, setUp=setup_doctest,
                 optionflags=flags)
         except ValueError, e:
             if 'has no test' in str(e):
                 return d
             else:
                 raise
         for i, x in py.builtin.enumerate(unpack_testsuite(test_suite)):
             name = 'doctest[%d]' % i
             d[name] = self.Function(name, self, (x,), obj=unittest_tester)
         return d


But this is kind of awkward -- I don't want to look for normal tests in
non-test_*-named files, only doctests, so I have to undo the filefilter
in Directory and then reapply it in Module.  And I get lots of modules
that have no tests in my reports, which wastes space.

So here's what I'd like to do, at which point I'd be fully happy:

* Get rid of spurious tracebacks; the unittest/doctest reporting should
override any py.test reporting entirely.  And of course reports
shouldn't be printed to stdout.  I've been thinking about making things
like doctest; simple input/output tests, except ones that don't occur in
a Python interepreter context, and this would be nice there as well.  I
figured subclassing py.test.item.Item would help somehow, but I don't
know how.  But I'd be quite happy if there was some
py.test.fail_report() function that printed out the message and no
traceback.  That seems easy to use and explain.

* A way to indicate that a module is boring and shouldn't be included,
after the module is loaded and is being collected.  Or maybe empty
modules should just be hidden always.  I don't think I have verbosity
turned up, but maybe I accidentally do and it wouldn't show it normally.

* And of course for unittesting and doctesting to go in py.test
directly, so I don't have complex conftest.py files.  But if that
happens it's really important to me personally that I have some control
over how doctests are made -- adding a setup function is important, and
overriding the option flags (I always want ELLIPSES on, for instance).
I'm trying to avoid boilerplate in the files that are being doctested,
so I'd like to apply these on a package level (i.e., in conftest).  This
seems easy enough to facilitate with a subclass of a to-be-written
DocTestCollector.

While I remember:

* I'd like a way to halt all tests.  With SQLObject your environment
must be configured properly, since you have to access a database.  It's
overwhelming when this isn't the case, because you get a huge list of
failing tests.  I'd like to raise an error when that happens so that
py.test will abort all future tests.  Hmm... I haven't tried SystemExit
or KeyboardInterrupt.  Maybe that'd do on its own.


-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org