[py-dev] Integrating doctests

[holger krekel]

Hi Ian, 

i have been away for a few days (and everyone seems to know this
and mails me like crazy).   I have the impression you might have
want to split your mail into multiple ones, but maybe i am just 
missing the red line somehow. 

On Tue, May 10, 2005 at 18:04 -0500, Ian Bicking wrote:
> OK, so I really want to integrate doctests, along the lines I mention in
> this post:
> 
> http://blog.ianbicking.org/building-testability-into-web-applications.html

Can't reach that server at the moment (not the first time, btw). 

> So, here's what I have so far.  There's a couple parts.  Here's a
> function to test unit tests:

You started off with doctests, and now talk about unittests. Probably
your weblog entry would help understand me your line of thinking better ... 

> def unittest_tester(test):
>     result = test.defaultTestResult()
>     test.run(result)
>     for flavor, lst in [('FAIL', result.failures),
>                         ('ERROR', result.errors)]:
>         for test, err in lst:
>             print '%s: %s' % (flavor, test)
>             print err
>     if result.failures or result.errors:
>         py.test.fail('Errors occurred')
> 
> This is a little lame, but I haven't gotten far enough into py.test to
> figure out reporting.

There is definitely a missing py.test customization hook which
allows a user to customize reporting from e.g. a conftest.py
file or from simply invoking py.test.fail()

> Actually, I still feel totally mystified by py.test, I just
> wander around poking things until they stop breaking.  Is
> there anyplace I should be starting at to figure out quite
> what is going on?

I at least wrote some documentation about the internal workings: 

http://codespeak.net/py/current/doc/test.html#collecting-and-running-tests-implementation-remarks

(sorry for the long URL).  Have you read that and deemed it unworthy
or unhelpful? 

> It's kind of that Ravioli Code kind of feel; a description
> of the role of all the classes would be really useful.
> Though personally I like descriptions that follow the code
> path chronologically, it's a good compliment to the source
> which tends not to have many chronological hints.

I had the impression that i keep py-dev updated about refactorings
and the intentions and even provide some detail of what's going 
on.  But that can certainly be improved. 

> Anyway, I needed that function because I'm using doctest's
> TestSuite-building capability, though it'd probably be just as easy to
> use doctest.DocTest directly.  Anyway, I really think unittest testing
> is a useful feature that belong in there somewhere, 

You are mixing unittests and doctests freely here.  I admit i
still haven't looked to closely into doctest's internal workings 
so do you mean to imply that doctest and unittests are very
uniformly handled with Python already? 

> ...  in addition to doctests, since I don't always have the
> ability to change other projects' test structure.

so here you mean that py.test should grok existing
(unit/doc-)test structures?  I am not against the idea but it
is a long way to follow that road i am afraid and i am not
ready to follow that myself at the moment. (apart from the
fact that we have certain ways of running unittests/doctests
in PyPy with py.test mechanisms but it's code that i'd rather
not like to explain at the moment, because it jumps through
more hoops than neccessary: it additionally runs against 
the PyPy interpreter instead of against CPython). 

> I don't think the support has to be very sophisticated; fix
> the errors in that example and it might be good enough.  If
> it doesn't integrate particularly well into other UIs, like
> Tkinter, I don't think that's a big deal.

I'd like Jan Balster's tkinter frontend to remain as much
usable as possible and although i break Jan's code from
time to time (sorry, Jan!) i'd like it very much to be
an alternative frontend. 

> After that I have this stuff that traverses all modules and gets
> doctests from some of the modules:
> 
> class Directory(py.test.collect.Directory):
>     def filefilter(self, path):
>         return (path.check(fnmatch="*.py")
>                 and path.purebasename != 'conftest')
> 
> class Module(py.test.collect.Module):
> 
>     def buildname2items(self):
>         if (self.name.startswith('test_')
>             or self.name.endswith('_test')):
>             d = super(Module, self).buildname2items()
>         else:
>             d = {}
>         try:
>             test_suite = doctest.DocTestSuite(
>                 self.obj, setUp=setup_doctest,
>                 optionflags=flags)
>         except ValueError, e:
>             if 'has no test' in str(e):
>                 return d
>             else:
>                 raise
>         for i, x in py.builtin.enumerate(unpack_testsuite(test_suite)):
>             name = 'doctest[%d]' % i
>             d[name] = self.Function(name, self, (x,), obj=unittest_tester)
>         return d
> 
> 
> But this is kind of awkward -- I don't want to look for normal tests in
> non-test_*-named files, only doctests, so I have to undo the filefilter
> in Directory and then reapply it in Module.  And I get lots of modules
> that have no tests in my reports, which wastes space.

Right.  A better way would be to override the Directory collector 
and produce two kinds of Modules, usual py.test ones and ones 
with doctests with code like this (untested): 

    class Directory(py.test.collect.Directory): 
        def buildname2items(self): 
            # just let our base class do its building
            d = super(Directory, self).buildname2items()

            # let's look for doctests ... 
            for fn in self.fspath.listdir('*.py'): 
                if fn.basename in d or fn.basename == 'conftest.py': 
                    continue
                # we have a candidate for doctests 
                if not isdoctestmodule(fn): # if possible to implement that
                    continue
                d[fn.basename] = DoctestModule(fn, parent=self) 

    class DoctestModule(py.test.collect.Module): 
        def run(self): 
            module = self.fspath.pyimport()
            # setup module appropriately 
            try: 
                # run doctests from 'module'
            finally: 
                # teardown module appropriately 

Like said earlier, this DoctestModule should have a way to "take over" 
reporting somehow. It doesn't at the moment and be warned that there 
might be other slight issues.   But basically the above should  
work rather cleanly. 
            
> * I'd like a way to halt all tests.  With SQLObject your environment
> must be configured properly, since you have to access a database.  It's
> overwhelming when this isn't the case, because you get a huge list of
> failing tests.  I'd like to raise an error when that happens so that
> py.test will abort all future tests.  Hmm... I haven't tried SystemExit
> or KeyboardInterrupt.  Maybe that'd do on its own.

Have you tried py.test.exit(msg)? 

cheers, 

    holger


P.S.: Please note that py.test is still under development and there are 
      sometimes reasons why things are not done yet: often i prefer to 
      refactor and consolidate the (not so small number of) features 
      before i head off adding yet more features.