[Tutor] problems with doctest: apparent interferance between tests (LONG)

[Brian van den Broek]

Hi all,

I must apologize for the length of the post. I have explained my 
problem as tersely as able, and have done my level-best to produce the 
minimum code snippet illustrative of my difficulties. But, as (I hope) 
will become clear, any substantial attempts to further cull it made 
the problem go away. (For what it is worth, this is the product of 
several hours of investigating and attempting to cull.)


The quick story:
I have apparent interference between doctests embedded in the 
docstrings of different methods, and this interference also appears to 
be influenced by seemingly irrelevant things such as whether the 
module has a (non-doctest-containing) docstring or not. I'm flummoxed 
as to why there is this apparent interference. The long code snippet 
below is the minimal snippet I was able to make which still exhibited 
the problems; it is commented with indications of the dependencies of 
the tests.



Background:
(Perhaps the the reader could skip this `Background' and jump down to 
`The Problem', returning here if need be.) I have cut enough of the 
original code that the point of what remains may be obscure. (The 
original code includes complete documentation, more methods, etc.) A 
bit of explanation is perhaps needed to give the context of my problem.

I've been writing a module to allow convenient wall clock timings and 
generate a report of same on demand. The idea is to import it, make an 
instance of the Wall_clock class, and then call Wall_clock methods at 
places of interest in the code, so as to add _check_point's and start 
and stop _intervals. (Never mind if this isn't too useful or 
duplicates functionality available elsewhere--it is largely a learning 
exercise to explore various Python features, many of which are not 
reflected in the snippet below.)

All the timings are relative to the creation of the Wall_clock 
instance. To prevent confusion in the reports, my original design idea 
was to make the Wall_clock class `Singleton-like' in that there could 
only be one instance alive at a time, and any attempt to create 
another concurrent instance would raise an exception. (I've since 
decided to opt for using the Borg pattern 
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66531>, with 
a class attribute storing the time of first instantiation, but my 
issues depend on the first approach I explored.)

I made my `Singleton-like' class by having a class attribute 
is_instanced, set to True by the class __init__ method, and False by 
its __del__. The __init__ method raises an error if the attribute is 
True, thus preventing concurrent instances. (See the final code 
snippet for a working model with tests that pass as expected.)

So far, so good.



The Problem:
doctest is one of the aspects of Python that I've been trying to learn 
with this project. I have 2 problems using doctest on my module:

1) There is a test neat the start of Wall_clock.check_point's 
docstring that attempts to instantiate the Wall_clock class. It fails, 
in the manner an attempt to create a *second* concurrent instance of 
Wall_clock should fail. But, since the relevant test is not preceded 
in the docstring by any tests which create a Wall_clock instance, I 
don't understand how this could be happening. I had theorized that 
perhaps there was interference in that previously run tests might 
leave Wall_clock.is_instanced == True. The first test in 
Wall_clock.check_point appears to bare this out. Yet this doesn't make 
sense either. And it won't explain:

2) In my attempts to produce the minimal working sample that 
illustrated my problem, I found that removing any of:

     a) the Wall_clock class docstring (which contains no tests),

     b) the entire Wall_clock.stop_interval method (this method
        is not called anywhere in the code in the snippet, nor
        does it contain any tests), or

     c) a portion of the Wall_clock.interval method's doctests
        (this portion does not do anything to affect affect the
        Wall_clock.is_instanced class attribute)

all caused the failing test to pass. (These are all identified by 
comments in the code.) I cannot understand why these things, not in 
Wall_clock.check_point's doctest, should affect whether its tests pass 
or fail. I am utterly perplexed. I *am* sure there is a bug, but also 
think it much more likely to be mine than Tim Peters' :-)

In addition to the culled-code where I originally observed this 
problem, I have produced a similar bit of skeleton code which works as 
expected. I don't understand why they operate differently. First, my 
culled code, then the skeleton that works.



The Culled Code:
When I run this, the Wall_clock.interval method is tested first, then 
the Wall_clock.check_point method. (Assessed by the commented out 
tests that were intentional failures.) It is the 
Wall_clock.check_point docstring that is the locus of the problem. 
(The heavy edit on the code has robbed the tests and much of the code 
of their sense and relevance; please ignore that.)

<culled code>

import time

class Wall_clockInstanceError(Exception):
     def __init__(self):
         Exception.__init__(self)

class Interval_name_conflictError(Exception):
     def __init__(self):
         Exception.__init__(self)

class Wall_clock(object):
     '''A class for making wall clock timings.'''
     # If I remove the docstring above, the test passes. WHY?

     is_instanced = False

     def __init__(self):

         if self.__class__.is_instanced:
             raise Wall_clockInstanceError
         else:
             self.__class__.is_instanced = True
         self.intervals = []
         self.data = []
         self.check_point_names = set()
         self.interval_names = set()
         self.start_time = time.time()
         _Check_point.offset = _Interval.offset = self.start_time
         _Check_point.count = _Interval.count = 0

     def __del__(self):
         self.__class__.is_instanced = False

     def check_point(self, check_point_name = None):
         '''Creates a new _Check_point instance; appends it to .data.

         >>> # print "Testing check_point (intentional fail)."
         >>> print Wall_clock.is_instanced
         True
         >>> # Why should it be True at this point?
         >>> # Note, too, that any of the commented changes
         >>> # make the most recent test fail!
         >>> wclock = Wall_clock()      # Failing test WHY?
         >>> del(wclock)
         >>> new_wclock = Wall_clock()  # This passes
         '''
         # The test commented WHY? in this docstring fails unless
         # I make any of the changes elsewhere commented. I don't
         # understand why it fails in the present code, nor why the
         # elsewhere commented changes make it pass.
         # Since the marked test (and the subsequent del() test
         # fail, I don't understand why the final test passes.
         pass

     def interval(self, interval_name = None):
         '''
         >>> # print "Testing interval (intentional fail)."
         >>> wc = Wall_clock()
         >>> del(wc)
         >>> wclock = Wall_clock()
         >>> # If I omit the rest of the tests here, no failure
         >>> # of the test of interest in Wall_clock.check_point
         >>> an_interval = wclock.interval('F')
         >>> same_name = wclock.interval('F')
         Traceback (most recent call last):
             ...
         Interval_name_conflictError
         '''
         # The last few lines are key! (See comments in tests.) WHY?

         if interval_name:
             if type(interval_name) == int:
                 interval_name = str(interval_name)

             if not interval_name.startswith(_Interval.name_form %''):
                 interval_name = _Interval.name_form %interval_name

             if interval_name in self.interval_names:
                 raise Interval_name_conflictError

         new_interval = _Interval(interval_name)
         self.intervals.append(new_interval)
         self.data.append(new_interval)
         self.interval_names.add(str(new_interval.name))
         return new_interval

     # None of the code in the snippet references the method below.
     # But, if I remove the method, the test of interest in
     # Wall_clock.check_point passes. WTF!?!?!
     def stop_interval(self, interval=None):

         try:
             if None == interval:
                 interval = self.intervals.pop()
                 interval.stop()

             else:
                 self.intervals.remove(interval)
                 interval.stop()

         except IndexError, ValueError:
             raise "A custom error class my orig. code defined"

class _Check_point(object):

     count = None    # Wall_clock.__init__ will set these 2.
     offset = None   # Assignments here to make the existence of
                     # the attributes obvious.
     name_form = 'Check point %s'

     def __init__(self, name = None):

         _Check_point.count += 1
         if not name:
             name = _Check_point.name_form % _Check_point.count
         self.name = name
         self.data = time.time() - _Check_point.offset


class _Interval(object):

     count = None    # Wall_clock.__init__ will set these 2.
     offset = None   # Assignments here to make the existence of
                     # the attributes obvious.
     name_form = 'Interval %s'

     def __init__(self, name = None):

         _Interval.count += 1
         self.running = True
         if not name:
             name = _Interval.name_form % _Interval.count
         self.name = name

def _test():
     import doctest, sys
     doctest.testmod(sys.modules[__name__], report = True,
                            optionflags = doctest.ELLIPSIS)

if __name__ == '__main__':
     _test()
     print "\nIf you got this far in my post, my sincerest thanks!"


</culled code>



The Skeleton Code:

All tests pass in this, and it is, as far as I can tell, employing the 
same broad structure as my problematic code above, save that there are 
no `interfering' elements akin the the class docstring, etc.

<skeleton code>

'''
 >>> an_instance = Singleton_like()
 >>> another_instance = Singleton_like()
Traceback (most recent call last):
     ...
NameError: global name 'Concurrent_instancesError' is not defined
 >>> del(an_instance)
 >>> another_instance = Singleton_like()
'''

class Singleton_like(object):

     is_instanced = False

     def __init__(self):
         if self.__class__.is_instanced:
             raise Concurrent_instancesError
         self.__class__.is_instanced = True

     def __del__(self):
         self.__class__.is_instanced = False

     def a_method(self):
         '''>>> an_instance = Singleton_like()'''
         pass

     def another_method(self):
         '''>>> another_instance = Singleton_like()'''
         pass

if __name__ == '__main__':
     import doctest, sys
     doctest.testmod(sys.modules[__name__])

</skeleton code>

I, and what hair I've not yet torn out, would be most grateful for any 
suggestions.

Best to