[Tutor] problems with doctest: apparent interferance between tests (LONG)

[Brian van den Broek]

John Ridley said unto the world upon 2005-04-10 19:19:
>  --- Brian van den Broek <bvande at po-box.mcgill.ca> wrote: 
> 
>>But: it still leaves me wondering why removing either a) the one-line
>>no-doctest-containing docstring of the Wall_clock class or b) the
>>unreferenced Wall_clock.stop_interval method made my original test
>>code pass without needing to effect the changes you suggest. That
>>seems really odd to me.
> 
> 
> Hello again, Brian
> 
> Thought I'd better follow up on my previous post even though Kent has
> already done a much better job than me of explaining things.
> 
> I believe your two remaining puzzles can be solved by trying this
> little experiment. Using your original <culled code> example, add the
> extra line that I suggested and then run the script with the -v
> commandline option (i.e. python test.py -v).
> 
> What you should see is that the interval doctest is run first. But if
> you comment out the 'no-doctest-containing docstring' and re-run the
> script, the check_point doctest is run first! Needless to say, this
> rather awkward side-effect is going to make it hard to ensure objects
> are created and deleted at the right times ;-)
> 
> Now, I am not familiar enough with the doctest module to explain
> exactly why this change in the 'running-order' should occur. But maybe
> once you've begun to 'grok' doctest a little more you'll be able to
> shed some light on it for us!
> 
> Regards
> 
> John Ridley
> 
> Send instant messages to your online friends http://uk.messenger.yahoo.com 


Hi John and all,

Well, I think that has pointed the way to a solution! :-)

I had two methods in my culled code which included tests. I'd been 
puzzled as to why the presence or absence of apparently unrelated 
things made a difference to the passing or failing of my tests. You've 
just pointed out that only is the pass-state of the tests sensitive to 
the presence or absence of those elements but so is the order in which 
the two methods in my culled tests are run. (Indeed, that explains why 
the presence or absence matters, as the tests fail when ran in one 
ordering, while passing if ran in the other. Embarrassingly, I'd given 
myself some intentionally failing tests to confirm the order in which 
things were tested. So, the data was there, if only I'd had eyes to 
see :-[ .)

This suggests to me quite strongly that order in which the tests are 
run is determined at one or more points by dictionary access. (I 
suspect that one or both of the module objects to be examined and the 
docstrings in module objects are at some point stored in a 
dictionary.[*]) Since dictionary access is random, any fetching of a 
list of items to feed to the DocTest class could have the order of 
testing of two items altered by the presence of other items.

I've rarely read other people's code longer than 500 or so lines, so 
it is going to take me a while to understand doctest.py well enough to 
be certain of this hypothesis. But given the centrality of the 
dictionary to Python and how nicely the hypothesis matches the 
observed behaviour, it seems pretty well, though not conclusively, 
supported. I do indeed hope I can understand doctest.py well enough in 
the near future to confirm. But, for the moment, this has calmed the 
voices in my head. :-)

[*] "one or both" because one of objects whose presence mattered was a 
method without docstrings, the other a class docstring. So, I don't 
think it can simply be that there is a single dictionary involved. It 
would seem to be something more like one for objects to examine, 
another for docstrings of those objects. But now the armchair 
philosophy is well in front of the supporting evidence :-)

If (no, *when*!) I work it out, I will be certain to report back briefly.

Thanks again to all for the very useful help and observations!

Best,

Brian vdB