[ python-Bugs-1529297 ] unrelated variable messing up doctests

SourceForge.net noreply at sourceforge.net
Fri Jul 28 05:09:05 CEST 2006


Bugs item #1529297, was opened at 2006-07-26 15:02
Message generated for change (Comment added) made by macquigg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1529297&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Submitted By: David MacQuigg (macquigg)
Assigned to: Nobody/Anonymous (nobody)
Summary: unrelated variable messing up doctests

Initial Comment:
Version 2.5b2 with IDLE 1.2b2 running on Windows XP 
Home, fully updated.

Unpack the attached zip file.
Open program.py, test_suite.py, in separate IDLE 
windows.  
Run test_suite.py.  
  8 passed and 0 failed.
Now add a variable at the top of program.py, any name 
will do.
  xyz = 2
Re-run test_suite.py.  
  7 passed and 1 failed !!

I haven't been able to isolate the problem, but it is 
repeatable.  Taking out blocks of unrelated statements 
changes the behavior.  Feels like a memory allocation 
error, or something strange.


----------------------------------------------------------------------

>Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 20:09

Message:
Logged In: YES 
user_id=676422

I've got a workaround.  I changed the names of my test 
functions to _1connect, _2hello, _3envfrom, _4envrcpt, 
_5addheader, and _6eom.
Could we set the order explicitly, maybe
doctest.order = [connect, hello, envfrom, envrcpt, ... ]


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-07-27 18:31

Message:
Logged In: YES 
user_id=31435

> would it be possible to run the tests in the order
> they are written, rather than in alphabetical order?

Sorry, that's impossible.  doctest never sees the source
code, it crawls through module and class __dict__s to find
objects with docstrings, and the order things appear in
those is undefined.

----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 18:03

Message:
Logged In: YES 
user_id=676422

I hope this is not too much to ask, but would it be 
possible to run the tests in the order they are written, 
rather than in alphabetical order?  This isn't a big deal, 
as long as the order is stable, but it would abide by the 
rule of least astonishment. :>)

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-07-27 17:53

Message:
Logged In: YES 
user_id=31435

Glad it solved your mystery!  Note that 2.4.4 and 2.5 /will/
sort the tests by name again (the code for that has already
been checked in) -- it was very intentional that it did that
before 2.4, and it was a bug that it stopped doing it in 2.4
through 2.4.3.  OTOH, you're the only one so far who noticed :-)

----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 17:46

Message:
Logged In: YES 
user_id=676422

That fixed it!!  Nice catch.

I've been aware for some time of the sensitivity to order 
in the doctests, but it has never been a problem until now, 
and I've been writing programs with doctests since 2.3.  
When I add a test, and it breaks a later test, it has 
always been easy to nail down the piece that changed.

This "behind the scenes" change of test order seems like 
the kind of problem that rarely occurs, but can cause 
serious problems when it does.

Thanks again for an excellent followup.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-07-27 17:01

Message:
Logged In: YES 
user_id=31435

DocTestFinder has been changed to sort test names again, in
revs 50882 (on the trunk, for 2.5) and 50886 (on the 2.4
branch, for 2.4.4).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-07-27 16:21

Message:
Logged In: YES 
user_id=31435

I think I know what's happening here.  doctest was largely
rewritten for Python 2.4, and it lost a subtle intended
feature:  it used to (<= 2.3) sort tests by name before
running them, but in 2.4 started running them in whatever
order they happen to appear in the hash-code-based module
__dict__.

Changing the namespace in program.py in seemingly random
ways matters because test_suite.py starts with

from program import *

and so everything in program.py's module namespace
"pollutes" test_suite.py's namespace too.  That can change
the order tests run in.  And as I noted before, whether or
not the doctest for _envrcpt() passes or fails depends on
whether or not the doctests for _hello() run before them,
due to side effects on the shared `ci` instance.  Look at
your output carefully, and I expect you'll find that the
order of tests /does/ change depending on crap variable
names you stick in (or remove from) program.py.

If you're with me so far, try changing the end of
doctest.py's DocTestFinder.find() from

    return tests

to

    tests.sort()
    return tests


----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 14:48

Message:
Logged In: YES 
user_id=676422

Some observations:
If I give the test variable a private name, the problem 
goes away.
.   _xyz = 2
If I assign the variable to an object already in memory, 
the problem doesn't go away.
.   xyz = LocalADDRs
It doesn't matter that the test variable is never used. The 
problem persists, even if I add a second statement:
.   xyz = 2
.   if xyz == 2: pass
It doesn't matter where in the module the statement is 
placed, as long as it is module level, not inside a class 
or function.


----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 14:48

Message:
Logged In: YES 
user_id=676422

Some observations:
If I give the test variable a private name, the problem 
goes away.
.   _xyz = 2
If I assign the variable to an object already in memory, 
the problem doesn't go away.
.   xyz = LocalADDRs
It doesn't matter that the test variable is never used. The 
problem persists, even if I add a second statement:
.   xyz = 2
.   if xyz == 2: pass
It doesn't matter where in the module the statement is 
placed, as long as it is module level, not inside a class 
or function.


----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 10:32

Message:
Logged In: YES 
user_id=676422

The failure is not the specific output of doctest, but the 
fact that it *changes* when I add an unrelated variable 
xyz=2 in the module program.py.  I am not able to reproduce 
this failure with Tim's stripped down test_suite.py.

I agree the problem is not in IDLE.  I can do the same from 
a bash command line with
$ python2.4 test_suite.py
but not with 
$ python2.3 test_suite.py


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-07-27 09:14

Message:
Logged In: YES 
user_id=31435

I see exactly the same failure akuchling reported, on
Windows using current trunk, running from a command shell
(not IDLE -- didn't try IDLE).  The attached test_suite.py
throws away everything not needed to see the same failure;
in particular, it gets rid of doctest, the logging module,
and most of the functions in test_suite.py.

Reduced to this, I don't see a reason to suspect a Python
bug.  ci.IDrating is initialized to 100, and the

_envrcpt('zz1 at open-mail.org')

line produces (the expected)

CONTINUE [['zz1 at open-mail.org', 50, 2, 75, 101, '']]

if you /leave ci.IDrating alone/.  However, if you do

ci.hello('zz_IP_192.168.0.8_mailout7.simple.tld')

first, that has a (apparently deliberate) side effect of
changing ci.IDrating to None, and then this path is /not/ taken:

# . . Whitelist pass
    if self.IDrating > whitelvl:
        rcptinfo[4] = 101  # ham > 100% to ignore later
spamscore       ###2e
        self.R.append(rcptinfo)

The 101 vanishes from the expected output then, and I get

CONTINUE [['zz1 at open-mail.org', 50, 2, 75, 50, '']]

instead.

That exactly reproduces the only failure Andrew & I have
seen, and appears to be what the code is telling it to do
(via side effects on the shared `ci` instance).

----------------------------------------------------------------------

Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 08:22

Message:
Logged In: YES 
user_id=676422

On my Redhat Linux machine, I see the failure in 2.4.1, but 
not in 2.3.4.

Trying to isolate the problem, I've commented out some 
blocks of unrelated code, but I can't get very far with 
this, because many of the deletions make the problem go 
away.  It's as if the problem depends on the exact position 
in memory of the test variable.

I did manage to delete the DNS queries, so now you can run 
the tests without intermittent failures due to DNS timeouts.
I'll keep the latest at http://www.open-
mail.org/test/test_0607nn.tgz

Thanks for your efforts.


----------------------------------------------------------------------

Comment By: A.M. Kuchling (akuchling)
Date: 2006-07-27 05:15

Message:
Logged In: YES 
user_id=11375

On a Linux machine, both 2.4.3 and SVN HEAD report 7 tests
passed, one failed.

For both versions, the failure is: 
Failed example:
    _envrcpt('zz1 at open-mail.org')
Expected:
    CONTINUE [['zz1 at open-mail.org', 50, 2, 75, 101, '']]
Got:
    CONTINUE [['zz1 at open-mail.org', 50, 2, 75, 50, '']]


----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-26 20:59

Message:
Logged In: YES 
user_id=33168

Can you try with SVN head?  There was an unitialized read
that the doctests could trigger.  It has been fixed. 
Perhaps you are running into that?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1529297&group_id=5470


More information about the Python-bugs-list mailing list