[ python-Bugs-993394 ] test_dummy_threading vs -O

[SourceForge.net]

Bugs item #993394, was opened at 2004-07-18 11:31
Message generated for change (Comment added) made by bcannon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=993394&group_id=5470

Category: Threads
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Tim Peters (tim_one)
Assigned to: Brett Cannon (bcannon)
Summary: test_dummy_threading vs -O

Initial Comment:
In Python 2.4, running test_dummy_threading with -O 
leads to an exception after the test is finished, in 
threading's exit func.  Didn't happen under 2.3.  Sorry, 
no idea why, and as soon as I saw all the fiddling with 
sys.modules I ran screaming in terror <wink>.  Here on 
WinXP:

C:\Python23>python lib/test/regrtest.py 
test_dummy_threading # OK
test_dummy_threading
1 test OK.

C:\Python23>python -O lib/test/regrtest.py 
test_dummy_threading # OK
test_dummy_threading
1 test OK.

C:\Python23>cd \python24

C:\Python24>python lib/test/regrtest.py 
test_dummy_threading # OK
test_dummy_threading
1 test OK.

C:\Python24>python -O lib/test/regrtest.py 
test_dummy_threading # not OK
test_dummy_threading
1 test OK.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Python24\lib\atexit.py", line 20, in 
_run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python24\lib\threading.py", line 607, in 
__exitfunc
    self._Thread__delete()
  File "C:\Python24\lib\threading.py", line 497, in __delete
    del _active[_get_ident()]
KeyError: -1

C:\Python24>

----------------------------------------------------------------------

>Comment By: Brett Cannon (bcannon)
Date: 2004-07-19 19:59

Message:
Logged In: YES 
user_id=357491

OK, problem goes away if you add a currentThread() call in __exitfunc() 
(line 607).  This gives the desired side-effect all while being safe from 
causing shutdown problems since that function is only called by atexit.  
And I am about to run the whole test suite to make sure it doesn't cause 
evil to appear somewhere else.  =)

Does that solution work for you, Tim?

Otherwise I am afraid I am going to have to come up with a custom dict 
for threading._active that dummy_threading injects into its copy that 
tries to handle this properly.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-07-19 19:13

Message:
Logged In: YES 
user_id=357491

Can't add those lines back in without breaking the fixes to allow 
tracebacks triggered during interpreter shutdown from failing.  And I 
don't think doing it at the beginning will solve it since it will still be 
overwritten by subsequent threads in threading._active anyway.  Could 
be fixed if the call is made in the method registered with atexit.

Plan on spending tonight on this one.  Hopefully I will come up with 
something.  =)

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-07-19 16:57

Message:
Logged In: YES 
user_id=31435

Then I think you'll find the answer if you do

cvs diff -r r234 -r release23-maint threading.py

I was using the released 2.3.4 Python.  But on the current 
2.3 maint branch, you removed two lines like this:

        currentThread() # for side-effect

These *were* unconditionally executed in 2.3.4, but don't 
exist on the current 2.3 branch, or on HEAD.

I expect that explains why this didn't fail in 2.3.4.  I don't 
know that reverting those lines is a correct (or even the 
best) solution, though.  Perhaps doing currentThread() once, 
from the main thread, at the start of threading's shutdown 
dance, would be enough.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-07-18 23:13

Message:
Logged In: YES 
user_id=357491

OK, on the way to figuring this out, I think.

Line 503 of Lib/threading.py has the statement ``assert self is not 
currentThread()``.  Now currentThread() creates an instance of 
_DummyThread for the current thread and inserts it into 
threading._active with the key of threading._get_ident(), which is -1 for 
all dummy_threading Threads.

Problem is that this doesn't happen under -O so threading._active doesn't 
get an entry for -1 like the exit func expects there to be and thus leads 
to an error.  You can uncomment line 645 to see the difference when the 
test is run.

As for it being a new error in 2.4, my CVS checkout for 2.3 has this error 
as well.  Even if I back Lib/threading back to version 1.40 (before I made 
changes for printing the traceback from shutdown errors) and 
dummy_threading before Jim's threading local stuff (which puts it to the 
initial checkin) I still get the error.  So from what I can tell this is old.

Now, it isn't from the tests, that much I am sure.  If you change the 
imported module for _threading to 'threading' the tests pass either way.  
For some reason the initial thread created from the _MainThread() 
creation in 'threading' disappears before the end of execution for the 
test.  Not sure why.  Might have something to do with 
dummy_thread.get_ident() always returning -1.  Going to keep looking 
into it.  If it is from dummy_thread.get_ident() returning -1 then it might 
be a slight pain to try to fix.  Might have to come up with some modified 
dict for _active in dummy_threading for the code to use that just 
consistently has a -1 or somehow communicates with 
dummy_thread.get_ident() to keep everything in sync.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-07-18 15:16

Message:
Logged In: YES 
user_id=357491

This is just turning into the month of my regression tests breaking, or at 
least my regression tests that decide to play dangeously with 
sys.modules.  =)

I will take a look and see if I can figure this one out this week.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=993394&group_id=5470