[ python-Bugs-1061968 ] threads: segfault or Py_FatalError at exit

[SourceForge.net]

Bugs item #1061968, was opened at 2004-11-07 11:30
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1061968&group_id=5470

Category: Threads
Group: Python 2.3
Status: Open
Resolution: Accepted
Priority: 9
Submitted By: Armin Rigo (arigo)
Assigned to: Mark Hammond (mhammond)
Summary: threads: segfault or Py_FatalError at exit

Initial Comment:
I couldn't reproduce this bug on any machine but mine, but after investigation it looks like a real locking bug in pystate.c which triggers under a certain timing.  Attached is a small example which triggers the problem repeatedly on my machine, together with the output I get with a Python interpreter where I inserted debug prints in pystate.c:zapthreads() and pystate.c:PyThreadState_Delete().  My problem is that zapthreads() gets called before the last C thread finishes to call PyGILState_Release().  The very final call that PyGILState_Release() makes to PyThreadState_Delete() -- *after* it releases all locks -- then occurs after zapthreads() has cleared everything.  Crash.

Can't fix it myself because I don't know anything about PyGILState_*() and the intent of the various locks...

Boosted to priority 9 because it looks like it should be fixed before 2.4b3.

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2004-12-02 11:04

Message:
Logged In: YES 
user_id=31435

I may be wrong, but I'm pretty sure Armin already backported 
all the thread fixes that went into 2.4 (including this one).

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2004-12-02 02:34

Message:
Logged In: YES 
user_id=14198

OK, I'll take it on for 2.3, but it might be worth reminding
me next you get the smell of a new 2.3 release.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-12-02 01:30

Message:
Logged In: YES 
user_id=357491

Any chance this can get backported to 2.3?

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2004-11-08 09:12

Message:
Logged In: YES 
user_id=4771

It looks fine and I can't reproduce the problem any more.  Thanks for the fix!

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-11-07 23:32

Message:
Logged In: YES 
user_id=31435

I may not be able to do anything here on Monday, so I 
checked in the patch, but will leave the report open until 
Armin can add more info:

Misc/NEWS 1.1186
Python/pystate.c 2.38

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-11-07 20:12

Message:
Logged In: YES 
user_id=31435

Thanks, Mark!  It's great to get another set of "been there, 
done that" eyeballs on thread nasties.  The symptoms look 
the same as the ones Guido reported 4 years ago, and there 
seems to be a clear reason for why they reappeared, so I'm 
optimistic that fixing it the same way again <wink> will work.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2004-11-07 17:51

Message:
Logged In: YES 
user_id=14198

Damn - Tim beat me - I missed his comment and came up with
an almost identical patch.  I could not provoke the error on
my RH8 box, so can't offer any more assurance this fixes the
underlying bug

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-11-07 17:26

Message:
Logged In: YES 
user_id=31435

Back to Armin -- would you please try the attached patch 
(gilrelease.txt)?  I've never been able to provoke this bug, so 
I can only verify that other tests don't break.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-11-07 14:48

Message:
Logged In: YES 
user_id=31435

Nasty, nasty, nasty.  This is a reappearance of this miserable 
old bug:

http://www.python.org/sf/225673

BTW, the comments there explain why I'm unlikely to see it 
on Windows.

That bug got fixed in rev 2.41 of threadmodule.c, by 
introducing the new PyThreadState_DeleteCurrent() function, 
which deletes the current tstate *while* holding the GIL.  
Guido should have inserted a large block of screaming 
comments at the same time, explaining how horridly subtle 
the need for that was.

3 years and 7 months later, Mark reintroduced the problem, in 
rev 2.59 of threadmodule.c.  That replaced the 
PyThreadState_DeleteCurrent() call by a call to  
PyGILState_Release(), and the latter-- like Python before rev 
2.41 --does *not* hold the GIL when it decides to delete the 
tstate.  That's the source of the timing hole you're falling into 
here.

Offhand I'm not sure what to do about it, because I'm not 
familiar with the other bug Mark was trying to fix with rev 
2.59.  I'm assigning to Mark under the hope that it will be 
instantly obvious to him.  Mark, if it's not, please assign it to 
me, and I'll try to work it all out.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1061968&group_id=5470