tstate invalid crash with threads

[Stephen Robert Norris]

In article <000101beb237$88b71d80$999e2299 at tim>,
	"Tim Peters" <tim_one at email.msn.com> intoned:
> Followups to the Thread-SIG, please.
> 
> [Ray Loyzaga]
>> I have been playing with a multithreaded tcp based server for a while,
>> and during some serious stress testing continually hit a problem
>> involving the interpreter crashing with "Fatal Python error:
>> PyThreadState_Delete: invalid tstate".
>> ...
>> It appears to be a subtle race in PyThreadState_Delete ....
>> interestingly, if I uncomment the small sleep in "handle" in the server,
>> ie. make the server slower, it seems to work for ever ... 4m transactions
>> before I gave up. I think the problem only comes if you are creating and
>> destroying threads quickly in parallel.
> 
> PyThreadState_Delete is called from very few places, and one of them strikes
> me as suspicious:  at the end of threadmodule.c's t_bootstrap, we have:
> 
> 	PyThreadState_Clear(tstate);
> 	PyEval_ReleaseThread(tstate);
> 	PyThreadState_Delete(tstate);
> 	PyThread_exit_thread();
> 
> The suspicious thing here is that PyEval_ReleaseThread releases the global
> interpreter lock, so nothing is serializing calls to PyThreadState_Delete
> made from the following line.  PyThreadState_Delete in turn does no locking
> of its own either, but mutates a shared list.
> 
> If this isn't plain wrong, it's certainly not plain right <wink>.  Matches
> your symptoms, too (very rare blowups during high rates of thread death).
> 
> Guido?  I haven't been able to provoke Ray's problem under Win95, but the
> above just doesn't smell right.
> 
> win95-didn't-crash-but-the-TAB-and-ESC-keys-did-swap-their-
>     meanings!-ly y'rs  - tim

I've put a lock in pystate.c around anything that touches tstate,
and it seems to have fixed the problem (although reproducing it is
time-consuming).

This seems to be the same solution that Tim suggests, but at one remove.

	Stephen
-- 
Stephen Norris  srn at fn.com.au
PGP key available via finger srn at flibble.fn.com.au.
Farrow Norris Pty. Ltd.     http://www.fn.com.au/