[Python-Dev] pystate.c changes for Python 2.4.2

[Tim Peters]

[Tim]
>> ...
>> As a result, it so happens that core Python never uses the original
>> PyThreadState_Delete() anymore, except when Py_NewInterpreter() has
>> to throw away the brand new thread state it created because it turns out
>> it can't create a new interpreter.

[Michael]
> Um, PyThreadState_Delete() is called from zapthreads() is called from
> PyInterpreterState_Delete() is called from Py_Finalize()... so I don't
> entirely believe you here :)

That's only because I was wrong ;-)  Thanks!  That use was in fact
half the cause of Python's segfaulting shutdown races, so it was
exceptionally stupid of me to miss that one.

>> Since core Python never calls Py_NewInterpreter() or
>> PyThreadState_Delete(), you're in an intersection of areas no current
>> Python developer even thinks about, let alone tests.  But by the same
>> token, _because_ it's unused, there's nothing you can do to
>> PyThreadState_Delete() that can hurt core Python either.  That's why
>> people should be more encouraging here ;-)  It certainly makes sense
>> to me that if a thread state is going away, any record of it in the
>> auto-GIL-state machinery must also go away.

> I guess if the patch fixes the original problem, it should go in -- my
> uneasiness stems not from worrying that it might do harm, but rather
> from the fact that I think it's incomplete.  Maybe I just haven't
> thought it through enough.

Oh, I don't know whether it's "complete".  I don't really understand
multiple interpreters, have no idea why the OP is calling
PyThreadState_Delete(), and fully expect that nobody is going to add
tests to Python to check any of this.  But Gabriel showed real skill
at tracking down problems in this area, so if he creates new problems
for himself by doing this (who else plays in this intersection? 
nobody I know of), I'm sure he'll figure them out ;-)

For example, it seems possible that he'll hit the same kinds of
segfaulting shutdown races the Python core used to suffer.  If a
thread T releases the GIL, and goes on to delete its own thread state
via PyThreadState_Delete(T's_thread_state), that _is_ a race with
Python shutdown's zapthreads trying to delete the same thing.  That's
why PyThreadState_DeleteCurrent(void) was introduced, and any call to
PyThreadState_Delete() is suspect on this count.

> Also, every time I look at pystate.c, I think of things that could be
> done differently or better -- for example, I think it would be sane
> and possibly even sensible to only call PyThread_set_key_value() in
> _PyGILState_NoteThreadState() if "tstate->interp ==
> autoInterpreterState".

I joined Mark Hammond in ignoring the possibility of multiple
interpreters when this stuff went in.  The best thing to do with
untested gimmicks without an audience willing to work on them is to
remove them.  If you want to "rehabilitate" multiple interpreters
here, the best first step would be to write an addendum to PEP 311
documenting an intended design.  The GIL state API is documented well
enough on its own, but the _intended_ semantics when mixing threads
with multiple interpreters is clear as mud.  PEP 311 asked "but who
cares?", and nobody answered.