[Python-Dev] Extension modules, Threading, and the GIL

[Mark Hammond]

[Tim]
> I'd like to intensify the problem, though:

Good!  I was just taking small steps to get to the same endpoint.

> you're in a thread and you want
> to call a Python API function safely.  Period.  You don't
> know anything else.  You don't even know whether Python has
> been initialized yet, let alone whether there's already a
> thread state, and/or an interpreter state, sitting around
> available for you to use.

Agreed 100%.  In some ways, I believe this is just the conclusion from my 2
points taken together if we can ignore the current world order.  I split
them to try and differentiate the requirements from the current API, but if
we progress to Tim's description, my points become:

1) Becomes exactly as Tim stated.

2.1) Stays the same - release the GIL.
2.2) Goes away - if (1) requires no knowledge of Python's state, there is no
need for extensions to take special action just to enable this.

> I'd also like to postulate that proposed solutions can rely
> on a new Python
> C API supplying a portable spelling of thread-local storage.  We can
> implement that easily on pthreads and Windows boxes, it seems
> to me to cut
> to the heart of several problems, and I'm willing to say that Python
> threading doesn't work anymore on other boxes until platform wizards
> volunteer code to implement this API there too.

This sounds good to me.  After you have done the Win98 version, I volunteer
to port it to Win2k <wink>.

I believe we have a reasonable starting point.  Our PEP could have:
* All the usual PEP fluff <wink>
* Define the goal, basically as stated by Tim.
* Define a new C API specifically for this purpose, probably as an
  "optional extension" to the existing thread state APIs.
* Define a TLS interface that all ports must implement *iff* this new
  API is to be available.

This sounds reasonable to me unless we can see a number of other uses for
TLS - in which case the TLS interface would probably get its own PEP, with
this PEP relying on it.  However, I don't see too much need for TLS - once
we have our hands on a Python thread-state, we have a thread-specific
dictionary available today, and a TLS dictionary from inside your Python
code is trivial.  From what I can see, we just need platform TLS to get hold
of our thread-state, from which point we can (and do) manage our own thread
specific data.

How does this sound?

Mark.