[Python-Dev] Making python C-API thread safe (try 2)

[Harri Pesonen]

Phillip J. Eby wrote:

> At 07:53 PM 9/11/03 +0300, Harri Pesonen wrote:
>
>> Sorry about the previous post! :-)
>>
>> Background: I'm new to Python. I just embedded Python into a 
>> multi-threaded application, and extended Python by callbacks as well. 
>> During this process I realised that Python is not internally as 
>> beautiful as externally. It is not thread safe. This surprised me, 
>> because Python is a quite young programming language. I started 
>> thinking how to fix this, how to make Python thread safe. Currently 
>> Python does not support multiprocessor machines or hyperthreading 
>> processors. It is only a matter of time when this has to be fixed, 
>> better soon than later.
>>
>> How to make C-API thread safe: It's quite simple, in fact. Thread 
>> should first call
>>
>> PyThreadState* Py_NewInterpreter()
>>
>> and use this thread state pointer in every following API call. This 
>> thread state pointer should point to a memory structure, that holds 
>> everything that Python interpreter needs. There should be no global 
>> variables, not even None, except
>
>
> This only makes individual Python interpreters thread-safe.  That 
> might certainly be useful for some embedded applications, but it does 
> nothing for thread-safety *within* an interpreter.
>
> In other words, Python would *still* need a global interpreter lock, 
> in order for threaded programs written in Python to behave properly.
>
> So, your solution would help only creators of multithreaded embedded 
> applications, wherein each thread desired an isolated Python 
> interpreter.  Meanwhile, it would impose both a one-time and ongoing 
> cost penalty on *all other* Python users, without giving them any 
> benefit in return.

You are right, I thought only about making interpreters thread safe, 
thanks for pointing this out. But this is still needed, it is 
unavoidable in the long run.

What do you mean about the penalty? If you mean having the state pointer 
in stack in every call, I don't think so.

If we think about making Python threads thread-safe, within the 
interpreter, then I think that it should be left for the application 
deverloper, in principle. He should create "critical sections" and other 
locks when needed (I don't know how it is done currently). I can imagine 
that None and other values will cause problems, when two threads 
increment or decrement the reference count at the same time. Of course 
there is a simple solution for that, InterlockedIncrement and 
InterlockedDecrement in Win32 at least.

I can imagine that there are other problems with garbage collector, and 
so on. I don't want to go into details about this, I got my message 
through, I was almost losing sleep when I realised the current situation.

But my basic message is this: Python needs to be made thread safe. 
Making the individual interpreters thread safe is trivial, and benefits 
many people, and is a necessary first step; making threads within 
interpreter thread safe is possible as well, at least if you leave 
something for the developer, as you should, as you do in every other 
programming language as well.

Harri