[Python-Dev] Making python C-API thread safe (try 2)

[Harri Pesonen]

Phillip J. Eby wrote:

> At 08:47 PM 9/11/03 +0300, Harri Pesonen wrote:
>
>> But my basic message is this: Python needs to be made thread safe. 
>> Making the individual interpreters thread safe is trivial, and 
>> benefits many people, and is a necessary first step;
>
>
> It's far from trivial - you're talking about invalidating every piece 
> of C code written for Python over a multi-year people by dozens upon 
> dozens of extension authors.

The change is trivial in Python C API. I already said that it would 
break everything outside the Python distribution, but the change in 
other applications is also trivial.

> It doesn't benefit many people: only those using isolated interpreters 
> embedded in a multithreaded C program. 

I don't know how many people are writing threads in Python, either. I 
guess that not so many. In my case I only need a thread safe 
interpreter, I don't create threads in Python code. So just having what 
I described would be enough for me: no need for global interpreter lock, 
and Python would be really multithreading. It would benefit many people, 
I'm sure.

>
>> making threads within interpreter thread safe is possible as well, at 
>> least if you leave something for the developer, as you should, as you 
>> do in every other programming language as well.
>
>
> You misunderstand.  Those "critical sections" are for the most part in 
> Python's C code, not in the Python script. 

Yes, I'm aware of the None problem at least (only one instance of it). 
Please enlighten me about the other critical sections? Object 
allocation/freeing?

> I'm guessing you haven't done much writing of C extensions for Python 
> (or Python core C), or else you'd realize why trying to make 
> INCREF/DECREF threadsafe would absolutely decimate performance.  
> Reference count updates happen *way* too often in normal code flow.
>
I also knew that already. But how else can you do it?

Of course, changing Python to not have a single None would help a lot. 
Or, perhaps it could have a single None, but in case of None, the 
reference count would have no meaning, it would never be deallocated, 
because it would be checked in code. Maybe it does it already, I don't know.

I'm also wondering why this problem has not been addressed before? If I 
had the power to change Python, this would be the first thing I did.

Harri