[Python-Dev] Making python C-API thread safe (try 2)

[Harri Pesonen]

Shane Hathaway wrote:

> [Moved to python-list at python.org, where this thread belongs]
>
> Harri Pesonen wrote:
>
>> But my basic message is this: Python needs to be made thread safe. 
>> Making the individual interpreters thread safe is trivial, and 
>> benefits many people, and is a necessary first step; making threads 
>> within interpreter thread safe is possible as well, at least if you 
>> leave something for the developer, as you should, as you do in every 
>> other programming language as well.
>
>
> Lately, I've been considering an alternative to this line of thinking. 
> I've been wondering whether threads are truly the right direction to 
> pursue.  This would be heresy in the Java world, but maybe Pythonistas 
> are more open to this thought.
>
> The concept of a thread is composed of two concepts: multiple 
> processes and shared memory.  Supporting multiple simultaneous 
> processes is relatively simple and has proven value.  Shared memory, 
> on the other hand, results in a great number of complications.  Some 
> of the complications have remained difficult problems for a long time: 
> preventing deadlocks, knowing exactly what needs to be locked, finding 
> race conditions, etc.  I don't believe we should force the burden of 
> thread safety on every software engineer.  Engineers have better 
> things to do.
>
> At the same time, shared memory is quite valuable when you're ready to 
> take on the burden of thread safety.  Therefore, I'm looking for a 
> good way to split a process into multiple processes and share only 
> certain parts of a program with other processes.  I'd like some form 
> of *explicit* sharing with a Pythonic API.
>
> Imagine the following Python module:
>
>
> import pseudothreads
>
> data = pseudothreads.shared([])
>
> def data_collection_thread():
>     s = get_some_data()
>     data.append(s)
>
> for n in range(4):
>     pseudothreads.start_new_thread(data_collection_thread)
>
>
> In this made-up example, nothing is shared between threads except for 
> the "data" global.  The shared() function copies the list to shared 
> memory and returns a wrapper around the list that prevents access by 
> multiple threads simultaneously.  start_new_thread() is a thin wrapper 
> around os.fork().  Each pseudothread has its own global interpreter lock.
>
> I wonder whether others would consider such a thing valuable, or even 
> feasible. :-)

Sounds great! :-) Basically, you are suggesting that each thread that is 
created in Python, has its own separate interpreter, and if the 
programmer ever wants to communicate between different threads, he has 
to do it explicitly using the tools that are available. Sounds good to me.

So my original suggestion of removing all global data and having the 
interpreter state always on stack, expanded with this idea, it would 
make Python completely thread safe. No need for global interpreter lock 
anymore. Of course this would break the existing Python thread code, but 
something has to break anyhow. :-)

Harri