[Python-Dev] Python Interpreter Thread Safety?

["Martin v. Löwis"]

Evan Jones wrote:
> Due to the issue of thread safety in the Python memory allocator, I have 
> been wondering about thread safety in the rest of the Python 
> interpreter. I understand that the interpreter is not thread safe, but 
> I'm not sure that I have seen a discussion of the all areas where this 
> is an issue. Here are the areas I know of:

This very much depends on the definition of "thread safe". If this mean
"working correctly in the presence of threads", then your statement
is wrong - the interpreter *is* thread-safe. The global interpreter lock
(GIL) guarantees that only one thread at any time can execute critical
operations. Since all threads acquire the GIL before performing such
operations, the interpreter is thread-safe.

> 1. The memory allocator.
> 2. Reference counts.
> 3. The cyclic garbage collector.
> 4. Current interpreter state is pointed to by a single shared pointer.

This is all protected by the GIL.

> 5. Many modules may not be thread safe (?).

Modules often release the GIL through BEGIN_ALLOW_THREADS, if they know
that would be safe if another thread would enter the Python interpreter.

> Ignoring the issue of #5 for the moment, are there any other areas where 
> this is a problem? I'm curious about how much work it would be to allow 
> concurrent execution of Python code.

Define "concurrent". Webster's offers

1. operating or occurring at the same time

Clearly, on a single-processor system, no two activities can execute
concurrently - the processor can do at most one activity at any point
in time.

Perhaps you are asking whether it would be possible to change the
current coarse-grained lock into a more finer-grained lock (as working
without locks is not implementable). This is also known as "free
threading". There have been attempts to implement free threading, and
they have failed.

> Note: One of the reasons I am asking is that my memory allocator patch 
> is that it changes the current allocator from "sort of" thread safe to 
> obviously unsafe.

The allocator is thread-safe in the presence of the GIL - you are
supposed to hold the GIL before entering the allocator. Due to some
unfortunate historical reasons, there is code which enters free()
without holding the GIL - and that is what the allocator specifically
deals with. Except for this single case, all callers of the allocator
are required to hold the GIL.

> However, if it 
> was one of the components that permitted the interpreter to go 
> multi-threaded, then it would be worth it.

Again, the interpreter supports multi-threading today. Removing
the GIL is more difficult, though - nearly any container object
(list, dictionary, etc) would have to change, plus the reference
counting (which would have to grow atomic increment/decrement).

Regards,
Martin