[Python-Dev] Python Interpreter Thread Safety?

[Evan Jones]

On Jan 28, 2005, at 19:44, Martin v. Löwis wrote:
> Python threads can run truly parallel, as long as one of them
> invoke BEGIN_ALLOW_THREADS.

Except that they are really executing C code, not Python code.

> I think nobody really remembers - ask Google for "Python free 
> threading". Greg Stein did the patch, and the main problem apparently
> was that the performance became unacceptable - apparently primarily
> because of dictionary locking.

Thanks, I found the threads discussing it.

>> Right, but as said in a previous post, I'm not convinced that the 
>> current implementation is completely correct anyway.
> Why do you think so? (I see in your previous post that you claim
> it is not completely correct, but I don't see any proof).

There are a number of issues actually, but as Tim points, only if the 
blocks are managed by PyMalloc. I had written a description of three of 
them here, but they are not relevant. If the issue is calling 
PyMem_Free with a pointer that was allocated with malloc() while 
PyMalloc is doing other stuff, then no problem: That is possible to 
support, but I'll have to think rather hard about some of the issues.

> For example, for lists, the C API allows direct access to the pointers
> in the list. If the elements of the list could change in-between, an
> object in the list might go away after you got the pointer, but before
> you had a chance to INCREF it. This would cause a crash shortly
> afterwards. Even if that was changed to always return a new refence,
> lots of code would break, as it would create large memory leaks
> (code would have needed to decref the list items, but currently
> doesn't - nor is it currently necessary).

Ah! Right. In Java, the collections are all actually written in Java, 
and run on the VM. Thus, when some concurrent weirdness happens, it 
just corrupts the application, not the VM. However, in Python, this 
could actually corrupt the interpreter itself, crashing the entire 
thing with a very ungraceful Segmentation Fault or something similar.

Evan Jones