[Python-Dev] Python Interpreter Thread Safety?

[Tim Peters]

...

[Evan Jones]
> What I was trying to ask with my last email was what are the trouble
> areas? There are probably many that I am unaware of, due to my
> unfamiliarity the Python internals.

Google on "Python free threading".  That's not meant to be curt, it's
just meant to recognize that the task is daunting and has been
discussed often before.

[Martin v. Löwis]
>> Due to some unfortunate historical reasons, there is code which enters
>> free() without holding the GIL - and that is what the allocator specifically
>> deals with.

> Right, but as said in a previous post, I'm not convinced that the
> current implementation is completely correct anyway.

Sorry, I haven't had time for this.  From your earlier post:

> For example, is it possible to call PyMem_Free from two threads
> simultaneously?

Possible but not legal; undefined behavior if you try.  See the
"Thread State and the Global Interpreter Lock" section of the Python C
API manual.

    ... only the thread that has acquired the global interpreter lock may
        operate on Python objects or call Python/C API functions

There are only a handful of exceptions to the last part of that rule,
concerned with interpreter and thread startup and shutdown, and
they're explicitly listed in that section.  The memory-management
functions aren't among them.

In addition, it's not legal to call PyMem_Free regardless unless the
pointer passed to it was originally obtained from another function in
the PyMem_* family (that specifically excludes memory obtained from a
PyObject_* function).  In a release build, all of the PyMem_*
allocators resolve directly to the platform malloc or realloc, and all
PyMem_Free has to determine is that they *were* so allocated and thus
call the platform free() directly (which is presumably safe to call
without holding the GIL).  The hacks in PyObject_Free (== PyMem_Free)
are there solely so that question can be answered correctly in the
absence of holding the GIL.   "That question" == "does pymalloc
control the pointer passed to me, or does the system malloc?".

In return, that hack is there solely because in much earlier versions
of Python extension writers got into the horrible habit of allocating
object memory with PyObject_New but releasing it with PyMem_Free, and
because indeed Python didn't *have* a PyObject_Free function then. 
Other extension writers were just nuts, mixing PyMem_* calls with
direct calls to system free/malloc/realloc, and ignoring GIL issues
for all of those.  When pymalloc was new, we went to insane lengths to
avoid breaking that stuff, but enough is enough.

> Since the problem is that threads could call PyMem_Free without
> holding the GIL, it seems to be that it is possible.

Yes, but not specific to PyMem_Free.  It's clearly _possible_ to call
_any_ function from multiple threads without holding the GIL.

> Shouldn't it also be supported?

No. If what they want is the system malloc/realloc/free, that's what
they should call.

> In the current memory allocator, I believe that situation can lead to
> inconsistent state.

Certainly, but only if pymalloc controls the memory blocks.  If they
were actually obtained from the system malloc, the only part of
pymalloc that has to work correctly is the Py_ADDRESS_IN_RANGE()
macro.  When that returns false, the only other thing PyObject_Free()
does is call the system free() immediately, then return.  None of
pymalloc's data structures are involved, apart from the hacks ensuring
that the arena of base addresses is safe to access despite potentlly
current mutation-by-appending.

> ...
> Basically, if a concurrent memory allocator is the requirement,

It isn't.  The attempt to _exploit_ the GIL by doing no internal
locking of its own is 100% deliberate in pymalloc -- it's a
significant speed win (albeit on some platforms more than others).

> then I think some other approach is necessary.

If it became necessary, that's what this section of obmalloc is for:

SIMPLELOCK_DECL(_malloc_lock)
#define LOCK()		SIMPLELOCK_LOCK(_malloc_lock)
#define UNLOCK()	SIMPLELOCK_UNLOCK(_malloc_lock)
#define LOCK_INIT()	SIMPLELOCK_INIT(_malloc_lock)
#define LOCK_FINI()	SIMPLELOCK_FINI(_malloc_lock)

You'll see that PyObject_Free() calls LOCK() and UNLOCK() at
appropriate places already, but they have empty expansions now.

Back to the present:

[Martin]
>> Again, the interpreter supports multi-threading today. Removing
>> the GIL is more difficult, though - nearly any container object
>> (list, dictionary, etc) would have to change, plus the reference
>> counting (which would have to grow atomic increment/decrement).

[Evan]
> Wouldn't it be up to the programmer to ensure that accesses to shared
> objects, like containers, are serialized? For example, with Java's
> collections, there are both synchronized and unsynchronized versions.

Enormous mounds of existing threaded Python code freely manipulates
lists and dicts without explicit locking now.  We can't break that --
and wouldn't want to.  Writing threaded code is especially easy (a
relative stmt, not absolute) in Python because of it.