[Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

[Victor Stinner]

Hi,

I would like to remove the "GIL must be held" restriction from
PyMem_Malloc(). In my opinion, the restriction was motived by a bug in
Python, bug fixed by the issue #3329. Let me explain why.


The PyMem_Malloc() function is a thin wrapper to malloc(). It returns
NULL if the size is lager than PY_SSIZE_T_MAX and have a well defined
behaviour for PyMem_Malloc(0) (don't return NULL). So it is surprising
to read in Include/pymem.h:
"The GIL must be held when using these APIs."

The reason is more surprising: in debug mode, PyMem_Malloc() is no
more a thin wrapper to malloc(), but it calls internally
PyObject_Malloc(), the "Python allocator" (called pymalloc). (Many
other checks are done in debug mode, but it's unrelated to my point.)
The problem is that PyObject_Malloc() is not thread-safe, the GIL must
be held.

Short history:

  fb45791150d1 (Mar 23 2002) "gives Python a debug-mode pymalloc"
  f294fdd18b5b (Mar 28 2002) removes the "check API family"
  e16dbf875303 (Apr 22 2002) redirects indirectly PyMem_Malloc() to
PyObject_Malloc() in debug mode
  b6aff7a59803 (Sep 28 2009) reintroduces API checks

So the GIL issue is almost as old as the debug mode for Python memory
allocators.

My patch attached to http://bugs.python.org/issue3329 changes the
design of the debug memory allocators: they are now wrapper (hooks) on
the underlying memory allocator (PyMem: malloc, PyObject: pymalloc),
instead of always redirecting to pymalloc (ex: PyObject_Malloc).

Using my patch, PyMem_Malloc() now always calls malloc(), even in
debug mode. Removing the "GIL must be held" restriction is now safe.
Do you agree?

May it cause backward compatibility issue? PyMem_Malloc() and
PyMem_MALLOC() call malloc(), except if the Python source code was
manually modified. Does this use case concern many developers?


Removing the GIL restriction would help to replace direct calls to
malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an
application would be able to replace memory allocators, and these
allocators would be used "everywhere".
=> see http://bugs.python.org/issue18203

Victor