[Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

[Kristján Valur Jónsson]

> -----Original Message-----
> I would like to remove the "GIL must be held" restriction from
> PyMem_Malloc(). In my opinion, the restriction was motived by a bug in
> Python, bug fixed by the issue #3329. Let me explain why.
> 
...

> 
> Removing the GIL restriction would help to replace direct calls to
> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
> would be able to replace memory allocators, and these allocators would be
> used "everywhere".
> => see http://bugs.python.org/issue18203
> 

To keep this interesting, I have a somewhat different opinion to Victor :)
  have put comments in the original defect, but would like to repeat them here.
IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details.  Requiring it to be GIL free severely limits what it can do.  For example, it would be forbidden to delegate to PyObject_MALLOC when debugging. 

The approach CCP has taken (we have replaced all raw malloc calls with api calls) is this:
a) Add a "raw" api, PyMem_MALLOC_RAW.  This is guaranteed to be thread safe and call directly to the external memory api of python, as set by Py_SetAllocator()
b) Replace calls to malloc() in the source code with PyMem_MALLOC/PyMem_MALLOC_RAW as appropriate (in our case, using  an include file with #defines to mimimize changes)

There are only two or three places in the source code that require non-GIL protected malloc.  IMHO, requiring PyMem_MALLOC to be threadsafe just to cater to those three places is an overkill, and does more harm than good by limiting our options.

Cheers!
Kristján