[Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

[Victor Stinner]

I commited the new API (little bit different than my last patch on issue #3329):
http://hg.python.org/cpython/rev/6661a8154eb3

The documentation will be available in a few minutes at:
http://docs.python.org/3/c-api/memory.html

2013/6/14 Kristján Valur Jónsson <kristjan at ccpgames.com>:
>> Removing the GIL restriction would help to replace direct calls to
>> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
>> would be able to replace memory allocators, and these allocators would be
>> used "everywhere".
>> => see http://bugs.python.org/issue18203
>
> To keep this interesting, I have a somewhat different opinion to Victor :)
>   have put comments in the original defect, but would like to repeat them here.
> IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
> 1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details.  Requiring it to be GIL free severely limits what it can do.  For example, it would be forbidden to delegate to PyObject_MALLOC when debugging.

For my own pytracemalloc tool, holding the GIL while calling
PyMem_Malloc() is required to be able to retrieve the Python filename
and line number of the caller. So you convinced me :-) I am also
worried by the backward compatibility, even if I expect that only a
very few developers replaced Python memory allocators. A custom memory
allocator may not be thread-safe, so the GIL can also be convinient.

I added new functions in the "final" API: PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree(). These functions are just wrapper
for malloc(), realloc() and free(). The GIL does not need to be hold.
No process is done before/after at all. Behaviour of
PyMem_RawMalloc(0) is platform depend for example. "size >
PY_SSIZE_T_MAX" check is not done neither, but it may be interesting
to add this check for security reasons (it is already in place for
PyMem_Malloc and PyObject_Malloc).

Using these new functions instead of malloc/realloc/free is
interesting because the internal functions can be replaced with
PyMem_SetRawAllocators() and many checks are added in debug mode (ex:
check for buffer under- and overflow).

PyObject_Malloc() was not documented, so I did not document
PyObject_SetAllocators().

In the final API, I added a new PyMemAllocators structure to simplify
the API. I also made _PyObject_SetArenaAllocators() private because I
don't like its API (it is not homogenous with PyMem_SetAllocators) and
it is concerned by less use cases. I prefer to wait a little before
making this API public.

I didn't use "#ifndef Py_LIMITED_API", so all new functions are part
of the stable API. Is it correct?

Victor