Hi,
I'm working with Numpy in the context of supporting different memory types such as persistent memory and CXL attached. I would like to propose a minor change, but figured I would get some initial feedback from the developer community before submitting a PR.
In multiarray/alloc.c the allocator (beneath the cache) using the POSIX malloc/calloc/realloc/free. I propose that these should be changed to PyMem_RawXXX equivalents. The reason for this is that by doing so, one can use the python custom allocator functions (e.g. PyMem_GetAllocator/PyMem_SetAllocator) to intercept the memory allocator for NumPy arrays. This will be useful as heterogeneous memories need supporting.
There are likely other places in NumPy that could do with a rinse and repeat - may be someone could advise?
Thanks,
Daniel
---
Example patch for 1.19.x (I'm building with Python3.6)
diff --git a/numpy/core/src/multiarray/alloc.c b/numpy/core/src/multiarray/alloc.c
index 795fc7315..e9e888478 100644
--- a/numpy/core/src/multiarray/alloc.c
+++ b/numpy/core/src/multiarray/alloc.c
@@ -248,7 +248,7 @@ PyDataMem_NEW(size_t size)
void *result;
assert(size != 0);
- result = malloc(size);
+ result = PyMem_RawMalloc(size);
if (_PyDataMem_eventhook != NULL) {
NPY_ALLOW_C_API_DEF
NPY_ALLOW_C_API
@@ -270,7 +270,7 @@ PyDataMem_NEW_ZEROED(size_t size, size_t elsize)
{
void *result;
- result = calloc(size, elsize);
+ result = PyMem_RawCalloc(size, elsize);
if (_PyDataMem_eventhook != NULL) {
NPY_ALLOW_C_API_DEF
NPY_ALLOW_C_API
@@ -291,7 +291,7 @@ NPY_NO_EXPORT void
PyDataMem_FREE(void *ptr)
{
PyTraceMalloc_Untrack(NPY_TRACE_DOMAIN, (npy_uintp)ptr);
- free(ptr);
+ PyMem_RawFree(ptr);
Daniel G. Waddington
Principal Research Staff Member,
Phone: +1 408 927 2359