[Python-Dev] pyparallel and new memory API discussions...

[Victor Stinner]

"""
    So, Victor, I'm interested to hear how the new API you're proposing
    will affect this solution I've come up with for pyparallel; I'm
    going to be absolutely dependent upon the ability to lock main
    thread pages as read-only in one fell-swoop -- am I still going to
    be able to do that with your new API in place?
"""

2013/6/19 Trent Nelson <trent at snakebite.org>:
> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
>> >      1. All memory allocated in a parallel context is localized to a
>> >         private heap.
>>
>> How do you allocate memory in this "private" heap? Did you add new
>> functions to allocate memory?
>
>     Yup:
>         _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
>
>     All memory operations (PyObject_New/Malloc etc) get intercepted
>     during parallel thread execution and redirected to _PyHeap_Malloc(),
>     which is a very simple slab allocator.  (No need for convoluted
>     buckets because we never free individual objects during parallel
>     execution; instead, we just blow everything away at the end.)

Ok, so I don't think that the PEP 445 would change anything for you.

The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.

If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-------------------------- 8< -----------------
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
{
    PyMemBlockAllocator *ctx;
    if (Py_PXCTX)
        return _PxMem_Malloc(size))
    else
        return alloc->malloc(alloc->ctx, size);
}

...

PyMemBlockAllocator pyparallel_pyobject;

static void *
setup_pyparallel_allocator(void)
{
    PyMemBlockAllocator alloc;
    PyObject_GetAllocator(&pyparallel_pyobject);
    alloc.ctx = &pyparallel_pyobject;
    alloc.malloc = _PxMem_AllocMalloc;
    ...
    PyObject_SetAllocator(&alloc);
}
-------------------------- 8< -----------------

But I don't know if you want pyparallel to be an "optional" feature
chosen at runtime...

Victor