[Python-Dev] pyparallel and new memory API discussions...

Victor Stinner victor.stinner at gmail.com
Wed Jun 19 18:20:15 CEST 2013

    So, Victor, I'm interested to hear how the new API you're proposing
    will affect this solution I've come up with for pyparallel; I'm
    going to be absolutely dependent upon the ability to lock main
    thread pages as read-only in one fell-swoop -- am I still going to
    be able to do that with your new API in place?

2013/6/19 Trent Nelson <trent at snakebite.org>:
> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
>> >      1. All memory allocated in a parallel context is localized to a
>> >         private heap.
>> How do you allocate memory in this "private" heap? Did you add new
>> functions to allocate memory?
>     Yup:
>         _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
>     All memory operations (PyObject_New/Malloc etc) get intercepted
>     during parallel thread execution and redirected to _PyHeap_Malloc(),
>     which is a very simple slab allocator.  (No need for convoluted
>     buckets because we never free individual objects during parallel
>     execution; instead, we just blow everything away at the end.)

Ok, so I don't think that the PEP 445 would change anything for you.

The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.

If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-------------------------- 8< -----------------
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
    PyMemBlockAllocator *ctx;
    if (Py_PXCTX)
        return _PxMem_Malloc(size))
        return alloc->malloc(alloc->ctx, size);


PyMemBlockAllocator pyparallel_pyobject;

static void *
    PyMemBlockAllocator alloc;
    alloc.ctx = &pyparallel_pyobject;
    alloc.malloc = _PxMem_AllocMalloc;
-------------------------- 8< -----------------

But I don't know if you want pyparallel to be an "optional" feature
chosen at runtime...


More information about the Python-Dev mailing list