[Python-Dev] pyparallel and new memory API discussions...

[Trent Nelson]

On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
> """
>     So, Victor, I'm interested to hear how the new API you're proposing
>     will affect this solution I've come up with for pyparallel; I'm
>     going to be absolutely dependent upon the ability to lock main
>     thread pages as read-only in one fell-swoop -- am I still going to
>     be able to do that with your new API in place?
> """
> 
> 2013/6/19 Trent Nelson <trent at snakebite.org>:
> > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> >> >      1. All memory allocated in a parallel context is localized to a
> >> >         private heap.
> >>
> >> How do you allocate memory in this "private" heap? Did you add new
> >> functions to allocate memory?
> >
> >     Yup:
> >         _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
> >
> >     All memory operations (PyObject_New/Malloc etc) get intercepted
> >     during parallel thread execution and redirected to _PyHeap_Malloc(),
> >     which is a very simple slab allocator.  (No need for convoluted
> >     buckets because we never free individual objects during parallel
> >     execution; instead, we just blow everything away at the end.)
> 
> Ok, so I don't think that the PEP 445 would change anything for you.
> 
> The following change might have an impact: If _PyHeap_Malloc is not
> thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
> GIL is not held would avoid bugs in your code.

    Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by
    design, but I'm not sure if we're referring to the same sort of
    thread-safe problem here.

    For one, _PyHeap_Malloc won't ever run if the GIL isn't being held.

    (Parallel threads are only allowed to run when the main thread has
     the GIL held and has relinquished control to parallel threads.)

    Also, I interpret PyMem_RawMalloc() as a direct shortcut to
    malloc() (or something else that returns void *s that are then
    free()'d down the track).  Is that right?

    I don't think that would impact pyparallel.

> If you want to choose dynamically the allocator at runtime, you can
> replace PyObject_Malloc allocator using:
> -------------------------- 8< -----------------
> static void *
> _PxMem_AllocMalloc(void *ctx, size_t size)
> {
>     PyMemBlockAllocator *ctx;
>     if (Py_PXCTX)
>         return _PxMem_Malloc(size))
>     else
>         return alloc->malloc(alloc->ctx, size);
> }
> 
> ...
> 
> PyMemBlockAllocator pyparallel_pyobject;
> 
> static void *
> setup_pyparallel_allocator(void)
> {
>     PyMemBlockAllocator alloc;
>     PyObject_GetAllocator(&pyparallel_pyobject);
>     alloc.ctx = &pyparallel_pyobject;
>     alloc.malloc = _PxMem_AllocMalloc;
>     ...
>     PyObject_SetAllocator(&alloc);
> }
> -------------------------- 8< -----------------
> 
> But I don't know if you want pyparallel to be an "optional" feature
> chosen at runtime...

    Hmmm, those code snippets are interesting.  Time for some more
    homework.

        Trent.