[Python-ideas] solving multi-core Python

Trent Nelson trent at snakebite.org
Tue Jun 23 20:29:40 CEST 2015

On Sun, Jun 21, 2015 at 12:40:43PM +0200, Stefan Behnel wrote:
> Nick Coghlan schrieb am 21.06.2015 um 12:25:
> > On 21 June 2015 at 19:48, Antoine Pitrou wrote:
> >> On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:
> >>>
> >>> For inter-interpreter communication, the worst case scenario is having
> >>> to rely on a memcpy based message passing system (which would still be
> >>> faster than multiprocessing's serialisation + IPC overhead)
> >>
> >> And memcpy() updates pointer references to dependent objects magically?
> >> Surely you meant the memdeepcopy() function that's part of every
> >> standard C library!
> > 
> > We already have the tools to do deep copies of object trees (although
> > I'll concede I *was* actually thinking in terms of the classic C/C++
> > mistake of carelessly copying pointers around when I wrote that
> > particular message). One of the options for deep copies tends to be a
> > pickle/unpickle round trip, which will still incur the serialisation
> > overhead, but not the IPC overhead.
> > 
> > "Faster message passing than multiprocessing" sets the baseline pretty
> > low, after all.
> > 
> > However, this is also why Eric mentions the notions of object
> > ownership or limiting channels to less than the full complement of
> > Python objects. As an *added* feature at the Python level, it's
> > possible to initially enforce restrictions that don't exist in the C
> > level subinterpeter API, and then work to relax those restrictions
> > over time.
> If objects can make it explicit that they support sharing (and preferably
> are allowed to implement the exact details themselves), I'm sure we'll find
> ways to share NumPy arrays across subinterpreters. That feature alone tends
> to be a quick way to make a lot of people happy.

    FWIW, the following commit was all it took to get NumPy playing
    nicely with PyParallel:


    It uses thread-local buckets instead of static ones, and calls out
    to PyMem_Raw(Malloc|Realloc|Calloc|Free) instead of the normal libc
    counterparts.  This means PyParallel will intercept the call within
    a parallel context and divert it to the per-context heap.

    Example parallel callback using NumPy:


    (Also, datrie is a Cython module, and that seems to work fine as
     well, which is neat, as it means you could sub out the entire
     Python callback with a Cythonized version, including all the
     relatively-slow-compared-to-C http header parsing that happens in


More information about the Python-ideas mailing list