[Python-ideas] solving multi-core Python

Trent Nelson trent at snakebite.org
Tue Jun 23 20:29:40 CEST 2015


On Sun, Jun 21, 2015 at 12:40:43PM +0200, Stefan Behnel wrote:
> Nick Coghlan schrieb am 21.06.2015 um 12:25:
> > On 21 June 2015 at 19:48, Antoine Pitrou wrote:
> >> On Sun, 21 Jun 2015 16:31:33 +1000 Nick Coghlan wrote:
> >>>
> >>> For inter-interpreter communication, the worst case scenario is having
> >>> to rely on a memcpy based message passing system (which would still be
> >>> faster than multiprocessing's serialisation + IPC overhead)
> >>
> >> And memcpy() updates pointer references to dependent objects magically?
> >> Surely you meant the memdeepcopy() function that's part of every
> >> standard C library!
> > 
> > We already have the tools to do deep copies of object trees (although
> > I'll concede I *was* actually thinking in terms of the classic C/C++
> > mistake of carelessly copying pointers around when I wrote that
> > particular message). One of the options for deep copies tends to be a
> > pickle/unpickle round trip, which will still incur the serialisation
> > overhead, but not the IPC overhead.
> > 
> > "Faster message passing than multiprocessing" sets the baseline pretty
> > low, after all.
> > 
> > However, this is also why Eric mentions the notions of object
> > ownership or limiting channels to less than the full complement of
> > Python objects. As an *added* feature at the Python level, it's
> > possible to initially enforce restrictions that don't exist in the C
> > level subinterpeter API, and then work to relax those restrictions
> > over time.
> 
> If objects can make it explicit that they support sharing (and preferably
> are allowed to implement the exact details themselves), I'm sure we'll find
> ways to share NumPy arrays across subinterpreters. That feature alone tends
> to be a quick way to make a lot of people happy.

    FWIW, the following commit was all it took to get NumPy playing
    nicely with PyParallel:

        https://github.com/pyparallel/numpy/commit/046311ac1d66cec789fa8fd79b1b582a3dea26a8

    It uses thread-local buckets instead of static ones, and calls out
    to PyMem_Raw(Malloc|Realloc|Calloc|Free) instead of the normal libc
    counterparts.  This means PyParallel will intercept the call within
    a parallel context and divert it to the per-context heap.

    Example parallel callback using NumPy:

        https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/wiki/wiki.py?at=3.3-px#cl-285

    (Also, datrie is a Cython module, and that seems to work fine as
     well, which is neat, as it means you could sub out the entire
     Python callback with a Cythonized version, including all the
     relatively-slow-compared-to-C http header parsing that happens in
     async.http.server.)

        Trent.


More information about the Python-ideas mailing list