[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 21 June 2015 at 20:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 21 Jun 2015 20:25:47 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 21 June 2015 at 19:48, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Sun, 21 Jun 2015 16:31:33 +1000
>> > Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >>
>> >> For inter-interpreter communication, the worst case scenario is having
>> >> to rely on a memcpy based message passing system (which would still be
>> >> faster than multiprocessing's serialisation + IPC overhead)
>> >
>> > And memcpy() updates pointer references to dependent objects magically?
>> > Surely you meant the memdeepcopy() function that's part of every
>> > standard C library!
>>
>> We already have the tools to do deep copies of object trees
> [...]
>> "Faster message passing than multiprocessing" sets the baseline pretty
>> low, after all.
>
> What's the goal? 10% faster? Or 10x? copy.deepcopy() uses similar
> internal mechanisms as pickle...

I'd want us to eventually aim for zero-copy speed for at least known
immutable values (int, str, float, etc), immutable containers of
immutable values (tuple, frozenset), and for types that support both
publishing and consuming data via the PEP 3118 buffer protocol without
making a copy.

For everything else I'd be fine with a starting point that was at
least no slower than multiprocessing (which shouldn't be difficult,
since we'll at least save the IPC overhead even if there are cases
where communication between subinterpreters falls back to
serialisation rather than doing something more CPU and memory
efficient).

As an implementation strategy, I'd actually suggest starting with
*only* the latter for simplicity's sake, even though it misses out on
some of the potential speed benefits of sharing an address space.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia