[Python-ideas] solving multi-core Python

[Sturla Molden]

Devin Jeanpierre <jeanpierreda at gmail.com>
wrote:

> The comparison is unfair -- if you can share between subinterpreters
> using memcpy, then you can share between processes using just a socket
> write, and multiprocessing becomes nearly just as fast.

That is the main issue here. Writing to a pipe or a Unix socket is
implemented with a memcpy in the kernel. So there is just a tiny constant
overhead compared to using memcpy within a process. And with shared memory
as IPC even this tiny overhead can be removed. 

The main overhead in communicating Python objects in multiprocessing is the
serialization with pickle. So there is basically nothing to gain unless
this part can be omitted.

There is an errorneous belief among Windows programmers tht "IPC is slow".
But that is because they are using out-proc DCOM server, CORBA, XMLRPC or
something equally atrocious. A plain named pipe transaction is not in any
way slow on Windows.


Sturla