[Python-ideas] multiprocessing IPC

[shibturn]

On 12/02/2012 3:46am, Sturla Molden wrote:
> 1. start a process
> 2. in the new process, create some shared memory (use the mmap module)
> 3. make the parent process get access to it (should be easy, right?)

As Mike says, on Unix you can just create a file in /tmp to back an 
mmap.  On Linux, posix mmaps created with shm_open() seem to be normal 
files on a tmpfs file system, usually /dev/shm.  Since /tmp is also 
usually a tmpfs file system on Linux, I assume this whould be equivalent 
in terms of overhead.

On Windows you can use the tagname argument of mmap.mmap().

Maybe a BinaryBlob wrapper class could be created which lets an mmap be 
"pickled by reference".  Managing life time and reliable cleanup might 
be awkward though.

If the pickle overhead is the problem you could try 
Connection.send_bytes() and Connection.recv_bytes().  I suppose Queue 
objects could grow put_bytes() and get_bytes() methods too.  Or a 
BytesQueue class could be created.

> Can you do this? No?
>
> Then try the same thing with a lock (multiprocessing.Lock) or an event.

I have a patch (http://bugs.python.org/issue8713) to make 
multiprocessing on Unix work with fork+exec which has to do this because 
semaphores cannot be inherited across exec.  Making sure all the named 
semaphores get removed if the program terminates abnormally is a bit 
awkward though.  It could be modified to make them picklable in general.

On Windows dealing with "named objects" is easier since they are 
refcounted by the operating system and deleted when no more processes 
have handles for them.

If you make a feature request at bugs.python.org I might work on a patch.

Cheers

sbt