[Python-ideas] multiprocessing IPC

[Sturla Molden]

Den 12.02.2012 02:52, skrev Mike Meyer:
> First, I didn't ask about "BSD mmap", I asked about the "mmap module". 
> They aren't the same thing. 

Take a look at the implementation.

>> When working with multiprocessing for a while, one comes to the
>> conclusion that we really need named kernel objects.
> And both the BSD mmap (at least in recent systems) and the mmap module
> provide objects with names in the file system space. IIUC, while there
> are systems that won't let you create anonymous objects (like early
> versions of the mmap module), there aren't any - at least any longer -
> that won't let you create named objects.

Sure, you can memory map named files. You can even memory map from 
/dev/shm on a system that supports it, if you are willing to reserve 
some RAM for ramdisk.

But apart from that, show me how you would use the mmap module to make 
named shared memory on Linux or Windows. No, memory mapping file object 
-1 or 0 don't count, you get an anonymous memory mapping.

Here is a task for you to try:

1. start a process
2. in the new process, create some shared memory (use the mmap module)
3. make the parent process get access to it (should be easy, right?)

Can you do this? No?

Then try the same thing with a lock (multiprocessing.Lock) or an event.

Show me how you would code this.

>
> >  Use named kernel objects for IPC, pickle the name.
> You don't need to pickle the name if you use mmap's native name system
> - it's just a string.

Sure, multiprocessing does not pickle strings objects. Or whatever. Have 
you ever looked at the code?



> Since can use pickle, you're only dealing with small amounts of
> data.

What on earth are you talking about?

Every object passed in the "args" keyword argument to 
multiprocessing.Process is pickled. Same thing for any object you pass 
to multiprocessing.Queue.

Look at the code.



Sturla