[Python-ideas] multiprocessing IPC

[Mike Meyer]

On Sun, 12 Feb 2012 04:46:00 +0100
Sturla Molden <sturla at molden.no> wrote:

> Den 12.02.2012 02:52, skrev Mike Meyer:
> > First, I didn't ask about "BSD mmap", I asked about the "mmap module". 
> > They aren't the same thing. 
> Take a look at the implementation.

True, but we're talking about an API, not a specific implementation.

> >> When working with multiprocessing for a while, one comes to the
> >> conclusion that we really need named kernel objects.
> > And both the BSD mmap (at least in recent systems) and the mmap module
> > provide objects with names in the file system space. IIUC, while there
> > are systems that won't let you create anonymous objects (like early
> > versions of the mmap module), there aren't any - at least any longer -
> > that won't let you create named objects.
> Sure, you can memory map named files. You can even memory map from 
> /dev/shm on a system that supports it, if you are willing to reserve 
> some RAM for ramdisk.

And that's *not* the anonymous kernel object you complained about
getting from mmap.

> But apart from that, show me how you would use the mmap module to make 
> named shared memory on Linux or Windows. No, memory mapping file object 
> -1 or 0 don't count, you get an anonymous memory mapping.

The linux mmap has the same arguments as the BSD one, so I'd expect it
to work the same. I expect that the Python core will have made the
semantics work properly on Windows, but don't really care, and don't
have a Windows system to test it on. And that's why I'm talking about
the API, not the implementation.

> Here is a task for you to try:
> 
> 1. start a process
> 2. in the new process, create some shared memory (use the mmap module)
> 3. make the parent process get access to it (should be easy, right?)
> Can you do this? No?

Works exactly like I'd expect it to.

> Show me how you would code this.

Here's the code that creates the shared file:

    share_name = '/tmp/xyzzy'
    with open(share_name, 'wb') as f:
        f.write(b'hello')

Here's the code for the child:

    with open(share_name, 'r+b') as f:
        share = mmap(f.fileno(), 0)
        share[:5] = b'gone\n'

Here's the code for the parent:

    child = Process(target=proc)
    child.start()
    with open(share_name, mode='r+b') as f:
        share = mmap(f.fileno(), 0)
        while share[0] == ord('h'):
            sleep(1)
        print('main:', share.readline())

> > >  Use named kernel objects for IPC, pickle the name.
> > You don't need to pickle the name if you use mmap's native name system
> > - it's just a string.
> Sure, multiprocessing does not pickle strings objects. Or whatever. Have 
> you ever looked at the code?

I didn't say multiprocessing wouldn't pickle the name, *or* anything
else about the multiprocessing module. I said *you* didn't need to
pickle it. And I didn't. Did you read what I wrote?
  
> Every object passed in the "args" keyword argument to 
> multiprocessing.Process is pickled. Same thing for any object you pass 
> to multiprocessing.Queue.

Yes, but we're not talking about multiprocessing.Queue. We're talking
about mmap. multiprocessing.Queue doesn't use mmap. For that, you want
to us multiprocessing.Value and multiprocessing.Array.

> Look at the code.

Look at the text.

     <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org