[Python-ideas] multiprocessing IPC
mwm at mired.org
Sun Feb 12 09:02:07 CET 2012
On Sun, 12 Feb 2012 04:46:00 +0100
Sturla Molden <sturla at molden.no> wrote:
> Den 12.02.2012 02:52, skrev Mike Meyer:
> > First, I didn't ask about "BSD mmap", I asked about the "mmap module".
> > They aren't the same thing.
> Take a look at the implementation.
True, but we're talking about an API, not a specific implementation.
> >> When working with multiprocessing for a while, one comes to the
> >> conclusion that we really need named kernel objects.
> > And both the BSD mmap (at least in recent systems) and the mmap module
> > provide objects with names in the file system space. IIUC, while there
> > are systems that won't let you create anonymous objects (like early
> > versions of the mmap module), there aren't any - at least any longer -
> > that won't let you create named objects.
> Sure, you can memory map named files. You can even memory map from
> /dev/shm on a system that supports it, if you are willing to reserve
> some RAM for ramdisk.
And that's *not* the anonymous kernel object you complained about
getting from mmap.
> But apart from that, show me how you would use the mmap module to make
> named shared memory on Linux or Windows. No, memory mapping file object
> -1 or 0 don't count, you get an anonymous memory mapping.
The linux mmap has the same arguments as the BSD one, so I'd expect it
to work the same. I expect that the Python core will have made the
semantics work properly on Windows, but don't really care, and don't
have a Windows system to test it on. And that's why I'm talking about
the API, not the implementation.
> Here is a task for you to try:
> 1. start a process
> 2. in the new process, create some shared memory (use the mmap module)
> 3. make the parent process get access to it (should be easy, right?)
> Can you do this? No?
Works exactly like I'd expect it to.
> Show me how you would code this.
Here's the code that creates the shared file:
share_name = '/tmp/xyzzy'
with open(share_name, 'wb') as f:
Here's the code for the child:
with open(share_name, 'r+b') as f:
share = mmap(f.fileno(), 0)
share[:5] = b'gone\n'
Here's the code for the parent:
child = Process(target=proc)
with open(share_name, mode='r+b') as f:
share = mmap(f.fileno(), 0)
while share == ord('h'):
> > > Use named kernel objects for IPC, pickle the name.
> > You don't need to pickle the name if you use mmap's native name system
> > - it's just a string.
> Sure, multiprocessing does not pickle strings objects. Or whatever. Have
> you ever looked at the code?
I didn't say multiprocessing wouldn't pickle the name, *or* anything
else about the multiprocessing module. I said *you* didn't need to
pickle it. And I didn't. Did you read what I wrote?
> Every object passed in the "args" keyword argument to
> multiprocessing.Process is pickled. Same thing for any object you pass
> to multiprocessing.Queue.
Yes, but we're not talking about multiprocessing.Queue. We're talking
about mmap. multiprocessing.Queue doesn't use mmap. For that, you want
to us multiprocessing.Value and multiprocessing.Array.
> Look at the code.
Look at the text.
Mike Meyer <mwm at mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
More information about the Python-ideas