[Python-ideas] multiprocessing IPC

[shibturn]

On 12/02/2012 8:33pm, Sturla Molden wrote:
> It seems that on Linux /tmp is backed by shared memory.
>
> Which sounds rather strange to a Windows user, as the raison d'etre for
> tempfiles is temporary storage space that goes beyond physial RAM.

In reality /tmp is backed by swap space, so physical RAM does not impose 
a limit.  Anonymous mmaps are also backed by swap space.

> I've also read that the use of ftruncate in this context can result in
> SIGBUS.

Isn't that if you truncate the file to a smaller size *after* it has 
been mapped.  As far as I am aware, using ftruncate to set the length 
*before* it can be mapped for the first time is standard practice and 
harmless.

>> Below is Blob class which seems to work. Note that the process which
>> created the blob needs to wait for the other process to unpickle it
>> before allowing it to be garbage collected.
>>
>
> I would look at kernel refcounts before unlinking. (But I am not that
> familiar with Linux.)

Even if you have automatic refcounting like on Windows, you still need 
to cope with lifetime management issues.  If you put an object on a 
queue it may be a long time before the target process will unpickle the 
object and increase its refcount, and you must not decref the object 
until it has, or else it will disappear.

I don't know how to get the ref count for a file descriptor on Unix. 
(And posix shared memory does not seems to get a refcount either, even 
though System V shared memory does.)

sbt