[issue35813] shared memory construct to avoid need for serialization between processes

[Giampaolo Rodola']

Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

> The unlink() method is available on the SharedMemory class.  No manager is required. This is also captured in the docs.

I missed that, sorry.


>> 3) it seems "size" kwarg cannot be zero (current default)
> From the docs: When attaching to an existing shared memory block, set to 0

Mmm... this makes the API a bit weird because it's a arg which is optional only for existing memory object, not new ones. I see that internally *size* is passed to ftruncate(). Can't you just avoid calling ftruncate() if size is not passed (None)? Also, what happens if you alter the size of an existing object with a smaller value? Is the memory region overwritten?

    a = shared_memory.SharedMemory(None, size=10)
    b = shared_memory.SharedMemory(a.name, size=4)


>> Maybe something like this instead?
>>    SharedMemory(name=None, attach_if_exists=False, size=0)
> I think this misses the use case when wanting to ensure we only attach to an existing shared memory block and if it does not exist, we should raise an exception because we can not continue.

It appears this is already covered:

    >>> SharedMemory(name="non-existent")
    Traceback (most recent call last):
    ...
    _posixshmem.ExistentialError: No shared memory exists with the specified name


> I believe the two dominant use cases to address are:
> 1) I want to create a shared memory block (either with or without a pre-conceived name).
> 2) I want to attach to an existing shared memory block by its unique name.

Don't you also want to "create if it doesn't exist, else attach" as a single, atomic operation? It seems to me the only way to achieve that would be this, but it's racy:

    try:
        m = shared_memory.SharedMemory("foo", size=10)  # create
    except ExistentialError:
        m = shared_memory.SharedMemory("foo")  # attach (race condition)

Note that here I'm entering into an unknown territory, because I'm not sure if there are or should be sync primitives to "wait for another memory to join me" etc. Just mentioning it for the sake of mental lucubration. =)


>  If we move SharedMemoryManager to live in multiprocessing.managers, we need to decide how to handle (and communicate to the user appropriately) its potential absence.

How about:

    # managers.py
    try:
        from .shared_memory import SharedMemory, SharedList 
    except ImportError:
        pass
    else:
        class SharedMemoryManager: ...
        class SharedMemoryServer: ...
        del SharedMemory, SharedList


>> 4) I have some reservations about SharedMemory's "flags" and "mode" args.
> It sounds like you are agreeing with what I advocated in msg335660 (up above). Great!

If you mean drop those 2 args entirely then it probably there's no point in exposing WindowsNamedSharedMemory and PosixSharedMemory . If you mean to keep them then it seems "read_only=True" would conflict with "flags=O_RDWR". The fact that the Windows counterpart uses different constants makes me think this bitwise stuff should probably be handled internally, and exposed via friendly named-args like "read_only" / "attach_if_exists" or similar (but it's still not clear what those named args should be exactly).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35813>
_______________________________________