Shared Semaphores for synchronisation across unrelated processes
Hi, Python has integrated shared memory into standard library starting from 3.8 (https://docs.python.org/3/library/multiprocessing.shared_memory.html <https://docs.python.org/3/library/multiprocessing.shared_memory.html>), which provides a user friendly API to access shared memory across unrelated processes using names. But, there are no synchronisation mechanisms present in the standard library to prevent race conditions when shared memory is accessed across unrelated processes. I had earlier created an enhancement issue <https://bugs.python.org/issue38035> at bugs.python.org <http://bug.python.org/>, which contains more detailed discussion on the same. I am posting this here after a suggestion from a Python contributor. Feedback on the same will be very helpful.
On 05/06/2020 16:18, Vinay Sharma via Python-ideas wrote:
Python has integrated shared memory into standard library starting from 3.8 (https://docs.python.org/3/library/multiprocessing.shared_memory.html <https://docs.python.org/3/library/multiprocessing.shared_memory.html>), which provides a user friendly API to access shared memory across unrelated processes using names. But, there are no synchronisation mechanisms present in the standard library to prevent race conditions when shared memory is accessed across unrelated processes.
I had earlier created an enhancement issue <https://bugs.python.org/issue38035> at bugs.python.org <http://bug.python.org/>, which contains more detailed discussion on the same. I am posting this here after a suggestion from a Python contributor.
These are very nice statements. What is you actually want? -- Rhodri James *-* Kynesim Ltd
Hi Rhodri wrote:
These are very nice statements. What is you actually want?
Good question. Vinay provide a link to the issue he opened. In his opening message he wrote:
This behaviour works well when the file descriptors of these semaphores can be shared with children processes.
But, it doesn't work when an unrelated process which needs access to shared semaphore tries to open it.
Seems clear enough to me, but I'm not an expert. Here's the URL: https://bugs.python.org/issue38035 I think this answers Rhodri's question. I'm sure Vinay will correct me if I've mispoken, and add further comments appropriate. By the way, it was core developer Tal Einat who suggested that Vinay post to this list. kind regards Jonathan
On 05/06/2020 17:47, Jonathan Fine wrote:
Hi
Rhodri wrote:
These are very nice statements. What is you actually want?
Good question. Vinay provide a link to the issue he opened. In his opening message he wrote:
This behaviour works well when the file descriptors of these semaphores can be shared with children processes.
But, it doesn't work when an unrelated process which needs access to shared semaphore tries to open it.
Again, these are statements. I can take a guess at what Vinay wants, but since that involves a rather large worm can I'd rather they actually stated it outright. -- Rhodri James *-* Kynesim Ltd
Hi Rhodri wrote:
Again, these are statements. I can take a guess at what Vinay wants, but since that involves a rather large worm can I'd rather they actually stated it outright.
I'm puzzled. Reading https://bugs.python.org/issue38035, to me it seems that Ned Deily, Ido Michael and Tal Einat all understood Vinay's request well enough to suggest a way forward, a way to provide what Vinay wants. However Rhodri is able only to guess what Vinay wants. I suggest Rhodri and I bow out of this discussion for now, and allow Vinay to clarify, and others to contribute, if they wish. best regards Jonathan
On 05/06/2020 18:56, Jonathan Fine wrote:
Hi
Rhodri wrote:
Again, these are statements. I can take a guess at what Vinay wants, but since that involves a rather large worm can I'd rather they actually stated it outright.
I'm puzzled.
Reading https://bugs.python.org/issue38035, to me it seems that Ned Deily, Ido Michael and Tal Einat all understood Vinay's request well enough to suggest a way forward, a way to provide what Vinay wants. However Rhodri is able only to guess what Vinay wants.
I suggest Rhodri and I bow out of this discussion for now, and allow Vinay to clarify, and others to contribute, if they wish.
Since what I have asked for is for Vinay to clarify what is wanted, that involves no effort on my part :-) -- Rhodri James *-* Kynesim Ltd
Hi, As Barry wrote, I am asking for the ability to use shared semaphores in python across unrelated processes. So, that shared memory used across unrelated processes can be synchronised. Although, I am suggesting this only because I am unable to synchronise shared memory across unrelated processes, therefore I also welcome other ways apart from shared semaphores, if they seem better in any way.
On 05-Jun-2020, at 11:55 PM, Rhodri James <rhodri@kynesim.co.uk> wrote:
On 05/06/2020 18:56, Jonathan Fine wrote:
Hi Rhodri wrote:
Again, these are statements. I can take a guess at what Vinay wants, but since that involves a rather large worm can I'd rather they actually stated it outright.
I'm puzzled. Reading https://bugs.python.org/issue38035, to me it seems that Ned Deily, Ido Michael and Tal Einat all understood Vinay's request well enough to suggest a way forward, a way to provide what Vinay wants. However Rhodri is able only to guess what Vinay wants. I suggest Rhodri and I bow out of this discussion for now, and allow Vinay to clarify, and others to contribute, if they wish.
Since what I have asked for is for Vinay to clarify what is wanted, that involves no effort on my part :-)
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JLOTRR... Code of Conduct: http://python.org/psf/codeofconduct/
Hi, I posted this here because a core Developer (Tal Einat), asked me to, although I am not sure what is the protocol from here. Does this enhancement request look reasonable ? If yes, I would love to open a PR. If not, any suggestions, feedback is very welcome.
On 06-Jun-2020, at 9:56 AM, Vinay Sharma via Python-ideas <python-ideas@python.org> wrote:
Hi, As Barry wrote, I am asking for the ability to use shared semaphores in python across unrelated processes. So, that shared memory used across unrelated processes can be synchronised.
Although, I am suggesting this only because I am unable to synchronise shared memory across unrelated processes, therefore I also welcome other ways apart from shared semaphores, if they seem better in any way.
On 05-Jun-2020, at 11:55 PM, Rhodri James <rhodri@kynesim.co.uk> wrote:
On 05/06/2020 18:56, Jonathan Fine wrote:
Hi Rhodri wrote:
Again, these are statements. I can take a guess at what Vinay wants, but since that involves a rather large worm can I'd rather they actually stated it outright.
I'm puzzled. Reading https://bugs.python.org/issue38035, to me it seems that Ned Deily, Ido Michael and Tal Einat all understood Vinay's request well enough to suggest a way forward, a way to provide what Vinay wants. However Rhodri is able only to guess what Vinay wants. I suggest Rhodri and I bow out of this discussion for now, and allow Vinay to clarify, and others to contribute, if they wish.
Since what I have asked for is for Vinay to clarify what is wanted, that involves no effort on my part :-)
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JLOTRR... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UM7MJT... Code of Conduct: http://python.org/psf/codeofconduct/
On 09/06/2020 17:16, Vinay Sharma wrote:
Hi, I posted this here because a core Developer (Tal Einat), asked me to, although I am not sure what is the protocol from here. Does this enhancement request look reasonable ?
If yes, I would love to open a PR. If not, any suggestions, feedback is very welcome.
I've been thinking about this for a couple of days now, and I think for what little it's worth my answer is mostly yes. You're basically asking for Named Semaphores, which are a perfectly sensible thing to want for inter-program communications. I hesitate only because I'm more used to thinking about these things in terms of message-passing protocols, but that's no reason to reject your request. My only bit of bikeshedding is that they ought to be a different class to the basic Semaphore to make it clear they have a different scope. Is it worth broadening out the discussion to other sorts of named objects? Named Pipes particularly spring to my mind... -- Rhodri James *-* Kynesim Ltd
Is it worth broadening out the discussion to other sorts of named objects? Named Pipes particularly spring to my mind...
The main reason why I requested this enhancement was to ensure that multiple processes can read/write a shared memory segment without corrupting that data in it. To ensure the same shared semaphores came to mind. Each shared memory segment, can have a corresponding shared semaphore. I don't think using named pipes would be a good way to ensure the same. Although, if you think otherwise please suggest so. I also thought keeping a semphore inside the shared memory segment would be much cleaner and easier to do, but unfortunately macos has deprecated sem_t. Other ways include storing an integer in the shared memory segment, which could be altered via atomic operations, this integer can be used as a counting semaphore. Python uses the following atomic operations to implement GIL, which can also be used above. https://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/Atomic-Builtins.html Also, the reason for asking named shared semaphore's was that they already exist in python library, but they are cleaned up, as soon as the process which created them dies. Since, macos has deprecated sem_t, the Semaphore in multiprocessor.synchronize uses SemLock class which in turn uses named semaphores, the only issue is that these semaphores are erased as soon as the process which created them dies. Therefore, I thought it would be much easier to provide a public API to use named semaphores, since they are already implemented, after a few changes of course.
On 09-Jun-2020, at 10:32 PM, Rhodri James <rhodri@kynesim.co.uk> wrote:
On 09/06/2020 17:16, Vinay Sharma wrote:
Hi, I posted this here because a core Developer (Tal Einat), asked me to, although I am not sure what is the protocol from here. Does this enhancement request look reasonable ? If yes, I would love to open a PR. If not, any suggestions, feedback is very welcome.
I've been thinking about this for a couple of days now, and I think for what little it's worth my answer is mostly yes. You're basically asking for Named Semaphores, which are a perfectly sensible thing to want for inter-program communications. I hesitate only because I'm more used to thinking about these things in terms of message-passing protocols, but that's no reason to reject your request. My only bit of bikeshedding is that they ought to be a different class to the basic Semaphore to make it clear they have a different scope.
Is it worth broadening out the discussion to other sorts of named objects? Named Pipes particularly spring to my mind...
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FGOSCC... Code of Conduct: http://python.org/psf/codeofconduct/
On 09/06/2020 19:04, Vinay Sharma wrote:
The main reason why I requested this enhancement was to ensure that multiple processes can read/write a shared memory segment without corrupting that data in it.
To ensure the same shared semaphores came to mind. Each shared memory segment, can have a corresponding shared semaphore.
Yes, that's a perfectly good plan.
I don't think using named pipes would be a good way to ensure the same. Although, if you think otherwise please suggest so.
I understand, I was asking people in general if separately from your particular use case we might want to consider other shared primatives. I can think of other use cases where cross-process pipes, mutexes and so on could be useful. -- Rhodri James *-* Kynesim Ltd
On 5 Jun 2020, at 16:18, Vinay Sharma via Python-ideas <python-ideas@python.org> wrote:
Hi, Python has integrated shared memory into standard library starting from 3.8 (https://docs.python.org/3/library/multiprocessing.shared_memory.html <https://docs.python.org/3/library/multiprocessing.shared_memory.html>), which provides a user friendly API to access shared memory across unrelated processes using names. But, there are no synchronisation mechanisms present in the standard library to prevent race conditions when shared memory is accessed across unrelated processes.
I had earlier created an enhancement issue <https://bugs.python.org/issue38035> at bugs.python.org <http://bug.python.org/>, which contains more detailed discussion on the same. I am posting this here after a suggestion from a Python contributor.
Feedback on the same will be very helpful.
What you are asking for it the ability to open a named semaphores with the limitation of the current implementation that the semaphores must can only be shared with child processes? Please confirm that I have understood. If that is it sounds like a reasonable request to me. Barry
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/X4AKFF... Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
Barry Scott -
Jonathan Fine -
Rhodri James -
Vinay Sharma