Barrier Object in asyncio lib
Hi,the list, I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help. Yves
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races. With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running. Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7... Code of Conduct: http://python.org/psf/codeofconduct/
It does make sense to have a barrier synchronization primitive for asyncio. The idea is to make a coroutine block until at least X coroutines are waiting to enter the barrier. This is very useful, if certain actions need to be synchronized. Recently, I had to implement a barier myself for our use case. See code below: It is simple to implement, but I too would like to have one for asyncio, in order to be consistent with the concurrency primitives we have for threading. Jonathan class Barier: """ Make a coroutine block until there are at least X waiters. Similar to the threading Barier objects but for asyncio: https://docs.python.org/3/library/threading.html#barrier-objects """ def __init__(self, parties: int) -> None: self.parties = parties self._waiting: int self._event = asyncio.Event() def add_one(self) -> None: self._waiting += 1 if self._waiting == self.parties: self._event.set() async def wait(self, worker: "Worker") -> None: """ Wait until all we have at least `parties` waiters. """ self.add_one() await self._event.wait() Le jeu. 25 févr. 2021 à 16:42, Barry Scott <barry@barrys-emacs.org> a écrit :
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races.
With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running.
Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX... Code of Conduct: http://python.org/psf/codeofconduct/
On 25 Feb 2021, at 17:15, Jonathan Slenders <jonathan@slenders.be> wrote:
It does make sense to have a barrier synchronization primitive for asyncio. The idea is to make a coroutine block until at least X coroutines are waiting to enter the barrier. This is very useful, if certain actions need to be synchronized.
I do most of my async coding with twisted where what you calling a barrier is a DeferredList. The way its used is that you add in all the deferreds that you want to complete before you continue into the list. Once all the deferered have competed the DefferedList completes and its callback is run. Barry
Recently, I had to implement a barier myself for our use case. See code below:
It is simple to implement, but I too would like to have one for asyncio, in order to be consistent with the concurrency primitives we have for threading.
Jonathan
class Barier: """ Make a coroutine block until there are at least X waiters.
Similar to the threading Barier objects but for asyncio: https://docs.python.org/3/library/threading.html#barrier-objects <https://docs.python.org/3/library/threading.html#barrier-objects> """
def __init__(self, parties: int) -> None: self.parties = parties self._waiting: int self._event = asyncio.Event()
def add_one(self) -> None: self._waiting += 1 if self._waiting == self.parties: self._event.set()
async def wait(self, worker: "Worker") -> None: """ Wait until all we have at least `parties` waiters. """ self.add_one() await self._event.wait()
Le jeu. 25 févr. 2021 à 16:42, Barry Scott <barry@barrys-emacs.org <mailto:barry@barrys-emacs.org>> a écrit :
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com <mailto:yduprat@gmail.com>> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races.
With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running.
Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ <https://mail.python.org/mailman3/lists/python-ideas.python.org/> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7... <https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ <https://mail.python.org/mailman3/lists/python-ideas.python.org/> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX... <https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
Barry, What you describe sounds like `asyncio.gather(...)` if I understand correctly. The thing with a Barier is that it's usable in situations where we don't know the other tasks. Maybe there is no reference to them from the current scope. Maybe they are even not yet created. It certainly can be done with a list of `asyncio.Future` and `asyncio.gather(...)`, but that's a lot of boilerplate. IMHO, Yves is right. For both asyncio and threading, we have Lock, Event, Condition, Semaphore and BoundedSemaphore. Only Barier is missing among the asyncio primitives. (RLock doesn't make sense.) (I guess we can probably go to bugs.python.org with this proposal.) Jonathan Le jeu. 25 févr. 2021 à 23:38, Barry Scott <barry@barrys-emacs.org> a écrit :
On 25 Feb 2021, at 17:15, Jonathan Slenders <jonathan@slenders.be> wrote:
It does make sense to have a barrier synchronization primitive for asyncio. The idea is to make a coroutine block until at least X coroutines are waiting to enter the barrier. This is very useful, if certain actions need to be synchronized.
I do most of my async coding with twisted where what you calling a barrier is a DeferredList.
The way its used is that you add in all the deferreds that you want to complete before you continue into the list. Once all the deferered have competed the DefferedList completes and its callback is run.
Barry
Recently, I had to implement a barier myself for our use case. See code below:
It is simple to implement, but I too would like to have one for asyncio, in order to be consistent with the concurrency primitives we have for threading.
Jonathan
class Barier: """ Make a coroutine block until there are at least X waiters.
Similar to the threading Barier objects but for asyncio: https://docs.python.org/3/library/threading.html#barrier-objects """
def __init__(self, parties: int) -> None: self.parties = parties self._waiting: int self._event = asyncio.Event()
def add_one(self) -> None: self._waiting += 1 if self._waiting == self.parties: self._event.set()
async def wait(self, worker: "Worker") -> None: """ Wait until all we have at least `parties` waiters. """ self.add_one() await self._event.wait()
Le jeu. 25 févr. 2021 à 16:42, Barry Scott <barry@barrys-emacs.org> a écrit :
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races.
With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running.
Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX... Code of Conduct: http://python.org/psf/codeofconduct/
On 26 Feb 2021, at 08:31, Jonathan Slenders <jonathan@slenders.be> wrote:
Barry,
What you describe sounds like `asyncio.gather(...)` if I understand correctly.
The thing with a Barier is that it's usable in situations where we don't know the other tasks. Maybe there is no reference to them from the current scope. Maybe they are even not yet created. It certainly can be done with a list of `asyncio.Future` and `asyncio.gather(...)`, but that's a lot of boilerplate.
IMHO, Yves is right. For both asyncio and threading, we have Lock, Event, Condition, Semaphore and BoundedSemaphore. Only Barier is missing among the asyncio primitives. (RLock doesn't make sense.)
Why would you need locks for async? Is it to sync with things outside of the async process? With the large and complex async app I work on there are no locks at all.
(I guess we can probably go to bugs.python.org <http://bugs.python.org/> with this proposal.)
Having shown that a Barrier for async is a missing piece it would be good to get a thumbs up here. Barry
Jonathan
Le jeu. 25 févr. 2021 à 23:38, Barry Scott <barry@barrys-emacs.org <mailto:barry@barrys-emacs.org>> a écrit :
On 25 Feb 2021, at 17:15, Jonathan Slenders <jonathan@slenders.be <mailto:jonathan@slenders.be>> wrote:
It does make sense to have a barrier synchronization primitive for asyncio. The idea is to make a coroutine block until at least X coroutines are waiting to enter the barrier. This is very useful, if certain actions need to be synchronized.
I do most of my async coding with twisted where what you calling a barrier is a DeferredList.
The way its used is that you add in all the deferreds that you want to complete before you continue into the list. Once all the deferered have competed the DefferedList completes and its callback is run.
Barry
Recently, I had to implement a barier myself for our use case. See code below:
It is simple to implement, but I too would like to have one for asyncio, in order to be consistent with the concurrency primitives we have for threading.
Jonathan
class Barier: """ Make a coroutine block until there are at least X waiters.
Similar to the threading Barier objects but for asyncio: https://docs.python.org/3/library/threading.html#barrier-objects <https://docs.python.org/3/library/threading.html#barrier-objects> """
def __init__(self, parties: int) -> None: self.parties = parties self._waiting: int self._event = asyncio.Event()
def add_one(self) -> None: self._waiting += 1 if self._waiting == self.parties: self._event.set()
async def wait(self, worker: "Worker") -> None: """ Wait until all we have at least `parties` waiters. """ self.add_one() await self._event.wait()
Le jeu. 25 févr. 2021 à 16:42, Barry Scott <barry@barrys-emacs.org <mailto:barry@barrys-emacs.org>> a écrit :
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com <mailto:yduprat@gmail.com>> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races.
With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running.
Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ <https://mail.python.org/mailman3/lists/python-ideas.python.org/> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7... <https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ <https://mail.python.org/mailman3/lists/python-ideas.python.org/> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX... <https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
Why would you need locks for async? Is it to sync with things outside of the async process?
`asyncio.Lock` is needed to lock across async operations. (If there is an `await` in the body for the lock). Le ven. 26 févr. 2021 à 10:45, Barry Scott <barry@barrys-emacs.org> a écrit :
On 26 Feb 2021, at 08:31, Jonathan Slenders <jonathan@slenders.be> wrote:
Barry,
What you describe sounds like `asyncio.gather(...)` if I understand correctly.
The thing with a Barier is that it's usable in situations where we don't know the other tasks. Maybe there is no reference to them from the current scope. Maybe they are even not yet created. It certainly can be done with a list of `asyncio.Future` and `asyncio.gather(...)`, but that's a lot of boilerplate.
IMHO, Yves is right. For both asyncio and threading, we have Lock, Event, Condition, Semaphore and BoundedSemaphore. Only Barier is missing among the asyncio primitives. (RLock doesn't make sense.)
Why would you need locks for async? Is it to sync with things outside of the async process?
With the large and complex async app I work on there are no locks at all.
(I guess we can probably go to bugs.python.org with this proposal.)
Having shown that a Barrier for async is a missing piece it would be good to get a thumbs up here.
Barry
Jonathan
Le jeu. 25 févr. 2021 à 23:38, Barry Scott <barry@barrys-emacs.org> a écrit :
On 25 Feb 2021, at 17:15, Jonathan Slenders <jonathan@slenders.be> wrote:
It does make sense to have a barrier synchronization primitive for asyncio. The idea is to make a coroutine block until at least X coroutines are waiting to enter the barrier. This is very useful, if certain actions need to be synchronized.
I do most of my async coding with twisted where what you calling a barrier is a DeferredList.
The way its used is that you add in all the deferreds that you want to complete before you continue into the list. Once all the deferered have competed the DefferedList completes and its callback is run.
Barry
Recently, I had to implement a barier myself for our use case. See code below:
It is simple to implement, but I too would like to have one for asyncio, in order to be consistent with the concurrency primitives we have for threading.
Jonathan
class Barier: """ Make a coroutine block until there are at least X waiters.
Similar to the threading Barier objects but for asyncio: https://docs.python.org/3/library/threading.html#barrier-objects """
def __init__(self, parties: int) -> None: self.parties = parties self._waiting: int self._event = asyncio.Event()
def add_one(self) -> None: self._waiting += 1 if self._waiting == self.parties: self._event.set()
async def wait(self, worker: "Worker") -> None: """ Wait until all we have at least `parties` waiters. """ self.add_one() await self._event.wait()
Le jeu. 25 févr. 2021 à 16:42, Barry Scott <barry@barrys-emacs.org> a écrit :
On 25 Feb 2021, at 13:14, Yves Duprat <yduprat@gmail.com> wrote:
Hi,the list,
I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help.
I'm assuming that the barrier you are speaking of is the mechanism that is used to synchronise threads/processes running in parallel to prevent data races.
With async code that is never an issue. Each function runs to completion uninterrupted. There are no data races. Each time a async function runs it can know that the state of the objects it uses will not be changed while it is running.
Barry
Yves _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPX... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Feb 26, 2021 at 10:09 AM Yves Duprat <yduprat@gmail.com> wrote:
I was expecting an explanation about the initial request. Is there an oversight (??) or an another reason to not have a Barrier primitive in asyncio ?
Probably because nobody working on asyncio at the time had any experience using threading.Barrier. I know that I have used all the other primitives, but I don't recall ever having used Barrier. I don't think there's anything deeper than that. You should probably just open a bpo issue (mention this thread) and submit a PR linked there. I can't promise to review it though. :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Could somone provide a concrete example on a proposed use of such an asyncio.Barrier? On Fri, 2021-02-26 at 14:19 -0800, Guido van Rossum wrote:
On Fri, Feb 26, 2021 at 10:09 AM Yves Duprat <yduprat@gmail.com> wrote:
I was expecting an explanation about the initial request. Is there an oversight (??) or an another reason to not have a Barrier primitive in asyncio ?
Probably because nobody working on asyncio at the time had any experience using threading.Barrier. I know that I have used all the other primitives, but I don't recall ever having used Barrier. I don't think there's anything deeper than that. You should probably just open a bpo issue (mention this thread) and submit a PR linked there. I can't promise to review it though. :-) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GNUCKA... Code of Conduct: http://python.org/psf/codeofconduct/
Sorry, didn't see Jonathan's example. On Sat, 2021-02-27 at 01:12 +0000, Paul Bryan wrote:
Could somone provide a concrete example on a proposed use of such an asyncio.Barrier?
On Fri, 2021-02-26 at 14:19 -0800, Guido van Rossum wrote:
On Fri, Feb 26, 2021 at 10:09 AM Yves Duprat <yduprat@gmail.com> wrote:
I was expecting an explanation about the initial request. Is there an oversight (??) or an another reason to not have a Barrier primitive in asyncio ?
Probably because nobody working on asyncio at the time had any experience using threading.Barrier. I know that I have used all the other primitives, but I don't recall ever having used Barrier. I don't think there's anything deeper than that. You should probably just open a bpo issue (mention this thread) and submit a PR linked there. I can't promise to review it though. :-) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GNUCKA... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WIQBCD... Code of Conduct: http://python.org/psf/codeofconduct/
Hello! If anyone didn't note, there's a PR about this topic :) https://github.com/python/cpython/pull/24903 Cheers!
participants (6)
-
Barry Scott -
Emmanuel Arias -
Guido van Rossum -
Jonathan Slenders -
Paul Bryan -
Yves Duprat