Nathaniel, Thanks very much for taking the time to comment. Clearing the event after waiting for it will introduce a race condition: if the sender has gone around its loop again and set the semaphore after we have woken but before we've cleared it. As you said, this stuff is tricky! The only safe way is to make the wait-and-clear atomic, which can be done with a lock; and this comes essentially back to what I'm proposing. I realise this is not a fundamental new primitive - if it was, I wouldn't be able to build it in pure Python - but I've found it extremely useful in our generic threading and processing library. You're right about what you say regarding queues; I didn't want to go into the full details of the multi-threading and multi-processing situation at hand, but I will say that we have a pipeline of tasks that can run as either threads or processes, and we want to make it easy to construct this pipeline, "wiring" it as necessary; combining command queues with data queues just gets a real mess. Richard -----Original Message----- From: Nathaniel Smith Sent: Tuesday, March 26, 2019 4:24 PM To: richard.whitehead@ieee.org Cc: Python-Ideas Subject: Re: [Python-ideas] Simpler thread synchronization using "Sticky Condition" These kinds of low-level synchronization primitives are notoriously tricky, yeah, and I'm all in favor of having better higher-level tools. But I'm not sure that AutoResetEvent adds enough to be worth it. AFAICT, you can get this behavior with an Event just fine – using your pseudocode: def sender(): while alive(): wait_for_my_data_from_hardware() send_data_to_receiver() auto_event.set() def receiver(): while alive(): auto_event.wait() auto_event.clear() # <-- this line added receive_all_data_from_sender() process_data() It's true that if we use a regular Event then the .clear() doesn't happen atomically with the wakeup, but that doesn't matter. If we call auto_event.set() and then have new data arrive, then there are two cases: 1) the new data early enough to be seen by the current call to receive_all_data_from_sender(): this is fine, the new data will be processed in this call 2) the new data arrives too late to be seen by the current call to receive_all_data_from_sender(): that means the new data arrived after the call to receive_all_data_from_sender() started, which means it arrived after auto_event.clear(), which means that the call to auto_event.set() will successfully re-arm the event and another call to receive_all_data_from_sender() will happen immediately That said, this is still very tricky. It requires careful analysis, and it's not very general (for example, if we want to support multiple receivers than we need to throw out the whole approach and do something entirely different). In Trio we've actually discussed removing Event.clear(), since it's so difficult to use correctly: https://github.com/python-trio/trio/issues/637 You said your original problem is that you have multiple event sources, and the receiver needs to listen to all of them. And based on your approach, I guess you only have one receiver, and that it's OK to couple all the event sources directly to this receiver (i.e., you're OK with passing them all a Condition object to use). Under these circumstances, wouldn't it make more sense to use a single Queue, pass it to all the sources, and have them each do queue.put((source_id, event))? That's simple to implement, hard to mess up, and can easily be extended to multiple receivers. If you want to further decouple the sources from the receiver, then one approach would be to have each source expose its own Queue independently, and then define some kind of 'select' operation (like in Golang/CSP/concurrent ML) to let the receiver read from multiple Queues simultaneously. This is non-trivial to do, but in return you get a very general and powerful construct. There's some more links and discussion here: https://github.com/python-trio/trio/issues/242
Regarding the link you sent, I don't entirely agree with the opinion expressed: if you try to use a Semaphore for this purpose you will soon find that it is "the wrong way round", it is intended to protect resources from multiple accesses, not to synchronize those multiple accesses
Semaphores are extremely generic primitives – there are a lot of different ways to use them. I think the blog post is correct that an AutoResetEvent is equivalent to a semaphore whose value is clamped so that it can't exceed 1. Your 'auto_event.set()' would be implemented as 'sem.release()', and 'auto_event.wait()' would be 'sem.acquire()'. I guess technically the semantics might be slightly different when there are multiple waiters: the semaphore wakes up exactly one waiter, while I'm not sure what your AutoResetEvent would do. But I can't see any way to use AutoResetEvent reliably with multiple waiters anyway. -n -- Nathaniel J. Smith -- https://vorpus.org