[Async-sig] "read-write" synchronization

[Dima Tisnek]

- self.cond.wait()
+ await self.cond.wait()

I've no tests for this :P

On 26 June 2017 at 21:37, Dima Tisnek <dimaqq at gmail.com> wrote:
> Chris, here's a simple RWLock implementation and analysis:
>
> ```
> import asyncio
>
>
> class RWLock:
>     def __init__(self):
>         self.cond = asyncio.Condition()
>         self.readers = 0
>         self.writer = False
>
>     async def lock(self, write=False):
>         async with self.cond:
>             # write requested: there cannot be readers or writers
>             # read requested: there can be other readers but not writers
>             while self.readers and write or self.writer:
>                 self.cond.wait()
>             if write: self.writer = True
>             else: self.readers += 1
>             # self.cond.notifyAll() would be good taste
>             # however no waiters can be unblocked by this state change
>
>     async def unlock(self, write=False):
>         async with self.cond:
>             if write: self.writer = False
>             else: self.readers -= 1
>             self.cond.notifyAll()  # notify (one) could be used `if not write:`
> ```
>
> Note that `.unlock` cannot validate that it's called by same coroutine
> as `.lock` was.
> That's because there's no concept for "current_thread" for coroutines
> -- there can be many waiting on each other in the stack.
>
> Obv., this code could be nicer:
> * separate context managers for read and write cases
> * .unlock can be automatic (if self.writer: unlock_for_write()) at the
> cost of opening doors wide open to bugs
> * policy can be introduced if `.lock` identified itself (by an
> object(), since there's no thread id) in shared state
> * notifyAll() makes real life use O(N^2) for N being number of
> simultaneous write lock requests
>
> Feel free to use it :)
>
>
>
> On 26 June 2017 at 20:21, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>> On Mon, Jun 26, 2017 at 10:02 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> Chris, coming back to your use-case.
>>> Do you want to synchronise side-effect creation/deletion for the
>>> sanity of side-effects only?
>>> Or do you imply that callers' actions are synchronised too?
>>> In other words, do your callers use those directories out of band?
>>
>> If I understand your question, the former. The callers aren't / need
>> not be synchronized, and they aren't aware of the underlying
>> synchronization happening inside the higher-level create() and
>> delete() functions they would be using. (These are the two
>> higher-level functions described in my pseudocode.)
>>
>> The synchronization is needed inside these create() and delete()
>> functions since the low-level directory operations occur in different
>> threads (because they are wrapped by run_in_executor()).
>>
>> --Chris
>>
>>>
>>>
>>> P.S./O.T. when it comes to directories, you probably want hierarchical
>>> locks rather than RW.
>>>
>>>
>>> On 26 June 2017 at 11:28, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>>>> On Mon, Jun 26, 2017 at 1:43 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>> Perhaps you can share your use-case, both as pseudo-code and a link to
>>>>> real code.
>>>>>
>>>>> I'm specifically interested to see why/where you'd like to use a
>>>>> read-write async lock, to evaluate if this is something common or
>>>>> specific, and if, perhaps, some other paradigm (like queue, worker
>>>>> pool, ...) may be more useful in general case.
>>>>>
>>>>> I'm also curious if a full set of async sync primitives may one day
>>>>> lead to async monitors. Granted, simple use of async monitor is really
>>>>> a future/promise, but perhaps there are complex use cases in the
>>>>> UI/react domain with its promise/stream dichotomy.
>>>>
>>>> Thank you, Dima. In my last email I shared pseudo-code for an approach
>>>> to read-write synchronization that is independent of use case. [1]
>>>>
>>>> For the use case, my original purpose in mind was to synchronize many
>>>> small file operations on disk like creating and removing directories
>>>> that possibly share intermediate segments. The real code isn't public.
>>>> But these would be operations like os.makedirs() and os.removedirs()
>>>> that would be wrapped by loop.run_in_executor() to be non-blocking.
>>>> The directory removal using os.removedirs() is the operation I thought
>>>> should require exclusive access, so as not to interfere with directory
>>>> creations in progress.
>>>>
>>>> Perhaps a simpler, dirtier approach would be not to synchronize at all
>>>> and simply retry directory creations that fail until they succeed.
>>>> That could be enough to handle rare cases where simultaneous creation
>>>> and removal causes an error. You could view this an EAFP approach.
>>>>
>>>> Either way, I think the process of thinking through patterns for
>>>> read-write synchronization is helpful for getting a better general
>>>> feel and understanding of async.
>>>>
>>>> --Chris
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> d.
>>>>>
>>>>> On 25 June 2017 at 23:13, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>>>>>> I'm relatively new to async programming in Python and am thinking
>>>>>> through possibilities for doing "read-write" synchronization.
>>>>>>
>>>>>> I'm using asyncio, and the synchronization primitives that asyncio
>>>>>> exposes are relatively simple [1]. Have options for async read-write
>>>>>> synchronization already been discussed in any detail?
>>>>>>
>>>>>> I'm interested in designs where "readers" don't need to acquire a lock
>>>>>> -- only writers. It seems like one way to deal with the main race
>>>>>> condition I see that comes up would be to use loop.time(). Does that
>>>>>> ring a bell, or might there be a much simpler way?
>>>>>>
>>>>>> Thanks,
>>>>>> --Chris
>>>>>>
>>>>>>
>>>>>> [1] https://docs.python.org/3/library/asyncio-sync.html
>>>>>> _______________________________________________
>>>>>> Async-sig mailing list
>>>>>> Async-sig at python.org
>>>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/