[Async-sig] "read-write" synchronization

[Chris Jerdonek]

On Mon, Jun 26, 2017 at 10:02 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
> Chris, coming back to your use-case.
> Do you want to synchronise side-effect creation/deletion for the
> sanity of side-effects only?
> Or do you imply that callers' actions are synchronised too?
> In other words, do your callers use those directories out of band?

If I understand your question, the former. The callers aren't / need
not be synchronized, and they aren't aware of the underlying
synchronization happening inside the higher-level create() and
delete() functions they would be using. (These are the two
higher-level functions described in my pseudocode.)

The synchronization is needed inside these create() and delete()
functions since the low-level directory operations occur in different
threads (because they are wrapped by run_in_executor()).

--Chris

>
>
> P.S./O.T. when it comes to directories, you probably want hierarchical
> locks rather than RW.
>
>
> On 26 June 2017 at 11:28, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>> On Mon, Jun 26, 2017 at 1:43 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> Perhaps you can share your use-case, both as pseudo-code and a link to
>>> real code.
>>>
>>> I'm specifically interested to see why/where you'd like to use a
>>> read-write async lock, to evaluate if this is something common or
>>> specific, and if, perhaps, some other paradigm (like queue, worker
>>> pool, ...) may be more useful in general case.
>>>
>>> I'm also curious if a full set of async sync primitives may one day
>>> lead to async monitors. Granted, simple use of async monitor is really
>>> a future/promise, but perhaps there are complex use cases in the
>>> UI/react domain with its promise/stream dichotomy.
>>
>> Thank you, Dima. In my last email I shared pseudo-code for an approach
>> to read-write synchronization that is independent of use case. [1]
>>
>> For the use case, my original purpose in mind was to synchronize many
>> small file operations on disk like creating and removing directories
>> that possibly share intermediate segments. The real code isn't public.
>> But these would be operations like os.makedirs() and os.removedirs()
>> that would be wrapped by loop.run_in_executor() to be non-blocking.
>> The directory removal using os.removedirs() is the operation I thought
>> should require exclusive access, so as not to interfere with directory
>> creations in progress.
>>
>> Perhaps a simpler, dirtier approach would be not to synchronize at all
>> and simply retry directory creations that fail until they succeed.
>> That could be enough to handle rare cases where simultaneous creation
>> and removal causes an error. You could view this an EAFP approach.
>>
>> Either way, I think the process of thinking through patterns for
>> read-write synchronization is helpful for getting a better general
>> feel and understanding of async.
>>
>> --Chris
>>
>>
>>>
>>> Cheers,
>>> d.
>>>
>>> On 25 June 2017 at 23:13, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>>>> I'm relatively new to async programming in Python and am thinking
>>>> through possibilities for doing "read-write" synchronization.
>>>>
>>>> I'm using asyncio, and the synchronization primitives that asyncio
>>>> exposes are relatively simple [1]. Have options for async read-write
>>>> synchronization already been discussed in any detail?
>>>>
>>>> I'm interested in designs where "readers" don't need to acquire a lock
>>>> -- only writers. It seems like one way to deal with the main race
>>>> condition I see that comes up would be to use loop.time(). Does that
>>>> ring a bell, or might there be a much simpler way?
>>>>
>>>> Thanks,
>>>> --Chris
>>>>
>>>>
>>>> [1] https://docs.python.org/3/library/asyncio-sync.html
>>>> _______________________________________________
>>>> Async-sig mailing list
>>>> Async-sig at python.org
>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/