[Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Chris Jerdonek]

If you're already using PostgreSQL, you might also look at "advisory locks":
https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS

--Chris

On Tue, Apr 17, 2018 at 4:34 AM, Ludovic Gasc <gmludo at gmail.com> wrote:
> Hi Nickolai,
>
> Thanks for your suggestions, especially for the file system lock: We don't
> have often locks, but we must be sure it's locked.
>
> For 1) and 4) suggestions, in fact we have several systems to sync and also
> a PostgreSQL transaction, the request must be treated by the same worker
> from beginning to end and the other systems aren't idempotent at all, it's
> "old-school" proprietary systems, good luck to change that ;-)
>
> Regards.
> --
> Ludovic Gasc (GMLudo)
>
> 2018-04-17 12:46 GMT+02:00 Nickolai Novik <nickolainovik at gmail.com>:
>>
>> Hi, redis lock has own limitations and depending on your use case it may
>> or may not be suitable [1]. If possible I would redefine problem and also
>> considered:
>> 1) create worker per specific resource type to avoid locking
>> 2) optimistic locking
>> 3) File system lock like in twisted, but not sure about performance and
>> edge cases there
>> 4) make operation on resource idempotent
>>
>> [1]
>> http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
>> [2]
>> https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3d134799415/src/twisted/python/lockfile.py
>>
>> On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc <gmludo at gmail.com> wrote:
>>>
>>> Hi Roberto,
>>>
>>> Thanks for the pointer, it's exactly the type of feedbacks I'm looking
>>> for: Ideas that are out-of-box of my confort zone.
>>> However, in our use case, we are using gunicorn, that uses forks instead
>>> of multiprocessing to my knowledge, I can't use multiprocessing without to
>>> remove gunicorn.
>>>
>>> If somebody is using aioredlock in his project, I'm interested by
>>> feedbacks.
>>>
>>> Have a nice week.
>>>
>>>
>>> --
>>> Ludovic Gasc (GMLudo)
>>>
>>> 2018-04-17 7:19 GMT+02:00 Roberto Martínez <robertomartinezp at gmail.com>:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I don't know if there is a third party solution for this.
>>>>
>>>> I think the closest you can get today using the standard library is
>>>> using a multiprocessing.manager().Lock (which can be shared among processes)
>>>> and call the lock.acquire() function with asyncio.run_in_executor(), using a
>>>> ThreadedPoolExecutor to avoid blocking the asyncio event loop.
>>>>
>>>> Best regards,
>>>> Roberto
>>>>
>>>>
>>>> El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc (<gmludo at gmail.com>)
>>>> escribió:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm looking for a equivalent of asyncio.Lock
>>>>> (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
>>>>> shared between several processes on the same server, because I'm migrating a
>>>>> daemon from mono-worker to multi-worker pattern.
>>>>>
>>>>> For now, the closest solution in term of API seems aioredlock:
>>>>> https://github.com/joanvila/aioredlock#aioredlock
>>>>> But I'm not a big fan to use polling nor with a timeout because the
>>>>> lock I need is very critical, I prefer to block the code than unlock with
>>>>> timeout.
>>>>>
>>>>> Do I miss a new awesome library or do you have an easier approach ?
>>>>>
>>>>> Thanks for your responses.
>>>>> --
>>>>> Ludovic Gasc (GMLudo)
>>>
>>>
>
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>