[Async-sig] asyncio.Lock equivalent for multiple processes

[Dima Tisnek]

Hi Ludovic,

I believe it's relatively straightforward to implement the core
functionality, if you can at first reduce it to:
* allow only one coro to wait on lock at a given time (i.e. one user
per process / event loop)
* decide explicitly if you want other coros to continue (I assume so,
as blocking entire process would be trivial)
* don't care about performance too much :)

Once that's done, you can allow multiple users per event loop by
wrapping your inter-process lock in a regular async lock.

Wrt. performance, you can start with a simple client-server
implementation, for example where:
* single-threaded server listens on some port, accepts 1 connection at
a time, writes something on the connection and waits for connection to
be closed
* each client connects (not informative due to listen backlog) and
waits for data, when client gets the data, it has the lock
* when client wants to release the lock, it closes the connection,
which unblocks the server
* socket communication is relatively easy to marry to the event loop :)

If you want high performance (i.e. low latency), you'd probably want
to go with futex, but that may prove hard to marry to asyncio
internals.
I guess locking can always be proxied through a thread, at some cost
to performance.


If performance is important, I'd suggest starting with a thread proxy
from the start. It could go like this:
Each named lock gets own thread (in each process / event loop), a sync
lock and condition variable.
When a coro want to take the lock, it creates an empty Future,
ephemerally takes the sync lock, adds this future to waiters, and
signals on the condition variable and awaits this Future.
Thread wakes up, validates there's someone in the queue under sync
lock, tries to take classical inter-process lock (sysv or file or
whatever), and when that succeeds, resolves the future using
loop.call_soon_threadsafe().
I'm omitting implementation details, like what if Future is leaked
(discarded before it's resolved), how release is orchestrated, etc.
The key point is that offloading locking to a dedicated thread allows
to reduce original problem to synchronous interprocess locking
problem.


Cheers!


On 17 April 2018 at 06:05, Ludovic Gasc <gmludo at gmail.com> wrote:
> Hi,
>
> I'm looking for a equivalent of asyncio.Lock
> (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
> shared between several processes on the same server, because I'm migrating a
> daemon from mono-worker to multi-worker pattern.
>
> For now, the closest solution in term of API seems aioredlock:
> https://github.com/joanvila/aioredlock#aioredlock
> But I'm not a big fan to use polling nor with a timeout because the lock I
> need is very critical, I prefer to block the code than unlock with timeout.
>
> Do I miss a new awesome library or do you have an easier approach ?
>
> Thanks for your responses.
> --
> Ludovic Gasc (GMLudo)
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>