[Async-sig] asyncio.Lock equivalent for multiple processes

[Ludovic Gasc]

Hi Dima,

Thanks for your time and explanations :-)
However, I have the intuition that it will take me more time to implement
your idea compare to the builtin feature of PostgreSQL.

Nevertheless, I keep your idea in mind in case of I have problems with
PostgreSQL.

Have a nice day.

--
Ludovic Gasc (GMLudo)

2018-04-17 14:17 GMT+02:00 Dima Tisnek <dimaqq at gmail.com>:

> Hi Ludovic,
>
> I believe it's relatively straightforward to implement the core
> functionality, if you can at first reduce it to:
> * allow only one coro to wait on lock at a given time (i.e. one user
> per process / event loop)
> * decide explicitly if you want other coros to continue (I assume so,
> as blocking entire process would be trivial)
> * don't care about performance too much :)
>
> Once that's done, you can allow multiple users per event loop by
> wrapping your inter-process lock in a regular async lock.
>
> Wrt. performance, you can start with a simple client-server
> implementation, for example where:
> * single-threaded server listens on some port, accepts 1 connection at
> a time, writes something on the connection and waits for connection to
> be closed
> * each client connects (not informative due to listen backlog) and
> waits for data, when client gets the data, it has the lock
> * when client wants to release the lock, it closes the connection,
> which unblocks the server
> * socket communication is relatively easy to marry to the event loop :)
>
> If you want high performance (i.e. low latency), you'd probably want
> to go with futex, but that may prove hard to marry to asyncio
> internals.
> I guess locking can always be proxied through a thread, at some cost
> to performance.
>
>
> If performance is important, I'd suggest starting with a thread proxy
> from the start. It could go like this:
> Each named lock gets own thread (in each process / event loop), a sync
> lock and condition variable.
> When a coro want to take the lock, it creates an empty Future,
> ephemerally takes the sync lock, adds this future to waiters, and
> signals on the condition variable and awaits this Future.
> Thread wakes up, validates there's someone in the queue under sync
> lock, tries to take classical inter-process lock (sysv or file or
> whatever), and when that succeeds, resolves the future using
> loop.call_soon_threadsafe().
> I'm omitting implementation details, like what if Future is leaked
> (discarded before it's resolved), how release is orchestrated, etc.
> The key point is that offloading locking to a dedicated thread allows
> to reduce original problem to synchronous interprocess locking
> problem.
>
>
> Cheers!
>
>
> On 17 April 2018 at 06:05, Ludovic Gasc <gmludo at gmail.com> wrote:
> > Hi,
> >
> > I'm looking for a equivalent of asyncio.Lock
> > (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
> > shared between several processes on the same server, because I'm
> migrating a
> > daemon from mono-worker to multi-worker pattern.
> >
> > For now, the closest solution in term of API seems aioredlock:
> > https://github.com/joanvila/aioredlock#aioredlock
> > But I'm not a big fan to use polling nor with a timeout because the lock
> I
> > need is very critical, I prefer to block the code than unlock with
> timeout.
> >
> > Do I miss a new awesome library or do you have an easier approach ?
> >
> > Thanks for your responses.
> > --
> > Ludovic Gasc (GMLudo)
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180417/d33ba42c/attachment-0001.html>