If you're already using PostgreSQL, you might also look at "advisory locks": https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISOR...
--Chris
On Tue, Apr 17, 2018 at 4:34 AM, Ludovic Gasc gmludo@gmail.com wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
Regards.
Ludovic Gasc (GMLudo)
2018-04-17 12:46 GMT+02:00 Nickolai Novik nickolainovik@gmail.com:
Hi, redis lock has own limitations and depending on your use case it may or may not be suitable [1]. If possible I would redefine problem and also considered:
- create worker per specific resource type to avoid locking
- optimistic locking
- File system lock like in twisted, but not sure about performance and
edge cases there 4) make operation on resource idempotent
[1] http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html [2] https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3d13479...
On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc gmludo@gmail.com wrote:
Hi Roberto,
Thanks for the pointer, it's exactly the type of feedbacks I'm looking for: Ideas that are out-of-box of my confort zone. However, in our use case, we are using gunicorn, that uses forks instead of multiprocessing to my knowledge, I can't use multiprocessing without to remove gunicorn.
If somebody is using aioredlock in his project, I'm interested by feedbacks.
Have a nice week.
-- Ludovic Gasc (GMLudo)
2018-04-17 7:19 GMT+02:00 Roberto MartÃnez robertomartinezp@gmail.com:
Hi,
I don't know if there is a third party solution for this.
I think the closest you can get today using the standard library is using a multiprocessing.manager().Lock (which can be shared among processes) and call the lock.acquire() function with asyncio.run_in_executor(), using a ThreadedPoolExecutor to avoid blocking the asyncio event loop.
Best regards, Roberto
El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc (gmludo@gmail.com) escribió:
Hi,
I'm looking for a equivalent of asyncio.Lock (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but shared between several processes on the same server, because I'm migrating a daemon from mono-worker to multi-worker pattern.
For now, the closest solution in term of API seems aioredlock: https://github.com/joanvila/aioredlock#aioredlock But I'm not a big fan to use polling nor with a timeout because the lock I need is very critical, I prefer to block the code than unlock with timeout.
Do I miss a new awesome library or do you have an easier approach ?
Thanks for your responses.
Ludovic Gasc (GMLudo)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/