If you're already using PostgreSQL, you might also look at "advisory locks": https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISOR...
On Tue, Apr 17, 2018 at 4:34 AM, Ludovic Gasc email@example.com wrote:
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
Ludovic Gasc (GMLudo)
2018-04-17 12:46 GMT+02:00 Nickolai Novik firstname.lastname@example.org:
Hi, redis lock has own limitations and depending on your use case it may or may not be suitable . If possible I would redefine problem and also considered:
- create worker per specific resource type to avoid locking
- optimistic locking
- File system lock like in twisted, but not sure about performance and
edge cases there 4) make operation on resource idempotent
On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc email@example.com wrote:
Thanks for the pointer, it's exactly the type of feedbacks I'm looking for: Ideas that are out-of-box of my confort zone. However, in our use case, we are using gunicorn, that uses forks instead of multiprocessing to my knowledge, I can't use multiprocessing without to remove gunicorn.
If somebody is using aioredlock in his project, I'm interested by feedbacks.
Have a nice week.
-- Ludovic Gasc (GMLudo)
2018-04-17 7:19 GMT+02:00 Roberto Martínez firstname.lastname@example.org:
I don't know if there is a third party solution for this.
I think the closest you can get today using the standard library is using a multiprocessing.manager().Lock (which can be shared among processes) and call the lock.acquire() function with asyncio.run_in_executor(), using a ThreadedPoolExecutor to avoid blocking the asyncio event loop.
Best regards, Roberto
El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc (email@example.com) escribió:
I'm looking for a equivalent of asyncio.Lock (https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but shared between several processes on the same server, because I'm migrating a daemon from mono-worker to multi-worker pattern.
For now, the closest solution in term of API seems aioredlock: https://github.com/joanvila/aioredlock#aioredlock But I'm not a big fan to use polling nor with a timeout because the lock I need is very critical, I prefer to block the code than unlock with timeout.
Do I miss a new awesome library or do you have an easier approach ?
Thanks for your responses.
Ludovic Gasc (GMLudo)
Async-sig mailing list Asyncfirstname.lastname@example.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/