Re: [Async-sig] [python-tulip] Re: asyncio.Lock equivalent for multiple processes
Hi Antoine & Chris, Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem. The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers. Yours. -- Ludovic Gasc (GMLudo) 2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
You could simply use something like the first 64 bits of sha1("myapp:<lock name>") Regards Antoine. On Tue, 17 Apr 2018 15:04:37 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Antoine & Chris,
Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem.
The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers.
Yours.
-- Ludovic Gasc (GMLudo)
2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
2018-04-17 15:16 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
You could simply use something like the first 64 bits of sha1("myapp:<lock name>")
I have followed your idea, except I used hashtext directly, it's an internal postgresql function that generates an integer directly. For now, it seems to work pretty well but I didn't yet finished all tests. The final result is literally 3 lines of Python inside an async contextmanager, I like this solution ;-) : @asynccontextmanager async def lock(env, category='global', name='global'): # Alternative lock id with 'mytable'::regclass::integer OID await env['aiopg']['cursor'].execute("SELECT pg_advisory_lock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)}) yield None await env['aiopg']['cursor'].execute("SELECT pg_advisory_unlock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
Regards
Antoine.
On Tue, 17 Apr 2018 15:04:37 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Antoine & Chris,
Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem.
The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers.
Yours.
-- Ludovic Gasc (GMLudo)
2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Pretty sure you want to add a try/finally around that yield, so you release the lock on errors. On Tue, Apr 17, 2018, 14:39 Ludovic Gasc <gmludo@gmail.com> wrote:
2018-04-17 15:16 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
You could simply use something like the first 64 bits of sha1("myapp:<lock name>")
I have followed your idea, except I used hashtext directly, it's an internal postgresql function that generates an integer directly.
For now, it seems to work pretty well but I didn't yet finished all tests. The final result is literally 3 lines of Python inside an async contextmanager, I like this solution ;-) :
@asynccontextmanager async def lock(env, category='global', name='global'): # Alternative lock id with 'mytable'::regclass::integer OID await env['aiopg']['cursor'].execute("SELECT pg_advisory_lock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
yield None
await env['aiopg']['cursor'].execute("SELECT pg_advisory_unlock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
Regards
Antoine.
On Tue, 17 Apr 2018 15:04:37 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Antoine & Chris,
Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem.
The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers.
Yours.
-- Ludovic Gasc (GMLudo)
2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Indeed, thanks for the suggestion :-) Le mer. 18 avr. 2018 à 01:21, Nathaniel Smith <njs@pobox.com> a écrit :
Pretty sure you want to add a try/finally around that yield, so you release the lock on errors.
On Tue, Apr 17, 2018, 14:39 Ludovic Gasc <gmludo@gmail.com> wrote:
2018-04-17 15:16 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
You could simply use something like the first 64 bits of sha1("myapp:<lock name>")
I have followed your idea, except I used hashtext directly, it's an internal postgresql function that generates an integer directly.
For now, it seems to work pretty well but I didn't yet finished all tests. The final result is literally 3 lines of Python inside an async contextmanager, I like this solution ;-) :
@asynccontextmanager async def lock(env, category='global', name='global'): # Alternative lock id with 'mytable'::regclass::integer OID await env['aiopg']['cursor'].execute("SELECT pg_advisory_lock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
yield None
await env['aiopg']['cursor'].execute("SELECT pg_advisory_unlock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
Regards
Antoine.
On Tue, 17 Apr 2018 15:04:37 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Antoine & Chris,
Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem.
The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers.
Yours.
-- Ludovic Gasc (GMLudo)
2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Nickolai,
Thanks for your suggestions, especially for the file system lock: We don't have often locks, but we must be sure it's locked.
For 1) and 4) suggestions, in fact we have several systems to sync and also a PostgreSQL transaction, the request must be treated by the same worker from beginning to end and the other systems aren't idempotent at all, it's "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
FYI, advisory locks of PostgreSQL are working pretty well on production since one month now. Thanks again for your help. -- Ludovic Gasc (GMLudo) 2018-04-18 7:09 GMT+02:00 Ludovic Gasc <gmludo@gmail.com>:
Indeed, thanks for the suggestion :-)
Le mer. 18 avr. 2018 à 01:21, Nathaniel Smith <njs@pobox.com> a écrit :
Pretty sure you want to add a try/finally around that yield, so you release the lock on errors.
On Tue, Apr 17, 2018, 14:39 Ludovic Gasc <gmludo@gmail.com> wrote:
2018-04-17 15:16 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
You could simply use something like the first 64 bits of sha1("myapp:<lock name>")
I have followed your idea, except I used hashtext directly, it's an internal postgresql function that generates an integer directly.
For now, it seems to work pretty well but I didn't yet finished all tests. The final result is literally 3 lines of Python inside an async contextmanager, I like this solution ;-) :
@asynccontextmanager async def lock(env, category='global', name='global'): # Alternative lock id with 'mytable'::regclass::integer OID await env['aiopg']['cursor'].execute("SELECT pg_advisory_lock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
yield None
await env['aiopg']['cursor'].execute("SELECT pg_advisory_unlock( hashtext(%(lock_name)s) );", {'lock_name': '%s.%s' % (category, name)})
Regards
Antoine.
On Tue, 17 Apr 2018 15:04:37 +0200 Ludovic Gasc <gmludo@gmail.com> wrote:
Hi Antoine & Chris,
Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem.
The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers.
Yours.
-- Ludovic Gasc (GMLudo)
2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
On Tue, 17 Apr 2018 13:34:47 +0200 Ludovic Gasc <gmludo@gmail.com> wrote: > Hi Nickolai, > > Thanks for your suggestions, especially for the file system lock: We don't > have often locks, but we must be sure it's locked. > > For 1) and 4) suggestions, in fact we have several systems to sync and also > a PostgreSQL transaction, the request must be treated by the same worker > from beginning to end and the other systems aren't idempotent at all, it's > "old-school" proprietary systems, good luck to change that ;-)
If you already have a PostgreSQL connection, can't you use a PostgreSQL lock? e.g. an "advisory lock" as described in https://www.postgresql.org/docs/9.1/static/explicit-locking.html
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
participants (3)
-
Antoine Pitrou -
Ludovic Gasc -
Nathaniel Smith