Mailman 3 April 2018 - Async-sig

Re: [Async-sig] [python-tulip] Re: asyncio.Lock equivalent for multiple processes
by Ludovic Gasc 20 May '18

20 May '18

Hi Antoine & Chris, Thanks a lot for the advisory lock, I didn't know this feature in PostgreSQL. Indeed, it seems to fit my problem. The small latest problem I have is that we have string names for locks, but advisory locks accept only integers. Nevertheless, it isn't a problem, I will do a mapping between names and integers. Yours. -- Ludovic Gasc (GMLudo) 2018-04-17 13:41 GMT+02:00 Antoine Pitrou <solipsis(a)pitrou.net>: > On Tue, 17 Apr 2018 13:34:47 +0200 > Ludovic Gasc <gmludo(a)gmail.com> wrote: > > Hi Nickolai, > > > > Thanks for your suggestions, especially for the file system lock: We > don't > > have often locks, but we must be sure it's locked. > > > > For 1) and 4) suggestions, in fact we have several systems to sync and > also > > a PostgreSQL transaction, the request must be treated by the same worker > > from beginning to end and the other systems aren't idempotent at all, > it's > > "old-school" proprietary systems, good luck to change that ;-) > > If you already have a PostgreSQL connection, can't you use a PostgreSQL > lock? e.g. an "advisory lock" as described in > https://www.postgresql.org/docs/9.1/static/explicit-locking.html > > Regards > > Antoine. > > >

3 5

New blog post: Notes on structured concurrency, or: Go statement considered harmful
by Nathaniel Smith 27 Apr '18

27 Apr '18

Hi all, I just posted another essay on concurrent API design: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-con… This is the one that finally gets at the core reasons why Trio exists; I've been trying to figure out how to write it for at least a year now. I hope you like it. (Guido: this is the one you should read :-). Or if it's too much, you can jump to the conclusion [1], and I'm happy to come find you somewhere with a whiteboard, if that'd be helpful!) -n [1] https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-con… -- Nathaniel J. Smith -- https://vorpus.org

5 8

Cancelling a coroutine from a signal handler?
by Mark E. Haase 25 Apr '18

25 Apr '18

I am trying to understand some unexpected behavior in asyncio. My goal is to use a custom signal handler to cleanly unwind an asyncio program that has many different tasks running. Here's a simplified test case: 1 import asyncio, logging, random, signal, sys 2 3 logging.basicConfig(level=logging.DEBUG) 4 logger = logging.getLogger() 5 6 async def main(): 7 try: 8 await asyncio.Event().wait() 9 except asyncio.CancelledError: 10 logger.info('cancelled main()') 11 # cleanup logic 12 13 def handle_sigint(signal, frame): 14 global sigint_count 15 global main_coro 16 sigint_count += 1 17 if sigint_count == 1: 18 logger.warn('Received interrupt: shutting down...') 19 main_coro.throw(asyncio.CancelledError()) 20 # missing event loop logic? 21 else: 22 logger.warn('Received 2nd interrupt: exiting!') 23 main_coro.throw(SystemExit(1)) 24 25 sigint_count = 0 26 signal.signal(signal.SIGINT, handle_sigint) 27 loop = asyncio.get_event_loop() 28 main_coro = main() 29 try: 30 loop.run_until_complete(main_coro) 31 except StopIteration: 32 logger.info('run_until_complete() finished') The main() function is a placeholder that represents some long running task, e.g. a server that is waiting for new connections. The handle_sigint() function is supposed to attempt to cancel main() so that it can gracefully exit, but if it receives a second interrupt, then the process exits immediately. Here's an example running the program and then typing Ctrl+C. $ python test.py DEBUG:asyncio:Using selector: EpollSelector ^CWARNING:root:Received interrupt: shutting down... INFO:root:cancelled main() INFO:root:run_until_complete() finished This works as I expect it to. Of course my cleanup logic (line 10) isn't actually doing anything. In a real server, I might want to send goodbye messages to connected clients. To mock this behavior, I'll modify line 11: 11 await asyncio.sleep(0) Surprisingly, now my cleanup code hangs: $ python test.py DEBUG:asyncio:Using selector: EpollSelector ^CWARNING:root:Received interrupt: shutting down... INFO:root:cancelled main() ^CWARNING:root:Received 2nd interrupt: exiting! Notice that the program doesn't exit after the first interrupt. It enters the exception handler and appears to hang on the await expression on line 11. I have to interrupt it a second time, which throws SystemExit instead. I puzzled over this for quite some time until I realized that I can force main() to resume by changing line 20: 20 main_coro.send(None) With this change, the interrupt causes the cleanup logic to run to completion and the program exits normally. Of course, if I add a second await expression: 11 await asyncio.sleep(0); await asyncio.sleep(0) Then I also have to step twice: 20 main_coro.send(None); main_coro.send(None) My mental model of how the event loop works is pretty poor, but I roughly understand that the event loop is responsible for driving coroutines. It appears here that the event loop has stopped driving my main() coroutine, and so the only way to force it to complete is to call send() from my code. Can somebody explain *why* the event loop is not driving my coroutine? Is this a bug or am I missing something conceptually? More broadly, handling KeyboardInterrupt in async code seems very tricky, but I also cannot figure out how to make this interrupt approach work. Is one of these better than the other? What is the best practice here? Would it be terrible to add `while True: main_coro.send(None)` to my signal handler? Thanks, Mark

3 4

asyncio.Lock equivalent for multiple processes
by Ludovic Gasc 17 Apr '18

17 Apr '18

Hi, I'm looking for a equivalent of asyncio.Lock ( https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but shared between several processes on the same server, because I'm migrating a daemon from mono-worker to multi-worker pattern. For now, the closest solution in term of API seems aioredlock: https://github.com/joanvila/aioredlock#aioredlock But I'm not a big fan to use polling nor with a timeout because the lock I need is very critical, I prefer to block the code than unlock with timeout. Do I miss a new awesome library or do you have an easier approach ? Thanks for your responses. -- Ludovic Gasc (GMLudo)

5 7

Simplifying stack traces for tasks?
by Mark E. Haase 13 Apr '18

13 Apr '18

If an exception is thrown while the `asyncio` event loop is running, the stack trace is pretty complicated. Here's an example: Traceback (most recent call last): File "sample_client.py", line 616, in <module> loop.run_until_complete(main_task) File "/usr/lib/python3.6/asyncio/base_events.py", line 437, in run_until_complete self.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 405, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1382, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 126, in _run self._callback(*self._args) File "/usr/lib/python3.6/asyncio/tasks.py", line 315, in _wakeup self._step() File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "sample_client.py", line 319, in main await actions[args.action](args, socket) File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in __next__ return self.gen.send(None) File "sample_client.py", line 558, in sync_job await asyncio.sleep(args.delay) File "/usr/lib/python3.6/asyncio/coroutines.py", line 105, in __next__ return self.gen.send(None) File "/usr/lib/python3.6/asyncio/tasks.py", line 522, in sleep future, result) File "/usr/lib/python3.6/asyncio/base_events.py", line 525, in call_later timer = self.call_at(self.time() + delay, callback, *args) File "/usr/lib/python3.6/asyncio/base_events.py", line 541, in call_at timer = events.TimerHandle(when, callback, args, self) File "/usr/lib/python3.6/asyncio/events.py", line 148, in __init__ super().__init__(callback, args, loop) File "/usr/lib/python3.6/asyncio/events.py", line 92, in __init__ self._source_traceback = traceback.extract_stack(sys._getframe(1)) File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack stack = StackSummary.extract(walk_stack(f), limit=limit) File "/usr/lib/python3.6/traceback.py", line 352, in extract filename, lineno, name, lookup_line=False, locals=f_locals)) KeyboardInterrupt Most of the stack frames are inside asyncio (_run, _wakeup, _step) and reveal the mechanics of the event loop. The stack trace would be a lot easier to read (and more similar to a stack trace of an equivalent synchronous program) if it looked like this: Traceback (most recent call last): File "sample_client.py", line 616, in <module> loop.run_until_complete(main_task) ... File "sample_client.py", line 319, in main await actions[args.action](args, socket) ... File "sample_client.py", line 558, in sync_job await asyncio.sleep(args.delay) KeyboardInterrupt I recognize that the event loop mechanics are probably useful in some circumstances, but in my experience working with asyncio for the last year, I've found the stack traces are generally too noisy. For a while, I was pasting stack traces into a text editor and cleaning them up by hand, but a few months ago I wrote a custom excepthook that generates stack traces similar to the one above. def async_excepthook(type_, exc, tb): cause_exc = None cause_str = None if exc.__cause__ is not None: cause_exc = exc.__cause__ cause_str = 'The above exception was the direct cause ' \ 'of the following exception:' elif exc.__context__ is not None and not exc.__suppress_context__: cause_exc = exc.__context__ cause_str = 'During handling of the above exception, ' \ 'another exception occurred:' if cause_exc: async_excepthook(type(cause_exc), cause_exc, cause_exc.__traceback__) if cause_str: print('\n{}\n'.format(cause_str)) print('Async Traceback (most recent call last):') for frame in traceback.extract_tb(tb): head, tail = os.path.split(frame.filename) if (head.endswith('asyncio') or tail == 'traceback.py') and \ frame.name.startswith('_'): print(' ...') continue print(' File "{}", line {}, in {}' .format(frame.filename, frame.lineno, frame.name)) print(' {}'.format(frame.line)) print('{}: {}'.format(type_.__name__, exc)) The meat of it is towards the bottom, "if head.endswith('asyncio')..."There are a lot of debatable details and this implementation is pretty hacky and clumsy, but I have found it valuable in my own usage, and I haven't yet missed the omitted stack frames. I'm posting here to get constructive criticism on the concept and would also like to hear if Curio or Trio have done anything like this. (Based on a quick skim of the docs, I don't think they are.) I might publish a PyPI package if anybody is interested. Cheers, Mark

4 7