awaiting task is not chaining exception
Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected. I was able to reduce it down to the code below. The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate. The way I would describe it is that (at least in this case) awaiting on a task that raises an exception doesn't set __context__ when awaiting on the task inside an "except" block. Here is the code: import asyncio async def raise_error(): raise RuntimeError('exception #2') async def main(): try: raise RuntimeError('exception #1') except Exception: future = raise_error() # Uncommenting the next line makes exc.__context__ None. # future = asyncio.ensure_future(future) try: await future except Exception as exc: print(f'exc: {exc!r}, context: {exc.__context__!r}') raise asyncio.get_event_loop().run_until_complete(main()) Here is the output of the above, which works as expected: exc: RuntimeError('exception #2',), context: RuntimeError('exception #1',) Traceback (most recent call last): File "test.py", line 8, in main raise RuntimeError('exception #1') RuntimeError: exception #1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 20, in <module> asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 15, in main await future File "test.py", line 4, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 And here is the unexpected output when the commented line is uncommented (exception not getting chained): exc: RuntimeError('exception #2',), context: None Traceback (most recent call last): File "test.py", line 22, in <module> asyncio.get_event_loop().run_until_complete(main()) File "/.../python3.6/asyncio/base_events.py", line 466, in run_until_complete return future.result() File "test.py", line 17, in main await future File "test.py", line 6, in raise_error raise RuntimeError('exception #2') RuntimeError: exception #2 --Chris
On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected.
I was able to reduce it down to the code below.
The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate.
I think what you're seeing is collateral damage from some known bugginess in the generator/coroutine .throw() method: https://bugs.python.org/issue29587 In trio I work around this by never using throw(); instead I send() in the exception and re-raise it inside the coroutine: https://github.com/python-trio/trio/blob/389f1e1e01b410756e2833cffb992fd1ff8... But asyncio doesn't do this -- when an asyncio.Task awaits an asyncio.Future and the Future raises, the exception is throw()n into the Task, triggering the bug: https://github.com/python/cpython/blob/e184cfd7bf8bcfd160e3b611d4351ca3ce52d... (If you try profiling your code you may also see weird/impossible results in cases like this, because throw() also messes up stack introspection.) -n -- Nathaniel J. Smith -- https://vorpus.org
On Sat, Nov 11, 2017 at 12:39 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected.
I was able to reduce it down to the code below.
The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate.
I think what you're seeing is collateral damage from some known bugginess in the generator/coroutine .throw() method: https://bugs.python.org/issue29587
Ah, thanks for the great explanation, Nathaniel!
From the original bug report Nathaniel filed above: It's likely that more people will run into this in the future as async/await becomes more widely used. ...
:) --Chris
By the way, since we're already on the subject of asyncio tasks and (truncated) stack traces, this looks like a good opportunity to ask a question that's been on my mind for a while: There's a mysterious note at the end of the documentation of asyncio.Task's get_stack() method, where it says--
For reasons beyond our control, only one stack frame is returned for a suspended coroutine. (https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.get_stack )
What does the "For reasons beyond our control" mean? What is it that can possibly be beyond the control of Python? --Chris On Sat, Nov 11, 2017 at 1:47 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sat, Nov 11, 2017 at 12:39 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Nov 10, 2017 at 9:52 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Hi, I recently encountered a situation with asyncio where the stack trace is getting truncated: an exception isn't getting chained as expected.
I was able to reduce it down to the code below.
The reduced case seems like a pattern that can come up a lot, and I wasn't able to find an issue on the CPython tracker, so I'm wondering if I'm doing something wrong or if the behavior is deliberate.
I think what you're seeing is collateral damage from some known bugginess in the generator/coroutine .throw() method: https://bugs.python.org/issue29587
Ah, thanks for the great explanation, Nathaniel!
From the original bug report Nathaniel filed above:
It's likely that more people will run into this in the future as async/await becomes more widely used. ...
:)
--Chris
On Sun, Nov 12, 2017 at 2:53 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
By the way, since we're already on the subject of asyncio tasks and (truncated) stack traces, this looks like a good opportunity to ask a question that's been on my mind for a while:
There's a mysterious note at the end of the documentation of asyncio.Task's get_stack() method, where it says--
For reasons beyond our control, only one stack frame is returned for a suspended coroutine. (https://docs.python.org/3/library/asyncio-task.html# asyncio.Task.get_stack )
What does the "For reasons beyond our control" mean? What is it that can possibly be beyond the control of Python?
It's an odd phrasing, but it refers to the fact that a suspended generator frame (which is what a coroutine really is) does not have a "back link" to the frame that called it. Whenever a generator yields (or a coroutine awaits) its frame is disconnected from the current call stack, control is passed to the top frame left on that stack, and the single generator frame is just held on to by the generator object. When you call next() on that, it will be pushed on top of whatever is the current stack (i.e. whatever calls next()), which *may* be a completely different stack configuration than when it was suspended. That's all. -- --Guido van Rossum (python.org/~guido)
On Sun, Nov 12, 2017 at 7:27 AM, Guido van Rossum <guido@python.org> wrote:
On Sun, Nov 12, 2017 at 2:53 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
By the way, since we're already on the subject of asyncio tasks and (truncated) stack traces, this looks like a good opportunity to ask a question that's been on my mind for a while:
There's a mysterious note at the end of the documentation of asyncio.Task's get_stack() method, where it says--
For reasons beyond our control, only one stack frame is returned for a suspended coroutine.
(https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.get_stack )
What does the "For reasons beyond our control" mean? What is it that can possibly be beyond the control of Python?
It's an odd phrasing, but it refers to the fact that a suspended generator frame (which is what a coroutine really is) does not have a "back link" to the frame that called it. Whenever a generator yields (or a coroutine awaits) its frame is disconnected from the current call stack, control is passed to the top frame left on that stack, and the single generator frame is just held on to by the generator object. When you call next() on that, it will be pushed on top of whatever is the current stack (i.e. whatever calls next()), which *may* be a completely different stack configuration than when it was suspended.
It is possible to get the await/yield from stack though, even when a generator/coroutine is suspended, using the gi_yieldfrom / cr_await attributes. Teaching Task.get_stack to do this would be a nice little enhancement. -n -- Nathaniel J. Smith -- https://vorpus.org
participants (3)
-
Chris Jerdonek -
Guido van Rossum -
Nathaniel Smith