Reliably make unhandled exceptions crash the event loop
Hi, I have been trying to make unhandled exceptions reliably crash the event loop (eg replicated behaviour of boost::asio, for those familiar with that C++ library). I'm aiming to have any exception bubble up from run_forever or run_until_complete style functions. I had thought I had a perfectly acceptable solution and then hit a strange case that threw my understanding of the way the loop worked. In 'test_b' below, the only difference is we keep a reference to the created task. This test hangs - the exception is raised, but the custom exception handler is never called. I'd be very interested to understand exactly why this happens. I'd also appreciate any feedback on the best way to reliably crash the event loop on unhandled exceptions (my next attempt will be to replace AbstractEventLoop.call_exception and see what happens). # example.py import asyncio class CustomException(RuntimeError): pass class UnhandledExceptionError(RuntimeError): pass def run_until_unhandled_exception(*, loop=None): """Run the event until there is an unhandled error in a callback This function sets the exception handler on the loop """ loop = loop if loop is not None else asyncio.get_event_loop() ex = [] def handler(loop, context): print('handler') loop.default_exception_handler(context) loop.stop() ex.append(context.get('exception')) loop.set_exception_handler(handler) loop.run_forever() if len(ex) > 0: raise UnhandledExceptionError('Unhandled exception in loop') from ex[0] async def fail_after(delay): await asyncio.sleep(delay) print('raise CustomException(...)') raise CustomException(f'fail_after(delay={delay})') async def finish_after(delay): await asyncio.sleep(delay) return delay def test_a(event_loop): event_loop.create_task(fail_after(0.01)) run_until_unhandled_exception(loop=event_loop) def test_b(event_loop): task = event_loop.create_task(fail_after(0.01)) run_until_unhandled_exception(loop=event_loop) def run_test(test): try: test(asyncio.get_event_loop()) except Exception as ex: print(ex) if __name__ == '__main__': run_test(test_a) run_test(test_b) # This hangs
I've realised the error of my ways: because Task separates the scheduling from the response handling, you cannot know if an exception is unhandled until the task is deleted. So in my example the reference means the task is not deleted, so the exception is not yet unhandled. This is in contrast to APIs like call_soon(callable, success_callback, error_callback) where there the possibility of delayed error handling is not present. In that case the loop can reliably crash if either callback raises an exception. So, the 'solution' to this use-case is to always attach error handers to Tasks. A catch-all solution cannot catch every error case. On Tue., 26 Feb. 2019, 6:06 am Josh Quigley, <0zeroth@gmail.com> wrote:
Hi,
I have been trying to make unhandled exceptions reliably crash the event loop (eg replicated behaviour of boost::asio, for those familiar with that C++ library). I'm aiming to have any exception bubble up from run_forever or run_until_complete style functions. I had thought I had a perfectly acceptable solution and then hit a strange case that threw my understanding of the way the loop worked.
In 'test_b' below, the only difference is we keep a reference to the created task. This test hangs - the exception is raised, but the custom exception handler is never called.
I'd be very interested to understand exactly why this happens. I'd also appreciate any feedback on the best way to reliably crash the event loop on unhandled exceptions (my next attempt will be to replace AbstractEventLoop.call_exception and see what happens).
# example.py import asyncio
class CustomException(RuntimeError): pass
class UnhandledExceptionError(RuntimeError): pass
def run_until_unhandled_exception(*, loop=None): """Run the event until there is an unhandled error in a callback
This function sets the exception handler on the loop """ loop = loop if loop is not None else asyncio.get_event_loop() ex = []
def handler(loop, context): print('handler') loop.default_exception_handler(context) loop.stop() ex.append(context.get('exception'))
loop.set_exception_handler(handler) loop.run_forever() if len(ex) > 0: raise UnhandledExceptionError('Unhandled exception in loop') from ex[0]
async def fail_after(delay): await asyncio.sleep(delay) print('raise CustomException(...)') raise CustomException(f'fail_after(delay={delay})')
async def finish_after(delay): await asyncio.sleep(delay) return delay
def test_a(event_loop): event_loop.create_task(fail_after(0.01)) run_until_unhandled_exception(loop=event_loop)
def test_b(event_loop): task = event_loop.create_task(fail_after(0.01)) run_until_unhandled_exception(loop=event_loop)
def run_test(test): try: test(asyncio.get_event_loop()) except Exception as ex: print(ex)
if __name__ == '__main__': run_test(test_a) run_test(test_b) # This hangs
On Mon, Feb 25, 2019 at 4:15 PM Josh Quigley <0zeroth@gmail.com> wrote:
I've realised the error of my ways: because Task separates the scheduling from the response handling, you cannot know if an exception is unhandled until the task is deleted. So in my example the reference means the task is not deleted, so the exception is not yet unhandled.
This is in contrast to APIs like call_soon(callable, success_callback, error_callback) where there the possibility of delayed error handling is not present. In that case the loop can reliably crash if either callback raises an exception.
So, the 'solution' to this use-case is to always attach error handers to Tasks. A catch-all solution cannot catch every error case.
That's right. There are other ways to structure async code to avoid running into these cases, that are implemented in Trio, and there are discussions happening (slowly) about adding them into asyncio as well. See: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-cons... Also, I could swear I saw some library that tried to implement nurseries on asyncio, but I can't find it now... :-/ maybe someone else here knows? -n -- Nathaniel J. Smith -- https://vorpus.org
Maybe this one https://github.com/malinoff/aionursery ? On 26/02/19 02:14, Nathaniel Smith wrote:
On Mon, Feb 25, 2019 at 4:15 PM Josh Quigley <0zeroth@gmail.com> wrote:
I've realised the error of my ways: because Task separates the scheduling from the response handling, you cannot know if an exception is unhandled until the task is deleted. So in my example the reference means the task is not deleted, so the exception is not yet unhandled.
This is in contrast to APIs like call_soon(callable, success_callback, error_callback) where there the possibility of delayed error handling is not present. In that case the loop can reliably crash if either callback raises an exception.
So, the 'solution' to this use-case is to always attach error handers to Tasks. A catch-all solution cannot catch every error case. That's right. There are other ways to structure async code to avoid running into these cases, that are implemented in Trio, and there are discussions happening (slowly) about adding them into asyncio as well. See:
https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-cons...
Also, I could swear I saw some library that tried to implement nurseries on asyncio, but I can't find it now... :-/ maybe someone else here knows?
-n
That's a great way of thinking about async structure - and perhaps surprisingly (since switching from asyncio or trying fledgling implementations as part of my day job is a no-go ;) immediately useful. A large part of what I do is wrap existing libraries so they can be used with asyncio- the idea being that once wrapped correctly it becomes easy to throw together applications quickly and correctly. For example 'cassandra' or the python gRPC libraries. These both offer async-ish style APIs, smattered with some callback style stuff and a handful of functions that block but probably shouldn't. There are no standard examples of 'best practice' in how to do this - the asyncio docs focus on using existing asyncio components. As a result I end up with stuff that works until I need to worry about error handling, cancellation, restarting components and then I cry my heart out in the mailing lists and generally make a mess. My key takeaway after skimming your blogs is that implementations should 'respect causality'. Aim for await my_implementation() to not spawn any anonymous tasks that can't be controlled, to only complete when it really has finished everything it started under the hood, and to correctly respect cancellation. Limit APIs to coroutines only (ie limit yourself to a 'curio' style) to make things simpler to reason about. If you must spawn tasks, keep them in logical groups - eg within a single function (or nursery if you have such an implementation) and make sure they are all finished before the function ends. It seems to me like these are good guiding principles to knock together robust async/await APIs. At any rate, I'll keep them in mind and see if my next attempt end up with less subtle problems to worry about. Thanks - and I look forward to really getting to grips with the detail of asynchronous design! On Tue, 26 Feb 2019 at 12:14, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Feb 25, 2019 at 4:15 PM Josh Quigley <0zeroth@gmail.com> wrote:
I've realised the error of my ways: because Task separates the
scheduling from the response handling, you cannot know if an exception is unhandled until the task is deleted. So in my example the reference means the task is not deleted, so the exception is not yet unhandled.
This is in contrast to APIs like call_soon(callable, success_callback,
error_callback) where there the possibility of delayed error handling is not present. In that case the loop can reliably crash if either callback raises an exception.
So, the 'solution' to this use-case is to always attach error handers to
Tasks. A catch-all solution cannot catch every error case.
That's right. There are other ways to structure async code to avoid running into these cases, that are implemented in Trio, and there are discussions happening (slowly) about adding them into asyncio as well. See:
https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-cons...
Also, I could swear I saw some library that tried to implement nurseries on asyncio, but I can't find it now... :-/ maybe someone else here knows?
-n
-- Nathaniel J. Smith -- https://vorpus.org
participants (3)
-
Josh Quigley -
Nathaniel Smith -
Ovv