[Async-sig] Inadvertent layering of synchronous code as frameworks adopt asyncio

[Daniel Nugent]

Hello, I was hoping that the Async SIG might have some suggestions on how to deal with this sort of issue:

More frameworks are adopting asyncio as time marches on. A notable example of this is Jupyter and the Python kernels it supports (please see announcement here: blog.jupyter.org/ipython-7-0-async-repl-a35ce050f7f7). This was enabled by a change in Tornado version 5.0 to support the asyncio event loop.

The problem is that this makes any code which inadvertently ran an asyncio event loop (that is, calls through a blocking API provided by a library implemented in asyncio) fail. The Jupyter developers seem to feel that this is a deficiency in the asyncio event loop model and suggest all users encountering such a problem adopt the patch module nest_asyncio (github.com/jupyter/notebook/issues/3397#issuecomment-419474214).

However, it is my understanding that the Python team strongly feels this is not the correct path: bugs.python.org/issue33523bugs.python.org/issue29558bugs.python.org/issue22239

I have been trying to figure out the right way to work around this issue such that a library implemented with asyncio that provides a synchronous API will not cause a problem and have come up short thus far. I was considering investigating the janus sync/async queue as a way of facilitating communication between the different modes, but I am not sure that the scenario I describe reflects the intended usage. That is, an outer asyncio driven program fragment calls into middle synchronous code, which calls to inner asynchronous code. It seems that janus is mostly intended to facilitate communication between a single outer asynchronous layer and an inner synchronous layer. However, the documentation is a little sparse so I may just not understand it yet.

I don't believe I'm the only person struggling to figure out how to deal with this sort of situation, so I think this would be useful for the community to figure out a solid answer to. For example, I found this blog post which outlines the same sort of problem and suggests that they elected to use nest_asyncio threespeedlogic.com/python-tworoutines.html

If anyone could provide guidance on how to go forward, I would appreciate it.

I would also like to understand the decision making around not allowing event loop nesting/reentrancy as seen in the bugs.python.orgissues I referenced so that I may explain the tradeoffs of possibly adopting the nest_asyncio patch module (for the sake of argument, lets ignore the possible issues with non-standard event loops) better to my peers.

Thank you,

-Dan Nugent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20190322/c08b4675/attachment.html>