Mailman 3 async/await and synchronous code (and PEP492 ?) - Python-ideas

5 May 2015 · function

Hi all!
I am excited about seeing what's going on with asyncio and PEP492 etc. I 
really like that Python is becoming more suitable for the increasing 
amount of async code and that the distinction between async functions 
and generators is increasing.
In addition, however, I would also like to see the async functions and 
methods come even closer to regular functions and methods. This is 
something that is keeping me from using asyncio at the moment even if I 
would like to. Below I'll try to explain what and why, and a little bit 
of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async functions is 
that you can write asynchronous code as if it were synchronous, the 
difference in many places being just the use of "await" ("yield from") 
when calling something that may end up doing IO (somewhere down the 
function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to make it 
either synchronous or asynchronous. Regardless of the choice, the code 
will look roughly the same. But what if you want to be able to do both? 
Should you maintain two versions, one with "async" and "await" 
everywhere and one without?
Besides the keywords "async" and "await", async code of course differs 
from synchronous code by the functions/coroutines that are used for IO 
at the end of the function call chain. Here, I mean the end (close to) 
where the "yield" expressions are hidden in the async versions. At the 
other end of the calling chain, async code needs the event loop and 
associated framework (almost always asyncio?) which hides all the async 
scheduling fanciness etc. I'm not sure about the terminology, but I will 
use "L end" and "Y end" to refer to the two ends here. (L for event 
Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the code to 
work. While asyncio and the standard library might provide both ends in 
many cases, there can also be situations where a package would want to 
work with different combinations of L and Y end, or completely without 
an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different 
implementations of sleep() in a function that would pick the right one 
depending on the context. Perhaps something like this:
async def any_sleep(seconds):
      if __async__.framework is None:
          time.sleep(1)
      elif __async__.framework is asyncio:
          await asyncio.sleep(1)
      else:
          raise RuntimeError("Was called with an unsupported async 
framework.")
[You could of course replace sleep() with socket IO or whatever, but 
sleep is nice and simple. Also, a larger library would probably have a 
whole chain of async functions and methods before calling something like 
this]
But if await is only allowed inside "async def", then how can 
any_sleep() be conveniently run in non-async code? Also, there is 
nothing like __async__.framework. Below, I describe what I think a 
potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I consider 
only async function objects here. I describe the idea in three parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to next(...) with 
a default value of None. When an async framework, typically asyncio, 
starts an async function object (coroutine) with a call to next(...), it 
would do something like next(coro, async_framework = asyncio). Here, 
asyncio could of course be replaced with any object that identifies the 
framework. This information would then be somehow attached to the async 
function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as .framework 
that allows the code inside the async function to access the information 
passed to next(...) by the framework (L end) using the keyword argument 
of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not just within 
async functions. Inside async functions, the behavior of "await" would 
be the same as in PEP492, with the addition that it would somehow 
propagate the __async__.framework value to the awaited coroutine. 
Outside async functions, "await" would do roughly the same as this function:
def await(async_func_obj):
      try:
          next(async_func_obj)   # same as next(async_func_obj, 
async_framework = None)
      except StopIteration as si:
          return si.value
      raise RuntimeError("The function does not support synchronous 
execution")
(This function would, of course, work in Python 3.4, but it would be 
mostly useless because the async functions would not know that they are 
being called in a 'synchronous program'. IIUC, this function would be 
valid even with PEP492, but having this as a function would be ugly in 
the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that work both 
async and non-async. When await is not inside async def, one would 
expect it to potentially do blocking IO, just like an await inside async 
def would suggest that there is a yield/suspend somewhere in there.
For testing, I tried to see if there is a reasonable way to make a hack 
with __async__.framework that could be set by next(), but did not find 
an obvious way. For instance, coro.gi_frame.f_locals is read-only, I 
believe.
An alternative to this approach could be that await would implicitly 
start a temporary event loop for running the coroutine, but how would it 
know which event loop? This might also have a huge performance overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post it here 
now in case there is desire to prepare for something like this already 
in PEP492. It is not completely clear if/how this would need to affect 
PEP492, but some things come to mind. For example, this could 
potentially remove the need for __aenter__, __aiter__, etc. or even 
"async for" and "async with". If __aenter__ is defined as "async def", 
then a with statement would do an "await" on it, and the context manager 
would have __async__.framework (or whatever it would be called) 
available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this would be 
solving and how? I'd be happy to hear about any thoughts on this :).
Best regards,
Koos

async/await and synchronous code (and PEP492 ?)

implicit ;-)