Re: [Python-ideas] async/await and synchronous code (and PEP492 ?)

Quick notes:

• I don't think it's really possible to write realistic async code independently from an async framework.

• For synchronous code that wants to use some async code, the pattern is simple: asyncio.get_event_loop().run_until_complete(some_async_call(args, etc))
• We can probably wrap this in a convenience helper function so you can just write: asyncio.sync_wait(some_async_call(args, etc))

• Note that this will fail (and rightly so!) if called when the event loop is already running.

On Tue, May 5, 2015 at 11:00 AM, Andrew Barnert via Python-ideas

<python-ideas@python.org mailto:python-ideas@python.org> wrote:

It seems like it might be a lot easier to approach this from the
other end: Is it possible to write a decorator that takes an async
coroutine function, strips out all the awaits, and returns a
regular sync function? If so, all you need to do is write
everything as async, and then users can "from spam import sync as
spam" or "from spam import async as spam" (where async just
imports all the real functions, while sync imports them and calls
the decorator on all of them).

That also avoids the need to have all the
looking up the event
loop, switching between different code branches, etc. inside every
function at runtime. (Not that it matters for the performance of
sleep(1), but it might matter for the performance of other
functions—and, more importantly, it might make the implementation
of those functions simpler and easier to debug through.)

On Tuesday, May 5, 2015 7:01 AM, Koos
Zevenhoven
<koos.zevenhoven@aalto.fi <mailto:koos.zevenhoven@aalto.fi>> wrote:



    Hi all!

    I am excited about seeing what's going on with asyncio and
    PEP492 etc. I
    really like that Python is becoming more suitable for the
    increasing
    amount of async code and that the distinction between async
    functions
    and generators is increasing.

    In addition, however, I would also like to see the async
    functions and
    methods come even closer to regular functions and methods.
    This is
    something that is keeping me from using asyncio at the moment
    even if I
    would like to. Below I'll try to explain what and why, and a
    little bit
    of how. If it is not clear, please ask :)

    Motivation:

    One of the best things about asyncio and coroutines/async
    functions is
    that you can write asynchronous code as if it were
    synchronous, the
    difference in many places being just the use of "await"
    ("yield from")
    when calling something that may end up doing IO (somewhere
    down the
    function call chain) and that the code is run from an event loop.

    When writing a package that does IO, you have the option to
    make it
    either synchronous or asynchronous. Regardless of the choice,
    the code
    will look roughly the same. But what if you want to be able to
    do both?
    Should you maintain two versions, one with "async" and "await"
    everywhere and one without?

    Besides the keywords "async" and "await", async code of course
    differs
    from synchronous code by the functions/coroutines that are
    used for IO
    at the end of the function call chain. Here, I mean the end
    (close to)
    where the "yield" expressions are hidden in the async
    versions. At the
    other end of the calling chain, async code needs the event
    loop and
    associated framework (almost always asyncio?) which hides all
    the async
    scheduling fanciness etc. I'm not sure about the terminology,
    but I will
    use "L end" and "Y end" to refer to the two ends here. (L for
    event
    Loop; Y for Yield)

    The Y and L ends need to be compatible with each other for the
    code to
    work. While asyncio and the standard library might provide
    both ends in
    many cases, there can also be situations where a package would
    want to
    work with different combinations of L and Y end, or completely
    without
    an event loop, i.e. synchronously.

    In a very simple example, one might want to wrap different
    implementations of sleep() in a function that would pick the
    right one
    depending on the context. Perhaps something like this:

      async def any_sleep(seconds):
          if __async__.framework is None:
              time.sleep(1)
          elif __async__.framework is asyncio:
              await asyncio.sleep(1)
          else:
              raise RuntimeError("Was called with an unsupported
    async
    framework.")

    [You could of course replace sleep() with socket IO or
    whatever, but
    sleep is nice and simple. Also, a larger library would
    probably have a
    whole chain of async functions and methods before calling
    something like
    this]

    But if await is only allowed inside "async def", then how can
    any_sleep() be conveniently run in non-async code? Also, there is
    nothing like __async__.framework. Below, I describe what I
    think a
    potential solution might look like.



    Potential solution:

    This is simplified version; for instance, as "awaitables", I
    consider
    only async function objects here. I describe the idea in three
    parts:

    (1) next(...):

    Add a keyword argument "async_framework" (or whatever) to
    next(...) with
    a default value of None. When an async framework, typically
    asyncio,
    starts an async function object (coroutine) with a call to
    next(...), it
    would do something like next(coro, async_framework = asyncio).
    Here,
    asyncio could of course be replaced with any object that
    identifies the
    framework. This information would then be somehow attached to
    the async
    function object.


    (2) __async__.framework or something similar:

    Add something like __async__ that has an attribute such as
    .framework
    that allows the code inside the async function to access the
    information
    passed to next(...) by the framework (L end) using the keyword
    argument
    of next [see (1)].

    (3) Generalized "await":

    [When the world is ready:] Allow using "await" anywhere, not
    just within
    async functions. Inside async functions, the behavior of
    "await" would
    be the same as in PEP492, with the addition that it would somehow
    propagate the __async__.framework value to the awaited coroutine.
    Outside async functions, "await" would do roughly the same as
    this function:

      def await(async_func_obj):
          try:
              next(async_func_obj)  # same as next(async_func_obj,
    async_framework = None)
          except StopIteration as si:
              return si.value
          raise RuntimeError("The function does not support
    synchronous
    execution")

    (This function would, of course, work in Python 3.4, but it
    would be
    mostly useless because the async functions would not know that
    they are
    being called in a 'synchronous program'. IIUC, this *function*
    would be
    valid even with PEP492, but having this as a function would be
    ugly in
    the long run.)


    Some random thoughts:

    With this addition to Python, one could write libraries that
    work both
    async and non-async. When await is not inside async def, one
    would
    expect it to potentially do blocking IO, just like an await
    inside async
    def would suggest that there is a yield/suspend somewhere in
    there.

    For testing, I tried to see if there is a reasonable way to
    make a hack
    with __async__.framework that could be set by next(), but did
    not find
    an obvious way. For instance, coro.gi_frame.f_locals is
    read-only, I
    believe.

    An alternative to this approach could be that await would
    implicitly
    start a temporary event loop for running the coroutine, but
    how would it
    know which event loop? This might also have a huge performance
    overhead.

    Relation to PEP492:

    This of course still needs more thinking, but I wanted to post
    it here
    now in case there is desire to prepare for something like this
    already
    in PEP492. It is not completely clear if/how this would need
    to affect
    PEP492, but some things come to mind. For example, this could
    potentially remove the need for __aenter__, __aiter__, etc. or
    even
    "async for" and "async with". If __aenter__ is defined as
    "async def",
    then a with statement would do an "await" on it, and the
    context manager
    would have __async__.framework (or whatever it would be called)
    available, for determining what behavior is appropriate.

    Was this clear enough to understand which problem(s) this
    would be
    solving and how? I'd be happy to hear about any thoughts on
    this :).


    Best regards,
    Koos

    _______________________________________________
    Python-ideas mailing list
    Python-ideas@python.org <mailto:Python-ideas@python.org>
    https://mail.python.org/mailman/listinfo/python-ideas
    Code of Conduct: http://python.org/psf/codeofconduct/



_______________________________________________
Python-ideas mailing list
Python-ideas@python.org <mailto:Python-ideas@python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)