[Python-Dev] async/await behavior on multiple calls

[Roy Williams]

Thanks for the insight Guido.

I've mostly used async/await inside of HHVM/Hack, and used Guava/Java
Futures extensively in the past so I found this behavior to be quite
surprising.  I'd like to use Awaitables to represent a DAG of work that
needs to get done.  For example, I used to be one of the maintainers of
Buck (a build tool similar to Bazel) and we used a collection of futures
for building all of our dependencies.  For each rule, we'd effectively:

dependency_results = await asyncio.gather(*dependencies)
# Proceed with building.

Rules were free to depend on the same dependency and since the Future would
just return the same result when resolved more than one time things just
worked.

Similarly when building up the results for say a web request, I effectively
want to construct a DAG of work that needs to get done and then just await
on that DAG in a similar manner without having to enforce that the DAG is
actually a tree.  I can of course write a function to wrap everything in
Futures, but this seems to be against the spirit of async/await.

Thanks,
Roy

On Tue, Dec 15, 2015 at 12:08 PM, Guido van Rossum <guido at python.org> wrote:

> I think this goes back all the way to a debate we had when we were
> discussing PEP 380 (which introduced 'yield from', on which 'await' is
> built). In fact I believe that the reason PEP 380 didn't make it into
> Python 2.7 was that this issue was unresolved at the time (the PEP author
> and I preferred the current approach, but there was one vocal opponent who
> disagreed -- although my memory is only about 60% reliable on this :-).
>
> In any case, problem is that in order to implement the behavior you're
> asking for, the generator object would have to somehow hold on to its
> return value so that each time __next__ is called after it has already
> terminated it can raise StopIteration with the saved return value. This
> would extend the lifetime of the returned object indefinitely (until the
> generator object itself is GC'ed) in order to handle a pretty obscure
> corner case.
>
> I don't know how long you have been using async/await, but I wonder if
> it's possible that you just haven't gotten used to the typical usage
> patterns? In particular, your claim "anything that takes an `awaitable` has
> to know that it wasn't already awaited" makes me sound that you're just
> using it in an atypical way (perhaps because your model is based on other
> languages). In typical asyncio code, one does not usually take an
> awaitable, wait for it, and then return it -- one either awaits it and then
> extracts the result, or one returns it without awaiting it.
>
> On Tue, Dec 15, 2015 at 11:56 AM, Roy Williams <rwilliams at lyft.com> wrote:
>
>> Howdy,
>>
>> I'm experimenting with async/await in Python 3, and one very surprising
>> behavior has been what happens when calling `await` twice on an Awaitable.
>> In C#, Hack/HHVM, and the new async/await spec in Ecmascript 7.  In Python,
>> calling `await` multiple times results in all future results getting back
>> `None`.  Here's a small example program:
>>
>>
>> async def echo_hi():
>>     result = ''
>>     echo_proc = await asyncio.create_subprocess_exec(
>>             'echo', 'hello', 'world',
>>             stdout=asyncio.subprocess.PIPE,
>>             stderr=asyncio.subprocess.DEVNULL)
>>     result = await echo_proc.stdout.read()
>>     await echo_proc.wait()
>>     return result
>>
>> async def await_twice(awaitable):
>>     print('first time is {}'.format(await awaitable))
>>     print('second time is {}'.format(await awaitable))
>>
>> loop = asyncio.get_event_loop()
>> loop.run_until_complete(await_twice(echo_hi()))
>>
>> This makes writing composable APIs using async/await in Python very
>> difficult since anything that takes an `awaitable` has to know that it
>> wasn't already awaited.  Also, since the behavior is radically different
>> than in the other programming languages implementing async/await it makes
>> adopting Python's flavor of async/await difficult for folks coming from a
>> language where it's already implemented.
>>
>> In C#/Hack/JS calls to `await` return a Task/AwaitableHandle/Promise that
>> can be awaited multiple times and either returns the result or throws any
>> thrown exceptions.  It doesn't appear that the Awaitable class in Python
>> has a `result` or `exception` field but `asyncio.Future` does.
>>
>> Would it make sense to shift from having `await` functions return a `
>> *Future-like`* return object to returning a Future?
>>
>> Thanks,
>> Roy
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151215/248725ca/attachment-0001.html>