[Python-Dev] async/await behavior on multiple calls

Tue Dec 15 19:57:33 EST 2015

On Tue, Dec 15, 2015 at 4:39 PM, Roy Williams <rwilliams at lyft.com> wrote:

> Thanks for the insight Guido.
>
> I've mostly used async/await inside of HHVM/Hack, and used Guava/Java
> Futures extensively in the past so I found this behavior to be quite
> surprising.  I'd like to use Awaitables to represent a DAG of work that
> needs to get done.  For example, I used to be one of the maintainers of
> Buck (a build tool similar to Bazel) and we used a collection of futures
> for building all of our dependencies.  For each rule, we'd effectively:
>
> dependency_results = await asyncio.gather(*dependencies)
> # Proceed with building.
>
> Rules were free to depend on the same dependency and since the Future
> would just return the same result when resolved more than one time things
> just worked.
>
> Similarly when building up the results for say a web request, I
> effectively want to construct a DAG of work that needs to get done and then
> just await on that DAG in a similar manner without having to enforce that
> the DAG is actually a tree.  I can of course write a function to wrap
> everything in Futures, but this seems to be against the spirit of
> async/await.
>

Why would that be against the spirit? It's the only thing that will work
the way you're asking, and there is in fact already a function that does
this (asyncio.ensure_future()).

> Thanks,
> Roy
>
> On Tue, Dec 15, 2015 at 12:08 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> I think this goes back all the way to a debate we had when we were
>> discussing PEP 380 (which introduced 'yield from', on which 'await' is
>> built). In fact I believe that the reason PEP 380 didn't make it into
>> Python 2.7 was that this issue was unresolved at the time (the PEP author
>> and I preferred the current approach, but there was one vocal opponent who
>> disagreed -- although my memory is only about 60% reliable on this :-).
>>
>> In any case, problem is that in order to implement the behavior you're
>> asking for, the generator object would have to somehow hold on to its
>> return value so that each time __next__ is called after it has already
>> terminated it can raise StopIteration with the saved return value. This
>> would extend the lifetime of the returned object indefinitely (until the
>> generator object itself is GC'ed) in order to handle a pretty obscure
>> corner case.
>>
>> I don't know how long you have been using async/await, but I wonder if
>> it's possible that you just haven't gotten used to the typical usage
>> patterns? In particular, your claim "anything that takes an `awaitable` has
>> to know that it wasn't already awaited" makes me sound that you're just
>> using it in an atypical way (perhaps because your model is based on other
>> languages). In typical asyncio code, one does not usually take an
>> awaitable, wait for it, and then return it -- one either awaits it and then
>> extracts the result, or one returns it without awaiting it.
>>
>> On Tue, Dec 15, 2015 at 11:56 AM, Roy Williams <rwilliams at lyft.com>
>> wrote:
>>
>>> Howdy,
>>>
>>> I'm experimenting with async/await in Python 3, and one very surprising
>>> behavior has been what happens when calling `await` twice on an Awaitable.
>>> In C#, Hack/HHVM, and the new async/await spec in Ecmascript 7.  In Python,
>>> calling `await` multiple times results in all future results getting back
>>> `None`.  Here's a small example program:
>>>
>>>
>>> async def echo_hi():
>>>     result = ''
>>>     echo_proc = await asyncio.create_subprocess_exec(
>>>             'echo', 'hello', 'world',
>>>             stdout=asyncio.subprocess.PIPE,
>>>             stderr=asyncio.subprocess.DEVNULL)
>>>     result = await echo_proc.stdout.read()
>>>     await echo_proc.wait()
>>>     return result
>>>
>>> async def await_twice(awaitable):
>>>     print('first time is {}'.format(await awaitable))
>>>     print('second time is {}'.format(await awaitable))
>>>
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(await_twice(echo_hi()))
>>>
>>> This makes writing composable APIs using async/await in Python very
>>> difficult since anything that takes an `awaitable` has to know that it
>>> wasn't already awaited.  Also, since the behavior is radically different
>>> than in the other programming languages implementing async/await it makes
>>> adopting Python's flavor of async/await difficult for folks coming from a
>>> language where it's already implemented.
>>>
>>> In C#/Hack/JS calls to `await` return a Task/AwaitableHandle/Promise
>>> that can be awaited multiple times and either returns the result or throws
>>> any thrown exceptions.  It doesn't appear that the Awaitable class in
>>> Python has a `result` or `exception` field but `asyncio.Future` does.
>>>
>>> Would it make sense to shift from having `await` functions return a `
>>> *Future-like`* return object to returning a Future?
>>>
>>> Thanks,
>>> Roy
>>>
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151215/61fb846a/attachment.html>