[Python-ideas] The async API of the future: yield-from

Steve Dower Steve.Dower at microsoft.com
Tue Oct 16 03:45:25 CEST 2012 

That's basically exactly the same as ours (I worked on it with Dino). We assume that yielded objects are futures and wire up the callback.

I think the difference is that all the yielded futures are hidden within the decorator, which returns one main future to the caller. This may be slightly inefficient, but it also makes it far easier for end-users. An event loop (in our terminology, a 'context') is only necessary if you need to ensure that callbacks (in this case, the next step in the generator) is run in a certain context (such as a UI thread). Without one, calling an @async method simply gives you back a future that you can wait on.

The most important part of PEP 380 for this approach is not yield from, but allowing return <expr> inside a generator. It makes the methods that much more natural.

Probably the most important part is that we assume whatever context is available (through contexts.get_current()) has a post() method for scheduling a callback. Basically, we approached this as less of a "how do I run this asynchronously" problem and more of a "how do I run something after this finishes" problem.

We also have some ideas about associating properties with futures in a way that lets the caller decide how to run continuations, so you can opt-out of coming back to the calling thread or provide a cancellation token of some sort. These aren't written up yet (obviously), but we've certainly considered it.

Cheers,
Steve

________________________________________
From: Python-ideas [python-ideas-bounces+steve.dower=microsoft.com at python.org] on behalf of Guido van Rossum [guido at python.org]
Sent: Monday, October 15, 2012 6:17 PM
To: Dino Viehland
Cc: python-ideas at python.org
Subject: Re: [Python-ideas] The async API of the future: yield-from

On Mon, Oct 15, 2012 at 2:45 PM, Dino Viehland <dinov at microsoft.com> wrote:
> They look remarkably similar.  The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along.  So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously.  But frameworks can replace the default context so that work is posted into an event loop of some form.

But do your Futures use threads? NDB doesn't. NDB's event loop doesn't
know about Futures; however the @ndb.tasklet decorator does, and the
Futures know about the event loop. When you wait for a Future, a
callback is added to the Future that will resume the generator when it
is done, and in order to run them, the Future passes its callbacks to
the event loop to be run.

--Guido

> -----Original Message-----
> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum
> Sent: Monday, October 15, 2012 12:34 PM
> To: Dino Viehland
> Cc: ironfroggy at gmail.com; Nick Coghlan; python-ideas at python.org
> Subject: Re: [Python-ideas] The async API of the future: yield-from
>
> Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop:
> http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py
>
> On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov at microsoft.com> wrote:
>> I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space.  We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3.  Our version of this is also based around futures so that an @async API will return a future.  The big difference here might be that we always return a future from a call rather than yielding it up the stack.  So our API works with just simple yields rather than yield froms.  This is what a simple usage of the API looks like:
>>
>>         from concurrent.futures import ThreadPoolExecutor
>>         from urllib.request import urlopen
>>
>>         executor = ThreadPoolExecutor(max_workers=5)
>>
>>         def load_url(url):
>>             return urlopen(_url).read()
>>
>>         @async
>>         def get_image_async(url):
>>             buffer = yield executor.submit(load_url, url)
>>             return Image(buffer)
>>
>>         def main(image_uri):
>>             img_future = get_image_async(image_uri)
>>             # perform other tasks while the image is downloading
>>             img = img_future.result()
>>
>>         main("http://www.python.org/images/python-logo.gif")
>>
>> This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future.  So inside of an async method anything which is yielded should be a future.  The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future.  Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
>>
>> Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there.  Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
>>
>> The big outstanding item we're still working through is I/O, but we think the contexts help here too.  We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue.  We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
>>
>> Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
>>
>> Contexts.py: http://pastebin.com/ndS53Cd8 Tk context:
>> http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN
>> Hardwork.py: http://pastebin.com/nMMytdTG
>>
>>
>>
>>
>> -----Original Message-----
>> From: Python-ideas
>> [mailto:python-ideas-bounces+dinov=microsoft.com at python.org] On Behalf
>> Of Calvin Spealman
>> Sent: Monday, October 15, 2012 7:16 AM
>> To: Nick Coghlan
>> Cc: python-ideas at python.org
>> Subject: Re: [Python-ideas] The async API of the future: yield-from
>>
>> On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>>>> Currently, "with yield expr:" is not valid syntax, surprisingly.
>>>
>>> It's not that surprising, it's the general requirement that yield
>>> expressions must be enclosed in parentheses except when used
>>> standalone or in a simple assignment statement.
>>>
>>> "with (yield expr):" is valid syntax though, so I'm reluctant to
>>> endorse doing anything substantially different if the parentheses are
>>> omitted.
>>
>> Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
>>
>>> I think the combination of "yield from" to delegate control
>>> (including exception handling) completely to a subgenerator and
>>> "context manager
>>> + for loop + explicit yield" when an operation needs to yield
>>> + multiple
>>> times and the exception handling behaviour should be left to the
>>> caller (as in the "as_completed" case) should cover the necessary
>>> behaviours.
>>
>> I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.  I think it is far less flexible.
>>
>> I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
>>
>> with yield collect() as tasks:
>>   yield task1()
>>   yield task2()
>> results = yield tasks
>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>>
>>
>> --
>> Read my blog! I depend on your acceptance of my opinion! I am interesting!
>> http://techblog.ironfroggy.com/
>> Follow me if you're into that sort of thing:
>> http://www.twitter.com/ironfroggy
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>>
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
>
>
>
>



--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
http://mail.python.org/mailman/listinfo/python-ideas

More information about the Python-ideas mailing list