[Python-ideas] The async API of the future: yield-from

Dino Viehland dinov at microsoft.com
Mon Oct 15 23:45:13 CEST 2012 

They look remarkably similar.  The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along.  So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously.  But frameworks can replace the default context so that work is posted into an event loop of some form.

-----Original Message-----
From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum
Sent: Monday, October 15, 2012 12:34 PM
To: Dino Viehland
Cc: ironfroggy at gmail.com; Nick Coghlan; python-ideas at python.org
Subject: Re: [Python-ideas] The async API of the future: yield-from

Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop:
http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets.py

On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov at microsoft.com> wrote:
> I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space.  We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3.  Our version of this is also based around futures so that an @async API will return a future.  The big difference here might be that we always return a future from a call rather than yielding it up the stack.  So our API works with just simple yields rather than yield froms.  This is what a simple usage of the API looks like:
>
>         from concurrent.futures import ThreadPoolExecutor
>         from urllib.request import urlopen
>
>         executor = ThreadPoolExecutor(max_workers=5)
>
>         def load_url(url):
>             return urlopen(_url).read()
>
>         @async
>         def get_image_async(url):
>             buffer = yield executor.submit(load_url, url)
>             return Image(buffer)
>
>         def main(image_uri):
>             img_future = get_image_async(image_uri)
>             # perform other tasks while the image is downloading
>             img = img_future.result()
>
>         main("http://www.python.org/images/python-logo.gif")
>
> This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future.  So inside of an async method anything which is yielded should be a future.  The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future.  Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
>
> Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there.  Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
>
> The big outstanding item we're still working through is I/O, but we think the contexts help here too.  We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue.  We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
>
> Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
>
> Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: 
> http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN
> Hardwork.py: http://pastebin.com/nMMytdTG
>
>
>
>
> -----Original Message-----
> From: Python-ideas 
> [mailto:python-ideas-bounces+dinov=microsoft.com at python.org] On Behalf 
> Of Calvin Spealman
> Sent: Monday, October 15, 2012 7:16 AM
> To: Nick Coghlan
> Cc: python-ideas at python.org
> Subject: Re: [Python-ideas] The async API of the future: yield-from
>
> On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>>> Currently, "with yield expr:" is not valid syntax, surprisingly.
>>
>> It's not that surprising, it's the general requirement that yield 
>> expressions must be enclosed in parentheses except when used 
>> standalone or in a simple assignment statement.
>>
>> "with (yield expr):" is valid syntax though, so I'm reluctant to 
>> endorse doing anything substantially different if the parentheses are 
>> omitted.
>
> Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
>
>> I think the combination of "yield from" to delegate control 
>> (including exception handling) completely to a subgenerator and 
>> "context manager
>> + for loop + explicit yield" when an operation needs to yield 
>> + multiple
>> times and the exception handling behaviour should be left to the 
>> caller (as in the "as_completed" case) should cover the necessary 
>> behaviours.
>
> I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.  I think it is far less flexible.
>
> I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
>
> with yield collect() as tasks:
>   yield task1()
>   yield task2()
> results = yield tasks
>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
>
>
> --
> Read my blog! I depend on your acceptance of my opinion! I am interesting!
> http://techblog.ironfroggy.com/
> Follow me if you're into that sort of thing: 
> http://www.twitter.com/ironfroggy 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



--
--Guido van Rossum (python.org/~guido)

More information about the Python-ideas mailing list