[Python-ideas] The async API of the future: yield-from

[Dino Viehland]

I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space.  We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3.  Our version of this is also based around futures so that an @async API will return a future.  The big difference here might be that we always return a future from a call rather than yielding it up the stack.  So our API works with just simple yields rather than yield froms.  This is what a simple usage of the API looks like:

        from concurrent.futures import ThreadPoolExecutor
        from urllib.request import urlopen
        
        executor = ThreadPoolExecutor(max_workers=5)

        def load_url(url):
            return urlopen(_url).read()

        @async
        def get_image_async(url):
            buffer = yield executor.submit(load_url, url)
            return Image(buffer)

        def main(image_uri):
            img_future = get_image_async(image_uri)
            # perform other tasks while the image is downloading
            img = img_future.result()

        main("http://www.python.org/images/python-logo.gif")

This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future.  So inside of an async method anything which is yielded should be a future.  The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future.  Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.

Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there.  Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.

The big outstanding item we're still working through is I/O, but we think the contexts help here too.  We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue.  We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.

Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):

Contexts.py: http://pastebin.com/ndS53Cd8
Tk context: http://pastebin.com/FuZwc1Ur
Tk app: http://pastebin.com/Fm5wMXpN
Hardwork.py: http://pastebin.com/nMMytdTG




-----Original Message-----
From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com at python.org] On Behalf Of Calvin Spealman
Sent: Monday, October 15, 2012 7:16 AM
To: Nick Coghlan
Cc: python-ideas at python.org
Subject: Re: [Python-ideas] The async API of the future: yield-from

On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>> Currently, "with yield expr:" is not valid syntax, surprisingly.
>
> It's not that surprising, it's the general requirement that yield 
> expressions must be enclosed in parentheses except when used 
> standalone or in a simple assignment statement.
>
> "with (yield expr):" is valid syntax though, so I'm reluctant to 
> endorse doing anything substantially different if the parentheses are 
> omitted.

Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.

> I think the combination of "yield from" to delegate control (including 
> exception handling) completely to a subgenerator and "context manager
> + for loop + explicit yield" when an operation needs to yield multiple
> times and the exception handling behaviour should be left to the 
> caller (as in the "as_completed" case) should cover the necessary 
> behaviours.

I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.  I think it is far less flexible.

I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.

with yield collect() as tasks:
  yield task1()
  yield task2()
results = yield tasks

> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia



--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________
Python-ideas mailing list
Python-ideas at python.org
http://mail.python.org/mailman/listinfo/python-ideas