Hi All, This week I finished some Python syntax on a Pypy fork. It was an experiment I was working on this week. We really needed a cleaner way of writing asynchronous code. So, instead of using the yield keyword and an @async decorator, we implemented the 'await' keyword, similar to c#. So, because I just now subscribed to python-ideas, I cannot reply immediately to the following thread: An async facade? (was Re: [Python-Dev] Socket timeout and completion based sockets) Anyway, like c# does, I implemented the await keyword for Python, and should say that I'm really confident of the usability of the result. Personally, I think this is a very clean solution for Twisted's @defer.inlineCalbacks, Tornado's @gen.engine, and similar functions in other async frameworks. We use it right now in a commercial web environment, where third party users should have to be able to write asynchronous code as easy as possible in a web based IDE. https://bitbucket.org/jonathanslenders/pypy Two interpreter hooks were added: (both accept a callable as parameter.)
sys.setawaithandler(wrapper) sys.setawaitresultwrapper(result_wrapper)
The first will set the I/O scheduler a functions for wrapping others functions which contain 'await' instead of 'yield'. This wrapper function will receive a generator as input. So, 'await' still acts like 'yield' for the interpreter, but the result is automatically wrapped by this function, if the await keyword was found. The second function will wrap the return result of asynchronous functions. So, unlike normal generators with 'yield' keywords, where 'await' has been used, we still can return a result. But this result will be wrapped by this function, so that the generator in the scheduler will be able te recognize the returned result. This: @defer.inlineCallbacks def async_function(deferred_param): a = yield deferred_param b = yield some_call(a) yield defer.returnValue(b) will now become: def async_function(deferred_param): a = await deferred_param b = await some_call(a) return b So, while still being explicit, it requires minimal syntax, and allows distinguishing between when to 'wait' for an asynchronous task, and when to pass the Future object around. I really have no idea whether this has been proposed before, I can only say that we are using it and it works pretty well. Cheers, Jonathan
Note that the "return b" is already being handled through the "StopIteration" proposal. I'm not a fan of the new syntax because it means that removing all the "await" keywords from a method changes the return value. Requiring the decorator means that this can cleanly be handled in all cases, even if the decorator implementation is a bit ugly. This means that all we have left is the "await" vs. "yield" vs. "yield from" discussion. I don't think the new valuable enough to warrant a new keyword. On Thu, Dec 20, 2012 at 5:52 PM, Jonathan Slenders <jonathan@slenders.be>wrote:
-- Jasper
As removing "yield" changes the return value of a function. Nothing different. For me +1 for the "StopIteration" proposal. That's certainly better, and more generic than what I said. So, the difference is still that the "await" proposal makes the @async decorator implicit. I'm still in favor of this because in asynchronous code, you can have really many functions with this decorator. And if someone forgets about that, getting a generator object instead of a Future is quite different in semantics. P.S. excuse me, Terry. 2012/12/21 Jasper St. Pierre <jstpierre@mecheye.net>
On Thu, Dec 20, 2012 at 3:34 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Carefully read PEP 3156, and the tulip implementation: http://code.google.com/p/tulip/source/browse/tulip/tasks.py . The @coroutine decorator is technically redundant when you use yield from. -- --Guido van Rossum (python.org/~guido)
Just read through the PEP3156. It's interesting to see. (I had no idea that yield from would return the result of the generator. It's clever, given that at this point it behaves different than a normal 'yield'.) One question. Why does @coroutine not convert the generator into a Future object right away? Just like @defer.inlineCallbacks in Twisted. This has the advantage that calling the function would simply start the coroutine. The point of my 'await' experiment was that I could do the following:
do_something() Task('do_something')
# (And there it starts executing) It's very personal, but I find it nicer to see the name of the called function as a Future instead of seeing a generator. Technically, coroutines and generators may be the same, but normally you wouldn't write a for-loop over a coroutine, and you can't make a Future of -say- an xrange-generator. And when not calling from another coroutine (like from the global scope during start-up), it's also a little more work to turn the generator into a Future every time. Here, "await" does what "yield" does. If you automatically turn coroutines into a Future object when calling, you'll never need a "yield from" in this case. I agree that "await" would be redundant, but somehow, if we had a hint to the interpreter that it would turn generator functions into Future objects during calling, that would be nice. I'm happy to get convinced otherwise. :) Jonathan 2012/12/21 Jonathan Slenders <jonathan@slenders.be>
On Fri, Dec 21, 2012 at 2:21 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Because once it is a Future, the scheduler has to get involved every time it yields, even if the yield doesn't do any I/O but just transfers control to a "subroutine". This is hard to get your head around, but it is worth it.
Just like @defer.inlineCallbacks in Twisted. This has the advantage that calling the function would simply start the coroutine.
But it would be much slower.
Yeah, and the same works with yield from in TUlip. The @coroutine decorator is not needed.
It's water under the bridge. We have PEP 380 in Python 3.3. I don't want to change the language again in 3.4. Maybe after that we can reconsider. -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 7:50 PM, Guido van Rossum <guido@python.org> wrote:
One thing I'll say is that I think the coroutine decorator should convert something like: @coroutine def blah(): return "result" into the generator equivalent. You can do a syntax hack with: @coroutine def blah(): if 0: yield return "result" but that feels bad. This sort of bug may seem unlikely, but a user may hit it if they're commenting out code, Maybe a generic @force_generator decorator might be useful... -- Jasper
On Fri, Dec 21, 2012 at 5:13 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
There's a tiny part of me that says that this might hide some bugs. But mostly I agree and not doing it might make certain changes harder. I did this in NDB too.
Right.
Maybe a generic @force_generator decorator might be useful...
That would be somebody else's PEP. :-) -- --Guido van Rossum (python.org/~guido)
On 12/20/2012 5:52 PM, Jonathan Slenders wrote: Please post plain text rather than html (same for all python.org lists). Html posts ofter come out a bit weird.
Anyway, like c# does, I implemented the await keyword for Python,
On my reader, this is normal size text.
Personally, I think this is a very clean solution for Twisted's
While this was half sized micro text. (It is normal here because by default Thunderbird converts to plain text for newsgroups and I am posting via news.gmane.org.) The alternation between full and half-height characters makes your post hard to read. -- Terry Jan Reedy
Have you read PEP 3156 and PEP 380? Instead of await, Python 3.3 has yield from, with the same semantics. This is somewhat more verbose, but has the advantage that it doesn't introduce a new keyword, and it's already in Python 3.3, so you can start using it now -- no fork of the language required. -- --Guido van Rossum (python.org/~guido)
Note that the "return b" is already being handled through the "StopIteration" proposal. I'm not a fan of the new syntax because it means that removing all the "await" keywords from a method changes the return value. Requiring the decorator means that this can cleanly be handled in all cases, even if the decorator implementation is a bit ugly. This means that all we have left is the "await" vs. "yield" vs. "yield from" discussion. I don't think the new valuable enough to warrant a new keyword. On Thu, Dec 20, 2012 at 5:52 PM, Jonathan Slenders <jonathan@slenders.be>wrote:
-- Jasper
As removing "yield" changes the return value of a function. Nothing different. For me +1 for the "StopIteration" proposal. That's certainly better, and more generic than what I said. So, the difference is still that the "await" proposal makes the @async decorator implicit. I'm still in favor of this because in asynchronous code, you can have really many functions with this decorator. And if someone forgets about that, getting a generator object instead of a Future is quite different in semantics. P.S. excuse me, Terry. 2012/12/21 Jasper St. Pierre <jstpierre@mecheye.net>
On Thu, Dec 20, 2012 at 3:34 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Carefully read PEP 3156, and the tulip implementation: http://code.google.com/p/tulip/source/browse/tulip/tasks.py . The @coroutine decorator is technically redundant when you use yield from. -- --Guido van Rossum (python.org/~guido)
Just read through the PEP3156. It's interesting to see. (I had no idea that yield from would return the result of the generator. It's clever, given that at this point it behaves different than a normal 'yield'.) One question. Why does @coroutine not convert the generator into a Future object right away? Just like @defer.inlineCallbacks in Twisted. This has the advantage that calling the function would simply start the coroutine. The point of my 'await' experiment was that I could do the following:
do_something() Task('do_something')
# (And there it starts executing) It's very personal, but I find it nicer to see the name of the called function as a Future instead of seeing a generator. Technically, coroutines and generators may be the same, but normally you wouldn't write a for-loop over a coroutine, and you can't make a Future of -say- an xrange-generator. And when not calling from another coroutine (like from the global scope during start-up), it's also a little more work to turn the generator into a Future every time. Here, "await" does what "yield" does. If you automatically turn coroutines into a Future object when calling, you'll never need a "yield from" in this case. I agree that "await" would be redundant, but somehow, if we had a hint to the interpreter that it would turn generator functions into Future objects during calling, that would be nice. I'm happy to get convinced otherwise. :) Jonathan 2012/12/21 Jonathan Slenders <jonathan@slenders.be>
On Fri, Dec 21, 2012 at 2:21 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Because once it is a Future, the scheduler has to get involved every time it yields, even if the yield doesn't do any I/O but just transfers control to a "subroutine". This is hard to get your head around, but it is worth it.
Just like @defer.inlineCallbacks in Twisted. This has the advantage that calling the function would simply start the coroutine.
But it would be much slower.
Yeah, and the same works with yield from in TUlip. The @coroutine decorator is not needed.
It's water under the bridge. We have PEP 380 in Python 3.3. I don't want to change the language again in 3.4. Maybe after that we can reconsider. -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 7:50 PM, Guido van Rossum <guido@python.org> wrote:
One thing I'll say is that I think the coroutine decorator should convert something like: @coroutine def blah(): return "result" into the generator equivalent. You can do a syntax hack with: @coroutine def blah(): if 0: yield return "result" but that feels bad. This sort of bug may seem unlikely, but a user may hit it if they're commenting out code, Maybe a generic @force_generator decorator might be useful... -- Jasper
On Fri, Dec 21, 2012 at 5:13 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
There's a tiny part of me that says that this might hide some bugs. But mostly I agree and not doing it might make certain changes harder. I did this in NDB too.
Right.
Maybe a generic @force_generator decorator might be useful...
That would be somebody else's PEP. :-) -- --Guido van Rossum (python.org/~guido)
On 12/20/2012 5:52 PM, Jonathan Slenders wrote: Please post plain text rather than html (same for all python.org lists). Html posts ofter come out a bit weird.
Anyway, like c# does, I implemented the await keyword for Python,
On my reader, this is normal size text.
Personally, I think this is a very clean solution for Twisted's
While this was half sized micro text. (It is normal here because by default Thunderbird converts to plain text for newsgroups and I am posting via news.gmane.org.) The alternation between full and half-height characters makes your post hard to read. -- Terry Jan Reedy
Have you read PEP 3156 and PEP 380? Instead of await, Python 3.3 has yield from, with the same semantics. This is somewhat more verbose, but has the advantage that it doesn't introduce a new keyword, and it's already in Python 3.3, so you can start using it now -- no fork of the language required. -- --Guido van Rossum (python.org/~guido)
participants (4)
-
Guido van Rossum -
Jasper St. Pierre -
Jonathan Slenders -
Terry Reedy