PEP 3156 - Asynchronous IO Support Rebooted
Dear python-dev *and* python-ideas, I am posting PEP 3156 here for early review and discussion. As you can see from the liberally sprinkled TBD entries it is not done, but I am about to disappear on vacation for a few weeks and I am reasonably happy with the state of things so far. (Of course feedback may change this. :-) Also, there has already been some discussion on python-ideas (and even on Twitter) so I don't want python-dev to feel out of the loop -- this *is* a proposal for a new standard library module. (But no, I haven't picked the module name yet. :-) There's an -- also incomplete -- reference implementation at http://code.google.com/p/tulip/ -- unlike the first version of tulip, this version actually has (some) unittests. Let the bikeshedding begin! (Oh, happy holidays too. :-) -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller <jnoller@gmail.com> wrote:
I really do like tulip as the name. It's quite pretty.
I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-) Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something... -- --Guido van Rossum (python.org/~guido)
Looks reasonable to me :) Comments: create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :) eof_received on protocols seems unusual. What's the rationale? I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP? A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield? Will comment more as I keep reading I'm sure :) On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum <guido@python.org> wrote:
-- cheers lvh
As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido? In case of sleep, sleep could be implemented to return a Future object. 2012/12/21 Laurens Van Houtven <_@lvh.cc>
On Fri, Dec 21, 2012 at 2:26 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Correct! Sounds like you got it now. That's the magic of yield from..
In case of sleep, sleep could be implemented to return a Future object.
It does; in tulip/futures.py: def sleep(when, result=None): future = Future() future._event_loop.call_later(when, future.set_result, result) return future -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 1:04 PM, Laurens Van Houtven <_@lvh.cc> wrote:
If you really need this, it's probably best to start out doing this as a nonstandard extension of an implementation. The current *implementation* makes it simple enough, but I don't think it's worth complicating the PEP. Working code might convince me otherwise.
eof_received on protocols seems unusual. What's the rationale?
Well how else would you indicate that the other end did a half-close (in Twisted terminology)? You can't call connection_lost() because you might still want to write more. E.g. this is how HTTP servers work if there's no Content-length or chunked encoding on a request body: they read until EOF, then do their thing and write the response.
Sorry, I have no idea what you're talking about. Can you clarify? I do know that the PEP is weakest in specifying how a coroutine can implement a transport. However my plans are clear: ild the old tulip code there's a BufferedReader; somehow the coroutine will receive a "stdin" and a "stdout" where the "stdin" is a BufferedReader, which has methods like read(), readline() etc. which return Futures and must be invoked using yield from; and "stdout" is a transport, which has write() and friends that don't return anything but just buffer stuff and start the I/O asynchronous (and may try to slow down the protocol by calling its pause() method).
In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.)
Will comment more as I keep reading I'm sure :)
Please do! -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum <guido@python.org> wrote: ... snip ... In PEP 3156 conformant code you're supposed always to use 'yield
Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters.
-- Jasper
We were tentatively calling it "concurrent.eventloop" at the 2011 language summit. -- Sent from my phone, thus the relative brevity :)
On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller <jnoller@gmail.com> wrote:
I really do like tulip as the name. It's quite pretty.
I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-) Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something... -- --Guido van Rossum (python.org/~guido)
Looks reasonable to me :) Comments: create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :) eof_received on protocols seems unusual. What's the rationale? I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP? A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield? Will comment more as I keep reading I'm sure :) On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum <guido@python.org> wrote:
-- cheers lvh
As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido? In case of sleep, sleep could be implemented to return a Future object. 2012/12/21 Laurens Van Houtven <_@lvh.cc>
On Fri, Dec 21, 2012 at 2:26 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
Correct! Sounds like you got it now. That's the magic of yield from..
In case of sleep, sleep could be implemented to return a Future object.
It does; in tulip/futures.py: def sleep(when, result=None): future = Future() future._event_loop.call_later(when, future.set_result, result) return future -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 1:04 PM, Laurens Van Houtven <_@lvh.cc> wrote:
If you really need this, it's probably best to start out doing this as a nonstandard extension of an implementation. The current *implementation* makes it simple enough, but I don't think it's worth complicating the PEP. Working code might convince me otherwise.
eof_received on protocols seems unusual. What's the rationale?
Well how else would you indicate that the other end did a half-close (in Twisted terminology)? You can't call connection_lost() because you might still want to write more. E.g. this is how HTTP servers work if there's no Content-length or chunked encoding on a request body: they read until EOF, then do their thing and write the response.
Sorry, I have no idea what you're talking about. Can you clarify? I do know that the PEP is weakest in specifying how a coroutine can implement a transport. However my plans are clear: ild the old tulip code there's a BufferedReader; somehow the coroutine will receive a "stdin" and a "stdout" where the "stdin" is a BufferedReader, which has methods like read(), readline() etc. which return Futures and must be invoked using yield from; and "stdout" is a transport, which has write() and friends that don't return anything but just buffer stuff and start the I/O asynchronous (and may try to slow down the protocol by calling its pause() method).
In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.)
Will comment more as I keep reading I'm sure :)
Please do! -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum <guido@python.org> wrote: ... snip ... In PEP 3156 conformant code you're supposed always to use 'yield
Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters.
-- Jasper
We were tentatively calling it "concurrent.eventloop" at the 2011 language summit. -- Sent from my phone, thus the relative brevity :)
participants (6)
-
Guido van Rossum -
Jasper St. Pierre -
Jesse Noller -
Jonathan Slenders -
Laurens Van Houtven -
Nick Coghlan