PEP 3156 - Asynchronous IO Support Rebooted

Dear python-dev *and* python-ideas, I am posting PEP 3156 here for early review and discussion. As you can see from the liberally sprinkled TBD entries it is not done, but I am about to disappear on vacation for a few weeks and I am reasonably happy with the state of things so far. (Of course feedback may change this. :-) Also, there has already been some discussion on python-ideas (and even on Twitter) so I don't want python-dev to feel out of the loop -- this *is* a proposal for a new standard library module. (But no, I haven't picked the module name yet. :-) There's an -- also incomplete -- reference implementation at http://code.google.com/p/tulip/ -- unlike the first version of tulip, this version actually has (some) unittests. Let the bikeshedding begin! (Oh, happy holidays too. :-) -- --Guido van Rossum (python.org/~guido)

On Friday, December 21, 2012 at 1:57 PM, Guido van Rossum wrote:
Dear python-dev *and* python-ideas,
I am posting PEP 3156 here for early review and discussion. As you can see from the liberally sprinkled TBD entries it is not done, but I am about to disappear on vacation for a few weeks and I am reasonably happy with the state of things so far. (Of course feedback may change this. :-) Also, there has already been some discussion on python-ideas (and even on Twitter) so I don't want python-dev to feel out of the loop -- this *is* a proposal for a new standard library module. (But no, I haven't picked the module name yet. :-)
There's an -- also incomplete -- reference implementation at http://code.google.com/p/tulip/ -- unlike the first version of tulip, this version actually has (some) unittests.
Let the bikeshedding begin!
(Oh, happy holidays too. :-)
-- --Guido van Rossum (python.org/~guido (http://python.org/~guido))
I really do like tulip as the name. It's quite pretty.

On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller <jnoller@gmail.com> wrote:
I really do like tulip as the name. It's quite pretty.
I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-) Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something... -- --Guido van Rossum (python.org/~guido)

Looks reasonable to me :) Comments: create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :) eof_received on protocols seems unusual. What's the rationale? I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP? A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield? Will comment more as I keep reading I'm sure :) On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum <guido@python.org> wrote:
On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller <jnoller@gmail.com> wrote:
I really do like tulip as the name. It's quite pretty.
I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-)
Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something...
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- cheers lvh

As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido? In case of sleep, sleep could be implemented to return a Future object. 2012/12/21 Laurens Van Houtven <_@lvh.cc>
A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield?
On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum <guido@python.org>wrote:
On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller <jnoller@gmail.com> wrote:
I really do like tulip as the name. It's quite pretty.
I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-)
Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something...
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- cheers lvh
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Fri, Dec 21, 2012 at 2:26 PM, Jonathan Slenders <jonathan@slenders.be> wrote:
As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido?
Correct! Sounds like you got it now. That's the magic of yield from..
In case of sleep, sleep could be implemented to return a Future object.
It does; in tulip/futures.py: def sleep(when, result=None): future = Future() future._event_loop.call_later(when, future.set_result, result) return future -- --Guido van Rossum (python.org/~guido)

On Fri, Dec 21, 2012 at 1:04 PM, Laurens Van Houtven <_@lvh.cc> wrote:
Looks reasonable to me :) Comments:
create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :)
If you really need this, it's probably best to start out doing this as a nonstandard extension of an implementation. The current *implementation* makes it simple enough, but I don't think it's worth complicating the PEP. Working code might convince me otherwise.
eof_received on protocols seems unusual. What's the rationale?
Well how else would you indicate that the other end did a half-close (in Twisted terminology)? You can't call connection_lost() because you might still want to write more. E.g. this is how HTTP servers work if there's no Content-length or chunked encoding on a request body: they read until EOF, then do their thing and write the response.
I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP?
Sorry, I have no idea what you're talking about. Can you clarify? I do know that the PEP is weakest in specifying how a coroutine can implement a transport. However my plans are clear: ild the old tulip code there's a BufferedReader; somehow the coroutine will receive a "stdin" and a "stdout" where the "stdin" is a BufferedReader, which has methods like read(), readline() etc. which return Futures and must be invoked using yield from; and "stdout" is a transport, which has write() and friends that don't return anything but just buffer stuff and start the I/O asynchronous (and may try to slow down the protocol by calling its pause() method).
A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield?
In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.)
Will comment more as I keep reading I'm sure :)
Please do! -- --Guido van Rossum (python.org/~guido)

On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum <guido@python.org> wrote: ... snip ... In PEP 3156 conformant code you're supposed always to use 'yield
from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.)
Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters.
Will comment more as I keep reading I'm sure :)
Please do!
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Jasper

On Fri, Dec 21, 2012 at 5:17 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum <guido@python.org> wrote:
... snip ...
In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.)
Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters.
That's also a possibility. If someone can figure out a patch that would be great. -- --Guido van Rossum (python.org/~guido)

We were tentatively calling it "concurrent.eventloop" at the 2011 language summit. -- Sent from my phone, thus the relative brevity :)

As the maintainer of a pretty large, complex app written in Twisted, I think this is great. I look forward to a future of being able to select from a broad library of async tools, and being able to write tools that can be used outside of Twisted. Buildbot began, lo these many years ago, doing a lot of things in memory on on local disk, neither of which require asynchronous IO. So a lot of API methods did not originally return Deferreds. Those methods are then used by other methods, many of which also do not return Deferreds. Now, we want to use a database backend, and parallelize some of the operations, meaning that the methods need to return a Deferred. Unfortunately, that requires a complete tree traversal of all of the methods and methods that call them, rewriting them to take and return Deferreds. There's no "halfway" solution. This is a little easier with generators (@inlineCallbacks), since the syntax doesn't change much, but it's a significant change to the API (in fact, this is a large part of the reason for the big rewrite for Buildbot-0.9.x). I bring all this up to say, this PEP will introduce a new "kind" of method signature into standard Python, one which the caller must know, and the use of which changes the signature of the caller. That can cause sweeping changes, and debugging those changes can be tricky. Two things can help: First, `yield from somemeth()` should work fine even if `somemeth` is not a coroutine function, and authors of async tools should be encouraged to use this form to assist future-compatibility. Second, `somemeth()` without a yield should fail loudly if `somemeth` is a coroutine function. Otherwise, the effects can be pretty confusing. In http://code.google.com/p/uthreads, I accomplished the latter by taking advantage of garbage collection: if the generator is garbage collected before it's begun, then it's probably not been yielded. This is a bit gross, but good enough as a debugging technique. On the topic of debugging, I also took pains to make sure that tracebacks looked reasonable, filtering out scheduler code[1]. I haven't looked closely at Tulip to see if that's a problem. Most of the "noise" in the tracebacks came from the lack of 'yield from', so it may not be an issue at all. Dustin [1] http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253

On Fri, Jan 4, 2013 at 2:38 PM, Dustin Mitchell <djmitche@gmail.com> wrote:
As the maintainer of a pretty large, complex app written in Twisted, I think this is great. I look forward to a future of being able to select from a broad library of async tools, and being able to write tools that can be used outside of Twisted.
Thanks. Me too. :-)
Buildbot began, lo these many years ago, doing a lot of things in memory on on local disk, neither of which require asynchronous IO. So a lot of API methods did not originally return Deferreds. Those methods are then used by other methods, many of which also do not return Deferreds. Now, we want to use a database backend, and parallelize some of the operations, meaning that the methods need to return a Deferred. Unfortunately, that requires a complete tree traversal of all of the methods and methods that call them, rewriting them to take and return Deferreds. There's no "halfway" solution. This is a little easier with generators (@inlineCallbacks), since the syntax doesn't change much, but it's a significant change to the API (in fact, this is a large part of the reason for the big rewrite for Buildbot-0.9.x).
I bring all this up to say, this PEP will introduce a new "kind" of method signature into standard Python, one which the caller must know, and the use of which changes the signature of the caller. That can cause sweeping changes, and debugging those changes can be tricky.
Yes, and this is the biggest unproven point of the PEP. (The rest is all backed by a decade or more of experience.)
Two things can help:
First, `yield from somemeth()` should work fine even if `somemeth` is not a coroutine function, and authors of async tools should be encouraged to use this form to assist future-compatibility. Second, `somemeth()` without a yield should fail loudly if `somemeth` is a coroutine function. Otherwise, the effects can be pretty confusing.
That would be nice. But the way yield from and generators work, that's hard to accomplish without further changes to the language -- and I don't want to have to change the language again (at least not immediately -- maybe in a few releases, after we've learned what the real issues are). The best I can do for the first requirement is to define @coroutine in a way that if the decorated function isn't a generator, it is wrapped in one. For the second requirement, if you call somemeth() and ignore the result, nothing happens at all -- this is indeed infuriating but I see no way to change this.(*) If you use the result, well, Futures have different attributes than most other objects so hopefully you'll get a loud AttributeError or TypeError soon, but of course if you pass it into something else which uses it, it may still be difficult to track. Hopefully these error messages provide a hint:
f.foo Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'Future' object has no attribute 'foo' f() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'Future' object is not callable
(*) There's a heavy gun we might use, but I would make this optional, as a heavy duty debugging mode only. @coroutine could wrap generators in a lightweight object with a __del__ method and an __iter__ method. If __del__ is called before __iter__ is ever called, it could raise an exception or log a warning. But this probably adds too much overhead to have it always enabled.
In http://code.google.com/p/uthreads, I accomplished the latter by taking advantage of garbage collection: if the generator is garbage collected before it's begun, then it's probably not been yielded. This is a bit gross, but good enough as a debugging technique.
Eh, yeah, what I said. :-)
On the topic of debugging, I also took pains to make sure that tracebacks looked reasonable, filtering out scheduler code[1]. I haven't looked closely at Tulip to see if that's a problem. Most of the "noise" in the tracebacks came from the lack of 'yield from', so it may not be an issue at all.
One of the great advantages of using yield from is that the tracebacks automatically look nice.
Dustin
[1] http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253
-- --Guido van Rossum (python.org/~guido)

Hello,
To get the current event loop, use get_event_loop(). This returns an instance of the EventLoop class defined below or an equivalent object. It is possible that get_event_loop() returns a different object depending on the current thread, or depending on some other notion of context.
To set the current event loop, use set_event_loop(event_loop), where event_loop is an instance of the EventLoop class or equivalent. This uses the same notion of context as get_event_loop().
So can we instantiate an EventLoop directly and then call set_event_loop() with it? Or is the use case different?
- ``create_transport(protocol_factory, host, port, **kwargs)``. Creates a transport and a protocol and ties them together. Returns a Future whose result on success is a (transport, protocol) pair. Note that when the Future completes, the protocol's ``connection_made()`` method has not yet been called; that will happen when the connection handshake is complete. When it is impossible to connect to the given host and port, the Future will raise an exception instead.
Optional keyword arguments:
- ``family``, ``type``, ``proto``, ``flags``: Address familty, socket type, protcol, and miscellaneous flags to be passed through to ``getaddrinfo()``. These all default to ``0`` except ``type`` which defaults to ``socket.SOCK_STREAM``.
- ``ssl``: Pass ``True`` to create an SSL transport (by default a plain TCP is created). Or pass an ``ssl.SSLContext`` object to override the default SSL context object to be used.
TBD: Should this be called create_connection()?
Either create_connection() or create_client(). create_transport() is wrong, since server transports wouldn't use that function. I would favour create_client() if this function is also meant to support UDP (I know you haven't thought about UDP yet, but it is an important and common use case). I have another question about that API: if I want to cancel the connection attempt after a given delay, how do I do that? If I call cancel() on the future, does it cancel the connect() call? As for SSL, there are security issues with having a "default SSL context" (notably, any decent client use of SSL *must* check the server certificate against an appropriate set of CAs). It's much better to force users to pass a context explicitly. Choosing default settings should only be for higher-level APIs like urllib.request. (btw, don't you mean that family defaults to AF_INET?)
If executor is None, a default ThreadPoolExecutor with 5 threads is used
Is it because Twisted's thread pool has minThreads=5? :)
The transport is free to buffer the bytes, but it must eventually cause the bytes to be transferred to the entity at the other end, and it must maintain stream behavior. That is, t.write(b'abc'); t.write(b'def') is equivalent to t.write(b'abcdef')
I think this is a bad idea. The kernel's network stack should do the buffering (and choose appropriate algorithms for that), not the user-level framework. The transport should write the bytes as soon as the fd is ready for writing, and it should write the same chunks as given by the user, not a concatenation of them. Besides, it would be better if transports weren't automatically *streaming* transports. There are connected datagram protocols, such as named pipes under Windows (multiprocessing already uses non-blocking Windows named pipes).
Proposal: let the transport call protocol.pause() and protocol.resume() if they exist; if they don't exist, the protocol doesn't support flow control.
+1. The Protocol base class can provide default no-op implementations.
TBD: Discuss whether user code needs to do anything to make sure that protocol and transport aren't garbage-collected prematurely.
The transport should be tied to the event loop as long as the connection holds, and the protocol will hold to the transport.
TBD: Need an interface to wait for the first of a collection of Futures.
Have you looked at Twisted's DeferredList? http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferre... I think par() could take a keyword-only argument to specify you want the callback to be triggered on the first result (and perhaps being able to choose between "the first success result" and "the first success or error result").
A trick used by Richard Oudkerk in the tulip project's proactor branch makes calls like recv() either return a regular result or raise a Future. The caller (likely a transport) must then write code like this:
Isn't it a case of premature optimization? If we want to keep this, there should be a nicer API, perhaps like Twisted's maybeDeferred: http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#m...
We might also introduce explicit locks (though these will be a bit of a pain to use, as we can't use the with lock: block syntax).
I don't understand why you couldn't use "with lock" in a coroutine. Am I misunderstanding something?
Is it reasonable to map write(), writelines(), data_received() to single datagrams?
Well, at least that's how Twisted does it (not sure about writelines()). Regards Antoine.

Inline. --Guido van Rossum (sent from Android phone) On Dec 21, 2012 11:47 AM, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
Hello,
To get the current event loop, use get_event_loop(). This returns an instance of the EventLoop class defined below or an equivalent object. It is possible that get_event_loop() returns a different object depending on the current thread, or depending on some other notion of context.
To set the current event loop, use set_event_loop(event_loop), where event_loop is an instance of the EventLoop class or equivalent. This uses the same notion of context as get_event_loop().
So can we instantiate an EventLoop directly and then call set_event_loop() with it? Or is the use case different?
That's an abstract class, but if you know a concert implementation you can use this. E.g. latest Tulip unit tests.
- ``create_transport(protocol_factory, host, port, **kwargs)``. Creates a transport and a protocol and ties them together. Returns a Future whose result on success is a (transport, protocol) pair. Note that when the Future completes, the protocol's ``connection_made()`` method has not yet been called; that will happen when the connection handshake is complete. When it is impossible to connect to the given host and port, the Future will raise an exception instead.
Optional keyword arguments:
- ``family``, ``type``, ``proto``, ``flags``: Address familty, socket type, protcol, and miscellaneous flags to be passed through to ``getaddrinfo()``. These all default to ``0`` except ``type`` which defaults to ``socket.SOCK_STREAM``.
- ``ssl``: Pass ``True`` to create an SSL transport (by default a plain TCP is created). Or pass an ``ssl.SSLContext`` object to override the default SSL context object to be used.
TBD: Should this be called create_connection()?
Either create_connection() or create_client(). create_transport() is wrong, since server transports wouldn't use that function.
I would favour create_client() if this function is also meant to support UDP (I know you haven't thought about UDP yet, but it is an important and common use case).
OK.
I have another question about that API: if I want to cancel the connection attempt after a given delay, how do I do that? If I call cancel() on the future, does it cancel the connect() call?
It does in Tulip, because it's really a task. Maybe this should be in the spec?
As for SSL, there are security issues with having a "default SSL context" (notably, any decent client use of SSL *must* check the server certificate against an appropriate set of CAs). It's much better to force users to pass a context explicitly. Choosing default settings should only be for higher-level APIs like urllib.request.
Hm. That makes simple tests harder. But I understand the concern.
(btw, don't you mean that family defaults to AF_INET?)
If executor is None, a default ThreadPoolExecutor with 5 threads is used
Is it because Twisted's thread pool has minThreads=5? :)
Yes, and to encourage the use of set_default_executor() ... :-)
The transport is free to buffer the bytes, but it must eventually cause the bytes to be transferred to the entity at the other end, and it must maintain stream behavior. That is, t.write(b'abc'); t.write(b'def') is equivalent to t.write(b'abcdef')
I think this is a bad idea. The kernel's network stack should do the buffering (and choose appropriate algorithms for that), not the user-level framework. The transport should write the bytes as soon as the fd is ready for writing, and it should write the same chunks as given by the user, not a concatenation of them.
I asked Glyph about this. It depends on the OS... Mac syscalls are so slow that it is better to join in user space. This should really be up to the transport, although for stream transports the given equivalency should definitely hold.
Besides, it would be better if transports weren't automatically *streaming* transports. There are connected datagram protocols, such as named pipes under Windows (multiprocessing already uses non-blocking Windows named pipes).
I think we need to support datagrams, but the default ought to be stream.
Proposal: let the transport call protocol.pause() and protocol.resume() if they exist; if they don't exist, the protocol doesn't support flow control.
+1. The Protocol base class can provide default no-op implementations.
OK.
TBD: Discuss whether user code needs to do anything to make sure that protocol and transport aren't garbage-collected prematurely.
The transport should be tied to the event loop as long as the connection holds, and the protocol will hold to the transport.
OK.
TBD: Need an interface to wait for the first of a collection of Futures.
Have you looked at Twisted's DeferredList?
http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferre... No, I am trying to stay away from them.
I think par() could take a keyword-only argument to specify you want the callback to be triggered on the first result (and perhaps being able to choose between "the first success result" and "the first success or error result").
Good idea. This is unexplored.
A trick used by Richard Oudkerk in the tulip project's proactor branch makes calls like recv() either return a regular result or raise a Future. The caller (likely a transport) must then write code like this:
Isn't it a case of premature optimization?
Yeah, we should not do this.
If we want to keep this, there should be a nicer API, perhaps like Twisted's maybeDeferred:
http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#m... Ugh.
We might also introduce explicit locks (though these will be a bit of a pain to use, as we can't use the with lock: block syntax).
I don't understand why you couldn't use "with lock" in a coroutine. Am I misunderstanding something?
If another task has the lock we must yield. But 'with' can't do that.
Is it reasonable to map write(), writelines(), data_received() to single datagrams?
Well, at least that's how Twisted does it (not sure about writelines()).
OK.
Regards
Antoine.
Thanks!

On Dec 21, 2012, at 1:10 PM, Guido van Rossum <guido@python.org> wrote:
The transport is free to buffer the bytes, but it must eventually cause the bytes to be transferred to the entity at the other end, and it must maintain stream behavior. That is, t.write(b'abc'); t.write(b'def') is equivalent to t.write(b'abcdef')
I think this is a bad idea. The kernel's network stack should do the buffering (and choose appropriate algorithms for that), not the user-level framework. The transport should write the bytes as soon as the fd is ready for writing, and it should write the same chunks as given by the user, not a concatenation of them.
I asked Glyph about this. It depends on the OS... Mac syscalls are so slow that it is better to join in user space. This should really be up to the transport, although for stream transports the given equivalency should definitely hold.
It's not so much that "mac syscalls are slow" as that "syscalls are not free, and the cost varies". Older versions of MacOS were particularly bad. Some versions of Linux had bizarre regressions in the performance of send() or recv() or pipe(). The things that pass for syscalls on Windows can be particularly catastrophically slow (although this is practically a consideration for filesystem APIs, not socket APIs, who knows what this the future will hold). There are a number of other reasons why this should be this way as well. User-space has the ability to buffer indefinitely, and the kernel does not. Sometimes, send() returns a truncated value, and you have to deal with this. Since you've allocated the memory for the value you're calling write() with anyway, you might as well stash it away in the framework. The alternative is to let every application implement - and by implement, I mean "screw up" - a low-performance buffering implementation. User-space has more information about the type of information being sent. If the user does write() write() write() within one loop iteration, the framework can hypothetically optimize that into a single syscall using scatter-gather I/O. (Fun fact: we tried this, and it turns out that some implementations of scatter-gather I/O are actually *slower* than naive repeated calls; information like this should, again, be preserved within the framework.) In order to preserve compatibility with other systems (Twisted, Tornado, et. al.), the framework must be within its rights to do the buffering itself, even if it actually does exactly what you're suggesting because that happens to be better for performance in some circumstances. Choosing different buffering strategies for different applications is an important tuning option. Applications which appear to work in some contexts if the boundaries of data passed to send() are exactly the same as the boundaries of the data sent to write() should not be coddled; this just makes them harder to debug later. They should be broken as soon as possible. This is a subtle, pernicious and nearly constant error that people new to networking make and the sooner it surfaces, the better. The segments passed to data_received() should be as different as possible from the segments passed to write().
Besides, it would be better if transports weren't automatically *streaming* transports. There are connected datagram protocols, such as named pipes under Windows (multiprocessing already uses non-blocking Windows named pipes).
I think we need to support datagrams, but the default ought to be stream.
In my humble (but entirely, verifiably correct) opinion, thinking of this as a "default" is propagating a design error in the BSD sockets API. Datagram and stream sockets have radically different semantics. In Twisted, "dataReceived" and "datagramReceived" are different methods for a good reason. Again, it's very very easy to fall into the trap of thinking that a TCP segment is a datagram and writing all your application code as if it were. After all, it probably works over localhost most of the time! This difference in semantics mirrored by a difference in method naming has helped quite a few people grok the distinction between streaming and datagrams over the years; I think it would be a good idea if Tulip followed suit. -glyph

On Mon, Dec 24, 2012 at 2:58 PM, Glyph <glyph@twistedmatrix.com> wrote:
In my humble (but entirely, verifiably correct) opinion, thinking of this as a "default" is propagating a design error in the BSD sockets API. Datagram and stream sockets have radically different semantics. In Twisted, "dataReceived" and "datagramReceived" are different methods for a good reason. Again, it's very very easy to fall into the trap of thinking that a TCP segment is a datagram and writing all your application code as if it were. After all, it probably works over localhost most of the time! This difference in semantics mirrored by a difference in method naming has helped quite a few people grok the distinction between streaming and datagrams over the years; I think it would be a good idea if Tulip followed suit.
Suppose PEP 3156 / Tulip uses data_received() for streams and datagram_received() for datagram protocols (which seems reasonable enough), what API should a datagram transport have for sending datagrams? write_datagram() and write_datagram_list()? (Given that the transport and protocol classes are different for datagrams, I suppose the create*() methods should also be different, rather than just having a type={SOCK_STREAM,SOCK_DATAGRAM} flag. But I can figure that out for myself. The naming and exact APIs for the client and server transport creation methods are in flux anyway.) -- --Guido van Rossum (python.org/~guido)

On Jan 4, 2013, at 3:56 PM, Guido van Rossum <guido@python.org> wrote:
On Mon, Dec 24, 2012 at 2:58 PM, Glyph <glyph@twistedmatrix.com> wrote:
In my humble (but entirely, verifiably correct) opinion, thinking of this as a "default" is propagating a design error in the BSD sockets API. Datagram and stream sockets have radically different semantics. In Twisted, "dataReceived" and "datagramReceived" are different methods for a good reason. Again, it's very very easy to fall into the trap of thinking that a TCP segment is a datagram and writing all your application code as if it were. After all, it probably works over localhost most of the time! This difference in semantics mirrored by a difference in method naming has helped quite a few people grok the distinction between streaming and datagrams over the years; I think it would be a good idea if Tulip followed suit.
Suppose PEP 3156 / Tulip uses data_received() for streams and datagram_received() for datagram protocols (which seems reasonable enough), what API should a datagram transport have for sending datagrams? write_datagram() and write_datagram_list()?
Twisted just have a different method called write() which has a different signature (data, address). Probably write_datagram is better. Why write_datagram_list though? Twisted's writeSequence is there to provide the (eventual) opportunity to optimize by writev; since datagrams are always sent one at a time anyway, write_datagram_list would seem to be a very minor optimization. -glyph

On Fri, Jan 4, 2013 at 8:19 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jan 4, 2013, at 3:56 PM, Guido van Rossum <guido@python.org> wrote:
On Mon, Dec 24, 2012 at 2:58 PM, Glyph <glyph@twistedmatrix.com> wrote:
In my humble (but entirely, verifiably correct) opinion, thinking of this as a "default" is propagating a design error in the BSD sockets API. Datagram and stream sockets have radically different semantics. In Twisted, "dataReceived" and "datagramReceived" are different methods for a good reason. Again, it's very very easy to fall into the trap of thinking that a TCP segment is a datagram and writing all your application code as if it were. After all, it probably works over localhost most of the time! This difference in semantics mirrored by a difference in method naming has helped quite a few people grok the distinction between streaming and datagrams over the years; I think it would be a good idea if Tulip followed suit.
Suppose PEP 3156 / Tulip uses data_received() for streams and datagram_received() for datagram protocols (which seems reasonable enough), what API should a datagram transport have for sending datagrams? write_datagram() and write_datagram_list()?
Twisted just have a different method called write() which has a different signature (data, address). Probably write_datagram is better. Why write_datagram_list though? Twisted's writeSequence is there to provide the (eventual) opportunity to optimize by writev; since datagrams are always sent one at a time anyway, write_datagram_list would seem to be a very minor optimization.
That makes sense (you can see I haven't tried to use UDP in a long time :-). Should write_datagram() perhaps return a future? Or is there still a use case for buffering datagrams? -- --Guido van Rossum (python.org/~guido)

On Jan 4, 2013, at 8:51 PM, Guido van Rossum <guido@python.org> wrote:
On Fri, Jan 4, 2013 at 8:19 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jan 4, 2013, at 3:56 PM, Guido van Rossum <guido@python.org> wrote:
On Mon, Dec 24, 2012 at 2:58 PM, Glyph <glyph@twistedmatrix.com> wrote:
In my humble (but entirely, verifiably correct) opinion, thinking of this as a "default" is propagating a design error in the BSD sockets API. Datagram and stream sockets have radically different semantics. In Twisted, "dataReceived" and "datagramReceived" are different methods for a good reason. Again, it's very very easy to fall into the trap of thinking that a TCP segment is a datagram and writing all your application code as if it were. After all, it probably works over localhost most of the time! This difference in semantics mirrored by a difference in method naming has helped quite a few people grok the distinction between streaming and datagrams over the years; I think it would be a good idea if Tulip followed suit.
Suppose PEP 3156 / Tulip uses data_received() for streams and datagram_received() for datagram protocols (which seems reasonable enough), what API should a datagram transport have for sending datagrams? write_datagram() and write_datagram_list()?
Twisted just have a different method called write() which has a different signature (data, address). Probably write_datagram is better. Why write_datagram_list though? Twisted's writeSequence is there to provide the (eventual) opportunity to optimize by writev; since datagrams are always sent one at a time anyway, write_datagram_list would seem to be a very minor optimization.
That makes sense (you can see I haven't tried to use UDP in a long time :-).
Should write_datagram() perhaps return a future? Or is there still a use case for buffering datagrams?
There's not much value in returning a future even if you don't buffer. UDP packets can be dropped, and there's no acknowledgement from the remote end either when they're received or when they're dropped. You can get a couple hints from ICMP, but you can't rely on it, because lots of networks just dumbly filter it. Personally I think its flow control should work the same way as a TCP stream just for symmetry, but the only time that this becomes important in an application is when you're actually saturating your entire outbound network, and you need to notice and slow down. Returning a future would let you do this too, but might mislead users into thinking that "once write_datagram completes, the datagram is sent and the other side has it", which is another pernicious idea it's hard to disabuse people of. -glyph

On Dec 21, 2012, at 1:10 PM, Guido van Rossum <guido@python.org> wrote:
TBD: Need an interface to wait for the first of a collection of Futures.
Have you looked at Twisted's DeferredList? http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferre...
No, I am trying to stay away from them.
Those who do not understand Deferreds are doomed to re-implement them poorly ;-). (And believe me, I've seen more than a few poor re-implementations at this point...) -g
participants (9)
-
Antoine Pitrou
-
Dustin Mitchell
-
Glyph
-
Guido van Rossum
-
Jasper St. Pierre
-
Jesse Noller
-
Jonathan Slenders
-
Laurens Van Houtven
-
Nick Coghlan