PEP 3156 / Tulip question: write/send callback/future
Hi,
I'm looking through PEP 3156 and the Tulip code, and either something is missing or I'm not looking in the right places.
I can't find any sort of callback / future return for asynchronous writes, e.g. in transport.
Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be checked later for status? For connections that aren't infinitely fast it's useful to know when the data was actually sent/written, or alternatively if an error has occurred. This is also important for when writing would actually block because of full buffers. boost::asio has such a handler for async_write.
Eli
On Fri, Jan 18, 2013 at 6:56 AM, Eli Bendersky eliben@gmail.com wrote:
I'm looking through PEP 3156 and the Tulip code, and either something is missing or I'm not looking in the right places.
I can't find any sort of callback / future return for asynchronous writes, e.g. in transport.
I guess you should read some Twisted tutorial. :-)
Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be checked later for status? For connections that aren't infinitely fast it's useful to know when the data was actually sent/written, or alternatively if an error has occurred. This is also important for when writing would actually block because of full buffers. boost::asio has such a handler for async_write.
The model is a little different. Glyph has convinced me that it works well in practice. We just buffer what is written (when it can't all be sent immediately). This is enough for most apps that don't serve 100MB files. If the buffer becomes too large, the transport will call .pause() on the protocol until it is drained, then it calls .resume(). (The names of these are TBD, maybe they will end up .pause_writing() and .resume_writing().) There are some default behaviors that we can add here too, e.g. suspending the task.
On Fri, Jan 18, 2013 at 1:02 PM, Guido van Rossum guido@python.org wrote:
On Fri, Jan 18, 2013 at 6:56 AM, Eli Bendersky eliben@gmail.com wrote:
I'm looking through PEP 3156 and the Tulip code, and either something is missing or I'm not looking in the right places.
I can't find any sort of callback / future return for asynchronous
writes,
e.g. in transport.
I guess you should read some Twisted tutorial. :-)
Yes, I noticed that Twisted also doesn't have it, so I suspected that influence.
Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be checked later for status? For connections that aren't infinitely fast it's
useful to
know when the data was actually sent/written, or alternatively if an
error
has occurred. This is also important for when writing would actually
block
because of full buffers. boost::asio has such a handler for async_write.
The model is a little different. Glyph has convinced me that it works well in practice. We just buffer what is written (when it can't all be sent immediately). This is enough for most apps that don't serve 100MB files. If the buffer becomes too large, the transport will call .pause() on the protocol until it is drained, then it calls .resume(). (The names of these are TBD, maybe they will end up .pause_writing() and .resume_writing().) There are some default behaviors that we can add here too, e.g. suspending the task.
I agree it can be made to work, but how would even simple "done sending" notification work? Or "send error" for that matter? AFAIR, low-level socket async API do provide this information. Are we confident enough it will never be needed to simply hide it away?
Eli
On Fri, Jan 18, 2013 at 1:40 PM, Eli Bendersky eliben@gmail.com wrote:
On Fri, Jan 18, 2013 at 1:02 PM, Guido van Rossum guido@python.org wrote:
On Fri, Jan 18, 2013 at 6:56 AM, Eli Bendersky eliben@gmail.com wrote:
I'm looking through PEP 3156 and the Tulip code, and either something is missing or I'm not looking in the right places.
I can't find any sort of callback / future return for asynchronous writes, e.g. in transport.
I guess you should read some Twisted tutorial. :-)
Yes, I noticed that Twisted also doesn't have it, so I suspected that influence.
Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be checked later for status? For connections that aren't infinitely fast it's useful to know when the data was actually sent/written, or alternatively if an error has occurred. This is also important for when writing would actually block because of full buffers. boost::asio has such a handler for async_write.
The model is a little different. Glyph has convinced me that it works well in practice. We just buffer what is written (when it can't all be sent immediately). This is enough for most apps that don't serve 100MB files. If the buffer becomes too large, the transport will call .pause() on the protocol until it is drained, then it calls .resume(). (The names of these are TBD, maybe they will end up .pause_writing() and .resume_writing().) There are some default behaviors that we can add here too, e.g. suspending the task.
I agree it can be made to work, but how would even simple "done sending" notification work? Or "send error" for that matter? AFAIR, low-level socket async API do provide this information. Are we confident enough it will never be needed to simply hide it away?
AFAIK the Twisted folks have found that most of the time (basically all of the time) you don't need a positive "done sending" notification; when the send eventually *fails*, the transport calls the protocol's connection_lost() method with an exception indicating what failed.
Also, ISTR that it's not always possible to consistently have that behavior everywhere (i.e. have it in the first place or fake it where it's not directly available), so it's of somewhat limited utility, since a protocol can't actually rely on it existing. Most behavior that requires it is generally implemented using IPushProducer/IPullProducer (i.e. the pause/resume API Guido mentioned earlier).
There's been some attempts at work towards a better producer/consumer API (e.g. supporting things like buffer changes, and generally just simplifying things that seem duplicated amongst transports and consumers/producers) called 'tubes', but I don't think any of that is ready enough to be a template for tulip :)
On Fri, Jan 18, 2013 at 11:25 PM, Guido van Rossum guido@python.org wrote:
On Fri, Jan 18, 2013 at 1:40 PM, Eli Bendersky eliben@gmail.com wrote:
On Fri, Jan 18, 2013 at 1:02 PM, Guido van Rossum guido@python.org
wrote:
On Fri, Jan 18, 2013 at 6:56 AM, Eli Bendersky eliben@gmail.com
wrote:
I'm looking through PEP 3156 and the Tulip code, and either something
is
missing or I'm not looking in the right places.
I can't find any sort of callback / future return for asynchronous writes, e.g. in transport.
I guess you should read some Twisted tutorial. :-)
Yes, I noticed that Twisted also doesn't have it, so I suspected that influence.
Should there be no "data_sent" parallel to "data_received" somewhere? Or, alternatively, "write" returning some sort of future that can be
checked
later for status? For connections that aren't infinitely fast it's useful to know when the data was actually sent/written, or alternatively if an error has occurred. This is also important for when writing would actually block because of full buffers. boost::asio has such a handler for
async_write.
The model is a little different. Glyph has convinced me that it works well in practice. We just buffer what is written (when it can't all be sent immediately). This is enough for most apps that don't serve 100MB files. If the buffer becomes too large, the transport will call .pause() on the protocol until it is drained, then it calls .resume(). (The names of these are TBD, maybe they will end up .pause_writing() and .resume_writing().) There are some default behaviors that we can add here too, e.g. suspending the task.
I agree it can be made to work, but how would even simple "done sending" notification work? Or "send error" for that matter? AFAIR, low-level
socket
async API do provide this information. Are we confident enough it will
never
be needed to simply hide it away?
AFAIK the Twisted folks have found that most of the time (basically all of the time) you don't need a positive "done sending" notification; when the send eventually *fails*, the transport calls the protocol's connection_lost() method with an exception indicating what failed.
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
participants (3)
-
Eli Bendersky -
Guido van Rossum -
Laurens Van Houtven