[Twisted-Python] Sending a stream via Protocol?
Hi, I have a simple protocol listening on a port. I send it a text message (a filename) and the protocol tries to stream back the bytes of the file. Mostly, this works but in some cases, the receiving side receives less bytes than is written by: self.transport.write(bytes) I read the API which says it can receive less than expected bytes or messages. Why? So how can I reliably send a sequences of bytes via "dataReceived" in my protocol? Thanks for any help! Darren
On 04:42 pm, darren@ontrenet.com wrote:
Hi, I have a simple protocol listening on a port. I send it a text message (a filename) and the protocol tries to stream back the bytes of the file. Mostly, this works but in some cases, the receiving side receives less bytes than is written by:
self.transport.write(bytes)
I read the API which says it can receive less than expected bytes or messages. Why?
That's how TCP works. Bytes you send with one call to write may be split into two or more chunks and delivered to the remote dataReceived separately, or bytes you send with two or more calls to write may be combined into fewer chunks and delivered to the remote dataReceived all as one string.
So how can I reliably send a sequences of bytes via "dataReceived" in my protocol?
If you want "framing" - a byte format built on top of the basic functionality provided by TCP, allowing you to differentiate bytes belonging to different logical messages from each other - then you should take a look at some of the protocols in twisted.protocols.basic (NetstringReceiver, Int32StringReceiver, etc) or twisted.protocols.amp which expands on this idea to a much greater degree. You can read a lot more about these ideas in this series of articles by Itamar Turner-Trauring: http://www.xml.com/pub/au/215 Jean-Paul
On Sat, 2010-02-06 at 17:13 +0000, exarkun@twistedmatrix.com wrote:
On 04:42 pm, darren@ontrenet.com wrote:
Hi, I have a simple protocol listening on a port. I send it a text message (a filename) and the protocol tries to stream back the bytes of the file. Mostly, this works but in some cases, the receiving side receives less bytes than is written by:
self.transport.write(bytes)
I read the API which says it can receive less than expected bytes or messages. Why?
That's how TCP works. Bytes you send with one call to write may be split into two or more chunks and delivered to the remote dataReceived separately, or bytes you send with two or more calls to write may be combined into fewer chunks and delivered to the remote dataReceived all as one string.
Yes, at the lower OSI layers it does. But in most modern programming languages, the notion of an I/O stream is built atop it (e.g. sockets will recombine and order the packets, but twisted does not?). So naturally, I don't want to emulate IP in my Twisted app.
So how can I reliably send a sequences of bytes via "dataReceived" in my protocol?
If you want "framing" - a byte format built on top of the basic functionality provided by TCP, allowing you to differentiate bytes belonging to different logical messages from each other - then you should take a look at some of the protocols in twisted.protocols.basic (NetstringReceiver, Int32StringReceiver, etc) or twisted.protocols.amp which expands on this idea to a much greater degree.
Ok, cool. I thought this would be easier in Twisted, but I will research it more.
You can read a lot more about these ideas in this series of articles by Itamar Turner-Trauring:
Thanks again.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 05:30 pm, darren@ontrenet.com wrote:
On Sat, 2010-02-06 at 17:13 +0000, exarkun@twistedmatrix.com wrote:
That's how TCP works. Bytes you send with one call to write may be split into two or more chunks and delivered to the remote dataReceived separately, or bytes you send with two or more calls to write may be combined into fewer chunks and delivered to the remote dataReceived all as one string.
Yes, at the lower OSI layers it does. But in most modern programming languages, the notion of an I/O stream is built atop it (e.g. sockets will recombine and order the packets, but twisted does not?). So naturally, I don't want to emulate IP in my Twisted app.
I don't think so. There's simply not enough information available for fragmented packets to be recombined, or for coalesced packets to be split up again. This is what the other APIs I mentioned are for: they add the necessary information to the underlying TCP byte stream so that this can be done. Ordering is preserved, though, even if you only do what you were trying to do, because TCP guarantees ordering. Jean-Paul
participants (2)
-
Darren Govoni -
exarkun＠twistedmatrix.com