[Twisted-Python] Packet Fragmentation
Hi, I'm using TCP to pass large files back and fourth. So far I've follow and expanded on the twisted code examples for a simple server/client. Everything works fine for me until the packets get large enough that fragmentation kicks in.
From one side, I'm sending packets like so:
class EchoClient(protocol.Protocol): def connectionMade(self): data = "" while len(data) < 10000: data=data+'a' self.transport.write(data) and on the receiving end: def dataReceived(self, data): print data I noticed after multiple iterations the data buffer fills up to the total 10000 bytes, but this takes 3 iterations of dataReceived(). Common sense suggests to have a simple "if len(data) == totalPacketSize" to solve my problem, but I have different sized packets coming in all the time, and can't hardcode this part. I was wondering if there's any way in Twisted/Python to handle packet fragmentation. Can I access the last fragment flag in the TCP packet somehow? Or is there a total packet size sent with the packet automatically? If not, what would be the best way to solve this? thanks, Nima
On 10:51 pm, nimag@rogers.com wrote:
I was wondering if there's any way in Twisted/Python to handle packet fragmentation. Can I access the last fragment flag in the TCP packet somehow? Or is there a total packet size sent with the packet automatically? If not, what would be the best way to solve this?
What you're dealing with here is not packet fragmentation. TCP packet fragmentation is a very specific phenomenon that occurs at a lower level than Twisted (and your application). When you send a large message over a TCP stream, it can get broken up into multiple segments. That's not the same as an individual packet getting fragmented. There's no way to re-assemble the message to discover the original boundaries of the "write" call that you used to emit it. In other words, your data stream needs to have message boundaries in it. TCP, at the application level, is a stream of octets, *not* a sequence of packets. There are numerous examples of this in Twisted; for example, twisted.protocols.basic includes NetstringReceiver, LineReceiver and (various subclasses of) IntNStringReceiver, each of which is a different strategy for breaking up a TCP stream into a discrete sequence of messages.
OoO En cette nuit nuageuse du jeudi 09 octobre 2008, vers 00:51, Nima Ghanavatian <nimag@rogers.com> disait :
I was wondering if there's any way in Twisted/Python to handle packet fragmentation. Can I access the last fragment flag in the TCP packet somehow? Or is there a total packet size sent with the packet automatically? If not, what would be the best way to solve this?
You can also use a protocol that keeps message boundaries like UDP. UDP is reliable and deliver messages in order if used on localhost. Otherwise, there are protocoles like SCTP that are reliable and keep message boundaries. However, I think there is no support for it in Twisted. -- BOFH excuse #93: Feature not yet implimented
On Thu, 09 Oct 2008 13:27:51 +0200, Vincent Bernat <bernat@luffy.cx> wrote:
OoO En cette nuit nuageuse du jeudi 09 octobre 2008, vers 00:51, Nima Ghanavatian <nimag@rogers.com> disait :
I was wondering if there's any way in Twisted/Python to handle packet fragmentation. Can I access the last fragment flag in the TCP packet somehow? Or is there a total packet size sent with the packet automatically? If not, what would be the best way to solve this?
You can also use a protocol that keeps message boundaries like UDP. UDP is reliable and deliver messages in order if used on localhost.
UDP is *not* reliable and will *not* necessarily deliver messages in order, *even* if used on localhost. Please refer to Glyph's post. Jean-Paul
OoO En ce début d'après-midi nuageux du jeudi 09 octobre 2008, vers 14:30, Jean-Paul Calderone <exarkun@divmod.com> disait :
You can also use a protocol that keeps message boundaries like UDP. UDP is reliable and deliver messages in order if used on localhost.
UDP is *not* reliable and will *not* necessarily deliver messages in order, *even* if used on localhost. Please refer to Glyph's post.
Sorry for the bad information. I have checked again and in fact, this is datagram unix socket that have those properties (on most Unixes). -- BOFH excuse #225: It's those computer people in X {city of world}. They keep stuffing things up.
Nima, The easiest way to do what you want is to put a count at the beginning of your message and read that many bytes. The protocols that Glyph suggested do this. With TCP, you are responsible for keeping boundaries in the message. TCP delivers a reliable two way STREAM of data. It hides all the packets and handshakes from the programmer. Since it is reliable, the easiest way to make messages is to add a count to the beginning of your message. Carl Quoting Jean-Paul Calderone <exarkun@divmod.com>:
On Thu, 09 Oct 2008 13:27:51 +0200, Vincent Bernat <bernat@luffy.cx> wrote:
OoO En cette nuit nuageuse du jeudi 09 octobre 2008, vers 00:51, Nima Ghanavatian <nimag@rogers.com> disait :
I was wondering if there's any way in Twisted/Python to handle packet fragmentation. Can I access the last fragment flag in the TCP packet somehow? Or is there a total packet size sent with the packet automatically? If not, what would be the best way to solve this?
You can also use a protocol that keeps message boundaries like UDP. UDP is reliable and deliver messages in order if used on localhost.
UDP is *not* reliable and will *not* necessarily deliver messages in order, *even* if used on localhost. Please refer to Glyph's post.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Carl Zmola -
glyph＠divmod.com -
Jean-Paul Calderone -
Nima Ghanavatian -
Vincent Bernat