[Twisted-Python] twisted TCP frame size
![](https://secure.gravatar.com/avatar/710bb35b7b65f97fa69b38c578b38247.jpg?s=120&d=mm&r=g)
Hi, I've recently written a threaded client server application using a custom message protocol. Each message is 49 bytes long and needs to arrive at the client as soon as possible. In the current app I send() each message as soon as all 49 bytes are available, I also recv(49) bytes at the client side (buffering as necessary). The threaded application thus always reads 49 bytes from the network buffers as soon as it is possible to do so. This application consumes about 9% cpu time at full throttle. Because I already use the twisted.enterprise adbapi on the client side I decided to get rid of all the client threads and use twisted for the TCP stuff as well. I've written a small test script to determine basic loads and performance: ----------------- #! /usr/bin/env python import psyco psyco.full() import array import sys from twisted.internet.protocol import Protocol,ReconnectingClientFactory from twisted.internet import reactor f = open("wakka", "w") class TRAUReceiver(Protocol): def dataReceived(self, data): print len(data) f.write(data) # f.flush() def connectionMade(self): """ * connection were made, send signon""" print "Signing on...", signon = array.array('B', [ord('T'), 1 ,255, 1, 255]) self.transport.write(signon.tostring()) print "Done" class TRAUClientFactory(ReconnectingClientFactory): def startedConnecting(self, connector): print 'Started to connect.' def buildProtocol(self, addr): print 'Connected.' print 'Resetting reconnection delay' self.resetDelay() return TRAUReceiver() def clientConnectionLost(self, connector, reason): print 'Lost connection. Reason:', reason ReconnectingClientFactory.clientConnectionLost(self, connector, reason) def clientConnectionFailed(self, connector, reason): print 'Connection failed. Reason:', reason ReconnectingClientFactory.clientConnectionFailed(self, connector, reason) reactor.connectTCP('localhost', 55555, TRAUClientFactory()) reactor.run() ------------------ Connecting the above to the server yielded somewhat surprising results: The length of the dataReceived() data between runs varies between 49 and multiples of 49 bytes. I understand this (I think) as the ethernet packet length sweet spot is about 1.3kB and Protocol is propably optimized arround that. This does imply that I need a more elaborate frame caching scheme on the client side than the one I currently have, as another socket connection signals if messages from the current connection must be stored or discarded. Surprisingly, the script data length is reported as continuous 49 bytes at leas 4/10 times it's run, on other runs the print len(data) line looks something like this: 49 12691 49 7938 49 Each second dataReceived() callback is 49 bytes, the rest multiples of 49. When the script reports all frames as 49; cpu consumption is ~33%, when staggered it's ~4%. The dataReceived() call seems overly expensive when compared with the 'raw' synchronous recv(). Paradoxically, padding the 49 byte message with 1000 pad bytes improves the script's performance 8 times; due to the decrease in dataReceived() calls probably. So, 1) is there a way to force dataReceived() to return when a certain data length has been received ? 2) Why is dataReceived() so expensive (if it is) ? 3) Is Protocol the correct tree or are there other ways to handle small time sensitive messages in twisted. Apologies for the longish post. Regards Thys
![](https://secure.gravatar.com/avatar/426d6dbf6554a9b3fca1fd04e6b75f38.jpg?s=120&d=mm&r=g)
Thys Meintjes wrote:
1) is there a way to force dataReceived() to return when a certain data length has been received ?
dataReceived() is called via doRead() on the file descriptor, which is called via a select([fd,], [], []) event in the default reactor. If you are doing something (executing code) and a TCP segment comes in, then another, then another, *then* you exit your code and the reactor falls back to select(), it'll return with the fd and doRead() will be called, and you'll get everything from the socket - 3 frames worth. The correct question is, given your test script has a very small dataReceived method that returns quickly (I presume), why isn't the reactor "beating" the server and getting 49 bytes every time? ...to which I don't have a good answer. Which version of Twisted are you running?
2) Why is dataReceived() so expensive (if it is) ?
Between select() returning and dataReceived() being called, the reactor does lots of stuff related to running callLater's and stuff. I've found that's all quite expensive in earlier versions of Twisted, but it doesn't look like you use any of that, so I can't imagine that's it.
3) Is Protocol the correct tree or are there other ways to handle small time sensitive messages in twisted.
Is there a reason you can't use UDP?
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 27 Feb 2006 16:44:18 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Of course, it isn't necessarily the case that multiple TCP packets, each with 49 bytes of application-level data, are arriving between the times doRead()/dataReceived() get invoked. Nagling on the server could cause data to arrive in multiples of 49 bytes just as easily as a slow receiver. This could happen with the synchronous version too, you just wouldn't be able to notice it. What Twisted does in this case is actually more efficient, since it involves fewer trips into recv(), less memory copying, etc. Of course all the other stuff Twisted does might outweigh this (and likely does - the overhead associated with synchronously reading from one socket is much lower than the overhead of performing the same task asynchronously, at least when using select()). Which is not to say that the Twisted version has to be slower, but you might have to work harder to see any benefits at this (essentially non-existent) level of concurrency. Jean-Paul
![](https://secure.gravatar.com/avatar/426d6dbf6554a9b3fca1fd04e6b75f38.jpg?s=120&d=mm&r=g)
Thys Meintjes wrote:
1) is there a way to force dataReceived() to return when a certain data length has been received ?
dataReceived() is called via doRead() on the file descriptor, which is called via a select([fd,], [], []) event in the default reactor. If you are doing something (executing code) and a TCP segment comes in, then another, then another, *then* you exit your code and the reactor falls back to select(), it'll return with the fd and doRead() will be called, and you'll get everything from the socket - 3 frames worth. The correct question is, given your test script has a very small dataReceived method that returns quickly (I presume), why isn't the reactor "beating" the server and getting 49 bytes every time? ...to which I don't have a good answer. Which version of Twisted are you running?
2) Why is dataReceived() so expensive (if it is) ?
Between select() returning and dataReceived() being called, the reactor does lots of stuff related to running callLater's and stuff. I've found that's all quite expensive in earlier versions of Twisted, but it doesn't look like you use any of that, so I can't imagine that's it.
3) Is Protocol the correct tree or are there other ways to handle small time sensitive messages in twisted.
Is there a reason you can't use UDP?
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Mon, 27 Feb 2006 16:44:18 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Of course, it isn't necessarily the case that multiple TCP packets, each with 49 bytes of application-level data, are arriving between the times doRead()/dataReceived() get invoked. Nagling on the server could cause data to arrive in multiples of 49 bytes just as easily as a slow receiver. This could happen with the synchronous version too, you just wouldn't be able to notice it. What Twisted does in this case is actually more efficient, since it involves fewer trips into recv(), less memory copying, etc. Of course all the other stuff Twisted does might outweigh this (and likely does - the overhead associated with synchronously reading from one socket is much lower than the overhead of performing the same task asynchronously, at least when using select()). Which is not to say that the Twisted version has to be slower, but you might have to work harder to see any benefits at this (essentially non-existent) level of concurrency. Jean-Paul
participants (3)
-
Jean-Paul Calderone
-
Phil Mayers
-
Thys Meintjes