Slow network?

[MRAB]

Chris Mellon wrote:
> On Mon, Jan 12, 2009 at 1:13 PM, Laszlo Nagy <gandalf at shopzeus.com> wrote:
>>  Hi All,
>>
>> To make the long story short, I have a toy version of an ORB being
>> developed, and the biggest problem is slow network speed over TCP/IP.
>>
>> There is an object called 'endpoint' on both sides, with incoming and
>> outgoing message queues. This endpoint object has a socket assigned, with
>> nodelay:
>>
>> conn.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
>>
>> The endpoint is running two separate threads - those are dedicated for
>> reading/writing messages from/into the socket object, as shown below:
>>
>>
>>   def _process_incoming(self):
>>       try:
>>           while not self.stop_requested.isSet():
>>               data = self.read_str()
>>               while not self.stop_requested.isSet():
>>                   try:
>>                       self.incoming.put(data,1)
>>                       break
>>                   except orb.util.smartqueue.Full:
>>                       pass
>>               if not self.stop_requested.isSet():
>>                   if self.router:
>>                       self.router.on_message_arrived(self)
>>       except Exception, e:
>>           if self.router:
>>               if not isinstance(e,TransportClosedError):
>>                   self.router.logger.error(dumpexc(e))
>>               self.router.unregister_endpoint(self)
>>           self.shutdown()
>>           raise SystemExit(0)
>>
>>   def _process_outgoing(self):
>>       try:
>>           while not self.stop_requested.isSet():
>>               data_ok = False
>>               while not self.stop_requested.isSet():
>>                   try:
>>                       data = self.outgoing.get(1)
>>                       data_ok = True
>>                       break
>>                   except orb.util.smartqueue.Empty:
>>                       pass
>>               if data_ok:
>>                   self.write_str(data)
>>       except Exception, e:
>>           if self.router:
>>               if not isinstance(e,TransportClosedError):
>>                   self.router.logger.error(dumpexc(e))
>>               self.router.unregister_endpoint(self)
>>           self.shutdown()
>>           raise SystemExit(0)
>>
>>
>> The main point is that the sender does not need to wait for the message to
>> be actually written into the socket (unless the outgoing queue becomes
>> full).
>>
>> When I try to send a message and receive an answer for it, I can only get as
>> much as 130 request+response message pairs per second. Apparently, it is the
>> same from messages size =77bytes to message size=16 Kbytes.
>>
>> However, if I send 100 outgoing messages first, then read back all answers
>> then the speed goes up to 1300 message pairs/sec. I suspect that this has
>> something to do with TCP/IP. Since this will be used for RPC/RMI, it would
>> be very important to lower the time needed to exchange messages. Is there
>> any way I can speed this up?
>>
> 
> It is very likely that nodelay is actually hurting you here.
> 
> Using the select module and doing non-blocking IO will be faster than
> using threads for this as well.
> 
You might also want to replace those 'pass' statements when smartqueue 
is empty or full with time.sleep() to avoid busy waiting.