Slow network?

[Chris Mellon]

On Mon, Jan 12, 2009 at 1:13 PM, Laszlo Nagy <gandalf at shopzeus.com> wrote:
>
>  Hi All,
>
> To make the long story short, I have a toy version of an ORB being
> developed, and the biggest problem is slow network speed over TCP/IP.
>
> There is an object called 'endpoint' on both sides, with incoming and
> outgoing message queues. This endpoint object has a socket assigned, with
> nodelay:
>
> conn.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
>
> The endpoint is running two separate threads - those are dedicated for
> reading/writing messages from/into the socket object, as shown below:
>
>
>   def _process_incoming(self):
>       try:
>           while not self.stop_requested.isSet():
>               data = self.read_str()
>               while not self.stop_requested.isSet():
>                   try:
>                       self.incoming.put(data,1)
>                       break
>                   except orb.util.smartqueue.Full:
>                       pass
>               if not self.stop_requested.isSet():
>                   if self.router:
>                       self.router.on_message_arrived(self)
>       except Exception, e:
>           if self.router:
>               if not isinstance(e,TransportClosedError):
>                   self.router.logger.error(dumpexc(e))
>               self.router.unregister_endpoint(self)
>           self.shutdown()
>           raise SystemExit(0)
>
>   def _process_outgoing(self):
>       try:
>           while not self.stop_requested.isSet():
>               data_ok = False
>               while not self.stop_requested.isSet():
>                   try:
>                       data = self.outgoing.get(1)
>                       data_ok = True
>                       break
>                   except orb.util.smartqueue.Empty:
>                       pass
>               if data_ok:
>                   self.write_str(data)
>       except Exception, e:
>           if self.router:
>               if not isinstance(e,TransportClosedError):
>                   self.router.logger.error(dumpexc(e))
>               self.router.unregister_endpoint(self)
>           self.shutdown()
>           raise SystemExit(0)
>
>
> The main point is that the sender does not need to wait for the message to
> be actually written into the socket (unless the outgoing queue becomes
> full).
>
> When I try to send a message and receive an answer for it, I can only get as
> much as 130 request+response message pairs per second. Apparently, it is the
> same from messages size =77bytes to message size=16 Kbytes.
>
> However, if I send 100 outgoing messages first, then read back all answers
> then the speed goes up to 1300 message pairs/sec. I suspect that this has
> something to do with TCP/IP. Since this will be used for RPC/RMI, it would
> be very important to lower the time needed to exchange messages. Is there
> any way I can speed this up?
>

It is very likely that nodelay is actually hurting you here.

Using the select module and doing non-blocking IO will be faster than
using threads for this as well.