Slow Queue.queue? (was: slow network)

bieffe62 at gmail.com bieffe62 at gmail.com
Thu Jan 15 12:14:26 CET 2009


On 15 Gen, 10:22, Laszlo Nagy <gand... at shopzeus.com> wrote:
> > then the speed goes up to 64 messages/sec on windows and 500
> > messages/sec on Linux.
>
> Finally I could reach 1500 messages/sec without using the queue. If I
> comment out one line (use the queue instead of direct write into socket)
> then speed decreases to 40-60 messages/sec. I don't understand why the
> slow version is slower by a factor of 40?
>
> Fast version:
>
>     def send_message(self,sender,recipient,msgtype,body,timeout=3600):
>
> self.write_str(self.serializer.serialize([sender,recipient,msgtype,body]))
>
> Slow version:
>
>     def send_message(self,sender,recipient,msgtype,body,timeout=3600):
>         self.outgoing.put(self.serializer.serialize([
>             sender,recipient,msgtype,body
>         ]),1,timeout)
>
> plus this method, executed in a different thread:
>
>     def _process_outgoing(self):
>         try:
>             while not self.stop_requested.isSet():
>                 data_ok = False
>                 while not self.stop_requested.isSet():
>                     try:
>                         data = self.outgoing.get(1,1)
>                         data_ok = True
>                         break
>                     except Queue.Empty:
>                         pass
>                 if data_ok:
>                     self.write_str(data)
>         except Exception, e:
>             if self.router:
>                 if not isinstance(e,TransportClosedError):
>                     self.router.logger.error(dumpexc(e))
>                 self.router.unregister_endpoint(self)
>             self.shutdown()
>             raise SystemExit(0)


I would try something like this inside _process_outgoing:

            while not self.stop_requested.isSet():
                data_ok = False
                while not self.stop_requested.isSet():
                    if not self.outgoing.empty():
                      try:
                        data = self.outgoing.get(True,0.1)
                        data_ok = True
                        break
                      except Queue.Empty:
                        pass
                    else:
                         time.sleep(0.1) # maybe, if the thread usess
too much CPU

The hypotesis I made for this suggestion are:

- if the queue is found empty, the queue.get could keep the global
interpreter lock until a message arrive
  (blocking also the thread that put the message in the queue)

- if the queue is found empty, the handling of the exception can slows
down the execution

Not sure they are good guesses, because at the rate of your messages
the queue should be almost always full,
and most probably the implementation of Queue.get is smarted than
myself :-). ANyway, it is worth a try ...

Also, is you are using fixed-length queues (the ones that make the
sender wait if the queue is full), try to increase
the queue size, or to use an infinite-size queue.

HTH

Ciao
----
FB



More information about the Python-list mailing list