Slow Queue.queue? (was: slow network)

[Laszlo Nagy]

> I would try something like this inside _process_outgoing:
>
>             while not self.stop_requested.isSet():
>                 data_ok = False
>                 while not self.stop_requested.isSet():
>                     if not self.outgoing.empty():
>                       try:
>                         data = self.outgoing.get(True,0.1)
> 			data_ok = True
>                         break
>                       except Queue.Empty:
>                         pass
>                     else:
>                          time.sleep(0.1) # maybe, if the thread usess too much CPU
>   
I tried your modifications. The result is 9 messages/sec. I guess it is 
because time.sleep(0.1) - if that is called after each message, at most 
10 message/sec is possible.
> The hypotesis I made for this suggestion are:
>
> - if the queue is found empty, the queue.get could keep the global
> interpreter lock until a message arrive
>   (blocking also the thread that put the message in the queue)
>   
I hope it is not that way. Consider two threads running intensive 
calculations. Both of them are running python code -> both of them must 
hold the GIL. Python runs the process on one processor, and GIL must be 
released on any thread that is not active on the CPU, regardless of what 
it is doing.
> - if the queue is found empty, the handling of the exception can slows
> down the execution
>   
Because of the timeout specified in my outgoing queue get() call, the 
exception can only happen once per second. It won't slow down anything.
> Not sure they are good guesses, because at the rate of your messages
> the queue should be almost always full,
> and most probably the implementation of Queue.get is smarted than
> myself :-). ANyway, it is worth a try ...
>   
The problem is with sending one message, and getting one answer for it. 
Most of the time, the queue is empty or has only one message inside it.

> Also, is you are using fixed-length queues (the ones that make the
> sender wait if the queue is full), try to increase
> the queue size, or to use an infinite-size queue.
>   
Queue size was 1000 in this example, but it never contains more than two 
messages.