On Fri, 27 Jun 2008 11:46:05 -0400, Doug Farrell <dfarrell@mypublisher.com> wrote:

Hi all,

I've built a Twisted server that handles Python logging messages sent from the logging.handlers.SocketHandler system. Right now my protocol implementation handles logging the received logRecord to a file and entering it into an in memory model for statistical purposes. I'm thinking of off-loading this activity to a separate thread by having the protocol just send the logRecord it receives into queue that is serviced by a separate thread. My question is whether using the twisted.internet.threads.deferToThread() method is a good way to kick off this other thread. The other thread will be running for as long as the Twisted server runs so it can handle logRecords in the queue. Would just starting a threading.Thread based class be a better solution, giving access to queue inside the class to the protocol?

deferToThread is based on a thread-pool. If there will be benefit to processing log records concurrently in different threads (ie, if most of the cost is disk I/O or an extension module that releases the GIL) and you have multiple CPUs, using a threadpool might give you a speed up. If any of these things isn't true, then one thread will give you just as much benefit as a whole pool full. Of course, even given all that, using deferToThread is simpler than writing a new Thread with a queue for input, so you may still want to use deferToThread. Jean-Paul