[Twisted-Python] Threading and twisted

I have some old server code that runs in multiple threads, and I'm in the process of converting the server to a twisted xmlrpc server. The code basically has multiple worker threads all pulling from a thread-safe queue of requests to process. Threads pull requests at will from the queue and process them, placing results in a thread-safe object which is used to deliver results to clients. The request processing will often block doing I/O or sleep while waiting for processes to complete, which is why they run in threads. My question is: can I keep the threaded system as-is and just use the xmlrpc system to add requests to the queue and pull results from it to deliver to clients, as long as the xmlrpc calls maintain thread safety when dealing with the shared queue and results object? Or, is it generally unsafe to use non-twisted threading in conjunction with a twisted app? Should I be using twisted's own thread pool and runInThread() system to do the request processing work? It probably won't be too much work to change over the current code to use twisted's threading framework...but I'm just wondering where I might run into trouble if I use the current setup instead. Am I endangering the xmlrpc server's ability to respond to requests if I have threads that aren't under twisted's control? Thanks for your help, -Dave -- Presenting: mediocre nebula.

On Mon, 2006-02-06 at 13:58 -0800, David Hirschfield wrote:
My question is: can I keep the threaded system as-is and just use the xmlrpc system to add requests to the queue and pull results from it to deliver to clients, as long as the xmlrpc calls maintain thread safety when dealing with the shared queue and results object? Or, is it generally unsafe to use non-twisted threading in conjunction with a twisted app? Should I be using twisted's own thread pool and runInThread() system to do the request processing work?
Twisted has a built-in thread pool. Lets say you have a function called runTask(t) which is supposed to run on a specific task, and is known to be thread-safe (e.g. doesn't call into Twisted). You can then do: def xmlrpc_runtask(self, t): return twisted.internet.threads.deferToThread(runTask, t) See http://twistedmatrix.com/projects/core/documentation/howto/threading.html You can do something similar with your own thread pool. The key is that threads can schedule callables to run in Twisted thread using reactor.callFromThread(). You don't want to do anything blocking (e.g. reading from Queue) in the Twisted thread.
participants (2)
-
David Hirschfield
-
Itamar Shtull-Trauring