On Tue, May 23, 2006 at 09:49:23PM +0100, Paul Moore wrote: [...]
The problem is that in rare cases, the blocking call can block indefinitely. In this case, I need to make the call time out. However, the underlying API offers no way to time out the call, so I have to do this externally.
There is a problem here Twisted cannot help you with -- you will leak threads that you cannot interrupt, because they are blocked indefinitely. setTimeout or other solutions will allow you to paper over this fact, but you need to be aware of it, because it will eventually stop your process from running. There's no way something like defer.setTimeout can magically cancel the underlying operation for you (even though it unblocks the deferred waiting on that operation), and part of the reason setTimeout is a deprecated is to force people to be aware of that. Also, deferToThread by default isn't going to give you with 50-100 parallel threads. The default threadpool size is 4. You'll need to use reactor.suggestThreadPoolSize to change that -- but realise that the default DNS resolver in current Twisted releases use that threadpool too, and perhaps so will other libraries, and you could starve those callers by swamping the threadpool with your blocking calls. So it may be better to use your own threadpool, as twisted.enterprise.adbapi does, for example. See twisted.python.threadpool. I don't suppose there's a non-blocking way to do what you want? Oh -- and remember that while you can't kill threads, you *can* kill processes. Consider using subprocesses to do your blocking work. -Andrew.