Re: [Twisted-Python] Has anybody integrated multiple threadpools into reactor?
Hello, thanks for the hint. Glyph Lefkowitz wrote:
Could you elaborate a bit on that? I have the situation that I have to wrap two C-libs which are essentially wrappers around proprietary protocols. They use blocking networked calls so that I'll want to use threads to make them play nicely with my twisted app. Now what I'd like to have is two distinct thread(s/ pools) for them. One of them would have a threadpool size of 1 (one) because the C-lib isn't threadsafe and I want implicit serialization without being forced to synchronize explicitly. My plan was to use a single thread (no pool) which blocks on a queue 'till it gets request from the reactor thread. When it has accomplished its duties it will use callFromThread to get the results back to the reactor threads. Could you tell me where the pitfalls are you mentioned above in this scenario? TIA, aa
On Thu, Oct 21, 2004 at 11:45:39AM +0200, Ames Andreas (MPA/DF) wrote: [...]
Now what I'd like to have is two distinct thread(s/ pools) for them. One of them would have a threadpool size of 1 (one) because the C-lib
If you're using a threadpool of size one, why use a thread pool at all? Just start a thread. You don't need (or want) any of the fancy worker thread management that a thread pool is designed for. There's nothing in Twisted that requires all your threads run in a Twisted threadpool; they're just provided for convenience (mainly the convenience of deferToThread). If all you need is a single thread, then t = threading.Thread(target=foo) works great. reactor.callFromThread works from any thread. In a similar vein, there are times when I've found it simpler to start a thread for database interactions, rather than use adbapi. The reason is much the same: the "pool of workers" model didn't fit what I was trying to do. -Andrew.
On Thu, Oct 21, 2004 at 01:35:43PM -0400, Glyph Lefkowitz wrote:
When other things are using the threadpool. It's possible that e.g. 20 other long-running callInThreads have been made. With the default threadpool settings, this means this foo wouldn't get run for a long time. Of course, if you have 20 threads, you may have other issues ;) Oh, and using threading.Thread directly allows you to make that thread daemonic, if you want. So they're practically the same. :) -Andrew.
On Thu, Oct 21, 2004 at 11:45:39AM +0200, Ames Andreas (MPA/DF) wrote: [...]
Now what I'd like to have is two distinct thread(s/ pools) for them. One of them would have a threadpool size of 1 (one) because the C-lib
If you're using a threadpool of size one, why use a thread pool at all? Just start a thread. You don't need (or want) any of the fancy worker thread management that a thread pool is designed for. There's nothing in Twisted that requires all your threads run in a Twisted threadpool; they're just provided for convenience (mainly the convenience of deferToThread). If all you need is a single thread, then t = threading.Thread(target=foo) works great. reactor.callFromThread works from any thread. In a similar vein, there are times when I've found it simpler to start a thread for database interactions, rather than use adbapi. The reason is much the same: the "pool of workers" model didn't fit what I was trying to do. -Andrew.
On Thu, Oct 21, 2004 at 01:35:43PM -0400, Glyph Lefkowitz wrote:
When other things are using the threadpool. It's possible that e.g. 20 other long-running callInThreads have been made. With the default threadpool settings, this means this foo wouldn't get run for a long time. Of course, if you have 20 threads, you may have other issues ;) Oh, and using threading.Thread directly allows you to make that thread daemonic, if you want. So they're practically the same. :) -Andrew.
participants (3)
-
Ames Andreas (MPA/DF) -
Andrew Bennetts -
Glyph Lefkowitz