Allen Bierbaum wrote:
- Has anyone extended threadpool to allow code to remove queued functions from the pending queue?
How about just having the functions themselves check to see if they really should run their intended task? (Being careful about thread-safety, of course).
Thanks for the idea. I will give it a try.
You also might want to take a look at my taskqueue sub-package in Twisted-Goodies, http://foss.eepatents.com/Twisted-Goodies. One feature that might come in handy is the IWorker implementors' ability to resign when they feel that they are no longer able to run jobs. Any jobs that the resigning worker has queued up individually at that point are returned to the main queue for dispatching to other workers, current or future. The whole thing revolves around a priority queue, which might be useful to you, too. Also, a ProcessWorker implementation of IWorker is pending that spawns a separate Python interpreter to run tasks. If the ProcessWorker hangs, you'll be able to just kill it and move right along, without hanging your main Python interpreter.
I am using threads because the calls I am making are into a non-blocking library. I may see about rewriting it in the future to be more twisted-friendly, but for now I have to live with it.
That was my original motivation for writing taskqueue, with its ThreadWorker implementation of IWorker. It's used in sAsync, which makes a wonderful but totally synchronous database API (SQLAlchemy) play very nicely (IMHO) with Twisted. Debian packages are in the works for both, by the way. Best regards, Ed Suominen