[Twisted-Python] Something like a ThreadPool
Is there any sort of job queue or similar , besides AsynQueue, which looked like exactly what i wanted, but is sadly undocumented, native to twisted? What is the twisted way to do such a thing? Is there something already available? -- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin@lazyweb.ca
On Wed, 2008-06-18 at 04:13 -0400, kevin beckford wrote:
Is there any sort of job queue or similar , besides AsynQueue, which looked like exactly what i wanted, but is sadly undocumented, native to twisted? What is the twisted way to do such a thing? Is there something already available?
What kind of work are you queuing? If it's e.g. fetching a web page, you can just use do it using Twisted's scalable asynchronous APIs (maybe using a DeferredSemaphore if you want to limit number of conccurent queries.) If it's a blocking API you're calling (RDBMS queries are the typical example) then Twisted has a thread pool, as well as RDBMS-specific wrapper - see our documentation.
What kind of work are you queuing? If it's e.g. fetching a web page, you can just use do it using Twisted's scalable asynchronous APIs (maybe using a DeferredSemaphore if you want to limit number of conccurent queries.) If it's a blocking API you're calling (RDBMS queries are the typical example) then Twisted has a thread pool, as well as RDBMS-specific wrapper - see our documentation.
Thanks very much. Found and read.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin@lazyweb.ca
For the benefit of future searchers (and myself), would either of you mind posting a link to specific place in the docs you are talking about? Kevin Horn On Wed, Jun 18, 2008 at 8:04 AM, kevin beckford <kevin@lazyweb.ca> wrote:
What kind of work are you queuing? If it's e.g. fetching a web page, you can just use do it using Twisted's scalable asynchronous APIs (maybe using a DeferredSemaphore if you want to limit number of conccurent queries.) If it's a blocking API you're calling (RDBMS queries are the typical example) then Twisted has a thread pool, as well as RDBMS-specific wrapper - see our documentation.
Thanks very much. Found and read.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin@lazyweb.ca
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, Jun 18, 2008 at 12:21 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
For the benefit of future searchers (and myself), would either of you mind posting a link to specific place in the docs you are talking
http://www.cs.lth.se/EDA046/assignments/assignment4/twisted/threading.html
kevin beckford wrote:
Is there any sort of job queue or similar , besides AsynQueue, which looked like exactly what i wanted, but is sadly undocumented, native to twisted? What is the twisted way to do such a thing? Is there something already available?
Kevin, I have to confess that you're right about the "sadly undocumented" part. I keep promising myself (and others) that I will rectify the situation, but the motivation is just not as high as I'd like. I think that's true of a lot of free software; the author write the stuff to scratch his particular itch and then moves on. As the originator, he of course needs no documentation, and writing code is much more fun than writing words. The only thing I can offer in substitute at the moment are some really extensive docstrings in the code itself, and the usage examples embodied in the sAsync and AsynCluster packages, which itself are built on AsynQueue. Also, see a usage example for AsynCluster at http://foss.eepatents.com/trac/AsynCluster/wiki/Example_AsynCluster. Best regards, Ed
usage examples embodied in the sAsync and AsynCluster packages, which itself are built on AsynQueue. Also, see a usage example for AsynCluster at http://foss.eepatents.com/trac/AsynCluster/wiki/Example_AsynCluster.
Acutally Ed, this will help. I'd like to offer to help you with the documentation if you would like. Tell me what it does and I'll write it up or something. I'd love to increase my knowledge of this code by documenting some examples. Seriously. -- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin@lazyweb.ca
participants (4)
-
Ed Suominen -
Itamar Shtull-Trauring -
kevin beckford -
Kevin Horn