Re: [Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests
Thanks for quick and great help, Terry and Jean-Paul!
You're welcome. If in any doubt about conflicting advice, do what JP suggests. The reason I pointed you to JP's Cooperator solution was that when you extract the set of links from a page that you also want to fetch, you could make another generator and submit that to the same cooperator's coiterate function. That would just schedule the additional getPage calls to be run. The only wrinkle (I think) in that is that in doing the very last getPage, the cooperator will run out of more work and fire the deferred you got originally. But the final page may contain more links. I'm not sure if I'm thinking clearly about that final bit. That's your job :-) Terry
On 01:59 pm, terry@jon.es wrote:
Thanks for quick and great help, Terry and Jean-Paul!
You're welcome. If in any doubt about conflicting advice, do what JP suggests.
The reason I pointed you to JP's Cooperator solution was that when you extract the set of links from a page that you also want to fetch, you could make another generator and submit that to the same cooperator's coiterate function. That would just schedule the additional getPage calls to be run.
The only wrinkle (I think) in that is that in doing the very last getPage, the cooperator will run out of more work and fire the deferred you got originally. But the final page may contain more links. I'm not sure if I'm thinking clearly about that final bit. That's your job :-)
cooperator and DeferredSemaphore are probably both good solutions. There's not much reason to prefer one over the other. :) Jean-Paul
exarkun@twistedmatrix.com wrote:
On 01:59 pm, terry@jon.es wrote:
Thanks for quick and great help, Terry and Jean-Paul! You're welcome. If in any doubt about conflicting advice, do what JP suggests.
The reason I pointed you to JP's Cooperator solution was that when you extract the set of links from a page that you also want to fetch, you could make another generator and submit that to the same cooperator's coiterate function. That would just schedule the additional getPage calls to be run.
The only wrinkle (I think) in that is that in doing the very last getPage, the cooperator will run out of more work and fire the deferred you got originally. But the final page may contain more links. I'm not sure if I'm thinking clearly about that final bit. That's your job :-)
cooperator and DeferredSemaphore are probably both good solutions. There's not much reason to prefer one over the other. :)
Jean-Paul
That list seems like far from the most active one, but the one filled with friendly people :) The thing I like when dealing with Twisted and it's documentation is that it is seen often that people who make it, and who use it, really love Twisted.
... can.. not.. reist.. posting... http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough .html :-) -- M
participants (4)
-
exarkun＠twistedmatrix.com -
Igor Katson -
Michał Pasternak -
Terry Jones