Re : [Twisted-Python] Multiple thread pools
Hi, I can see now how 'untwisted' my threads concerns are, please forget my newbie ununderstanding ;-) Still, I need to ensure that I won't have more than x similar calls at a time. Is there a way, maybe using some of the deferred classes (deferredqueue,..) to ensure that there is no more than x simultaneous similar calls (wrapped in deferreds), the others being pending, until one is finished. My question might be stupid, but I'm quite new to twisted and the way to implement this behaviour is not clear to me (even if I found great help in the o'reilly book and the docs). Thanks, Stéphane ----- Message d'origine ---- De : Stéphane Brault <stephane_brault@yahoo.fr> À : twisted-python@twistedmatrix.com Envoyé le : Mercredi, 10 Mai 2006, 12h27mn 16s Objet : [Twisted-Python] Multiple thread pools Hi, I want to use twisted to run a server that is also a SOA client: I developp a scheduler that schedule calls to a web service regularly, tasks may be added or terminated. The web service has different calls and each task calls a specific call (too many calls in that sentence ;-)) and then update a database. The fact is that I can only have a maximum number of thread per call, I was wondering if I could have a thread pool per type of call, feeding it with my scheduled tasks and being able to change its size dynamically. I was thinking of having different groups of tasks, depending of the specific call, and feeding my thread pool according to the schedule. I'm new to twisted, so I don't really know how to do that, the thread documentation is a bit light for me. I'm also not quite sure what type of reactor would be best to use. I hope I've been clear, thanks, Stéphane _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 11 May 2006 15:32:08 +0000 (GMT), Stéphane Brault <stephane_brault@yahoo.fr> wrote:
Hi, I can see now how 'untwisted' my threads concerns are, please forget my newbie ununderstanding ;-) Still, I need to ensure that I won't have more than x similar calls at a time. Is there a way, maybe using some of the deferred classes (deferredqueue,..) to ensure that there is no more than x simultaneous similar calls (wrapped in deferreds), the others being pending, until one is finished
I think twisted.internet.defer.DeferredSemaphore is what you're looking for. There is an example (someone else's example) of using it to do something similar here: http://pastebin.com/702986 Moe
Thanks Moe, the example seems to work, which is nice ;-) If I understand well DeferredSemaphore(x) guarantees that no more than x concurent calls that acquired it will be run together. Am I wrong when inferring that DeferredList is not compulsory ? And that I adding calls to the semaphore as I need them will ensure the behaviour I want (as my tasks come, acquire the semaphore, then release it) ? Thaks again, Stéphane ----- Message d'origine ---- De : Moe Aboulkheir <moe@divmod.com> À : twisted-python@twistedmatrix.com Envoyé le : Jeudi, 11 Mai 2006, 5h56mn 52s Objet : Re: Re : [Twisted-Python] Multiple thread pools On Thu, 11 May 2006 15:32:08 +0000 (GMT), Stéphane Brault <stephane_brault@yahoo.fr> wrote:
Hi, I can see now how 'untwisted' my threads concerns are, please forget my newbie ununderstanding ;-) Still, I need to ensure that I won't have more than x similar calls at a time. Is there a way, maybe using some of the deferred classes (deferredqueue,..) to ensure that there is no more than x simultaneous similar calls (wrapped in deferreds), the others being pending, until one is finished
I think twisted.internet.defer.DeferredSemaphore is what you're looking for. There is an example (someone else's example) of using it to do something similar here: http://pastebin.com/702986 Moe _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 11 May 2006 17:43:05 +0000 (GMT), Stéphane Brault <stephane_brault@yahoo.fr> wrote:
Am I wrong when inferring that DeferredList is not compulsory ?
It depends whether or not you want to know when all of the tasks have completed, which is what the example uses it for. Moe
Thanks Moe, that's what I inferred, I don't need to know when all tasks have completed. I only need to run tasks as they arrive, whether scheduled or called by a user and guarantee that no more than a certain number are running simultineaously. Stéphane ----- Message d'origine ---- De : Moe Aboulkheir <moe@divmod.com> À : Twisted general discussion <twisted-python@twistedmatrix.com> Envoyé le : Vendredi, 12 Mai 2006, 11h13mn 33s Objet : Re: Re : Re : [Twisted-Python] Multiple thread pools On Thu, 11 May 2006 17:43:05 +0000 (GMT), Stéphane Brault <stephane_brault@yahoo.fr> wrote:
Am I wrong when inferring that DeferredList is not compulsory ?
It depends whether or not you want to know when all of the tasks have completed, which is what the example uses it for. Moe _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Still, I need to ensure that I won't have more than x similar calls at a time. Is there a way, maybe using some of the deferred classes (deferredqueue,..) to ensure that there is no more than x simultaneous similar calls (wrapped in deferreds), the others being pending, until one is finished.
You are on the right track. A couple possible approaches are in this mailing list thread: http://twistedmatrix.com/pipermail/twisted-python/2006-January/012388.html -- Nicola Larosa - http://www.tekNico.net/ Much as everyone thinks they want financial security, the happiest people are not those who have it, but those who like what they do. -- Paul Graham, January 2006
Thanks Nicola, I'm investigating this further. Stéphane ----- Message d'origine ---- De : Nicola Larosa <nico@tekNico.net> À : twisted-python@twistedmatrix.com Envoyé le : Jeudi, 11 Mai 2006, 6h00mn 44s Objet : [Twisted-Python] Re: Re : Multiple thread pools
Still, I need to ensure that I won't have more than x similar calls at a time. Is there a way, maybe using some of the deferred classes (deferredqueue,..) to ensure that there is no more than x simultaneous similar calls (wrapped in deferreds), the others being pending, until one is finished.
You are on the right track. A couple possible approaches are in this mailing list thread: http://twistedmatrix.com/pipermail/twisted-python/2006-January/012388.html -- Nicola Larosa - http://www.tekNico.net/ Much as everyone thinks they want financial security, the happiest people are not those who have it, but those who like what they do. -- Paul Graham, January 2006 _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Moe Aboulkheir -
Nicola Larosa -
Stéphane Brault