[Twisted-Python] spawning tasks after reactor?
![](https://secure.gravatar.com/avatar/41c1ef3ff18edc7ed831ac32f7d6cd42.jpg?s=120&d=mm&r=g)
Hi, I'm new to twisted and have a simple question. I want to run a threaded server that monitors a remote service and spawns queued/threaded tasks as it finds messages. I'm having trouble understanding how to do this after a reactor is running since the tasks are not set up before reactor.run(). I see reactor.callLater(). But I want to spawn the task immediately. I tried: reactor.callLater(0,self.some_method,object) but the method is not immediately invoked. It seems its not invoked at all. I want to create a high-performance threaded scheduler using twist. Any tips? thank you! Darren
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 31 Dec 2008 09:37:26 -0500, Darren Govoni <darren@ontrenet.com> wrote:
Queued/threaded tasks? That do what?
The way to do things after the reactor is running is the same as the way to do things before the reactor is running. Just do them. There's no difference. If you're having trouble figuring out where to put the code that does them, think about when you want the tasks done. You said you want to do things when you "find messages". How do you find messages?
This will call `self.some_method(object)´ soon. If the call is never happening, then you've done something wrong. Perhaps you didn't start the reactor, or perhaps the signature of some_method is incompatible with that invocation so that when the call was attempted, it just raised a TypeError. It's hard to say exactly what's wrong without seeing a complete, self-contained, runnable example which demonstrates the behavior.
I want to create a high-performance threaded scheduler using twist. Any tips?
Think about ways to not use threads. Anything you can do without threads will be higher performance than the threaded equivalent. Jean-Paul
![](https://secure.gravatar.com/avatar/41c1ef3ff18edc7ed831ac32f7d6cd42.jpg?s=120&d=mm&r=g)
Hi Jean-Paul, Thank you for those suggestions. I can post some simple code after I get further. The psuedo code of what I want to do is like this: every few seconds, check a message queue (I see how this is done): if there is a message, spawn a deferred so I don't block: # The deferred should immediately call a method in a separate # thread (wisted kind) to process the message # Allow for 10 or so deferreds to be processing messages # in parallel while the reactor continues to receive them # but block scheduling the deferred if the pool is maxed # out Sorry if this is lacking in detail, i'm trying to test the idea in a simple way with twisted first. If I set up a deferred before calling reactor.run(), then of course, it runs, but after reactor.run() I have to invoke reactor.callLater(...) so its a bit different, but what I'd like is something like reactor.callNow(...) that doesn't block and invokes the generator in a thread pool. Thanks for any tips! Awesome package. Darren On Wed, 2008-12-31 at 10:17 -0500, Jean-Paul Calderone wrote:
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 31 Dec 2008 10:51:38 -0500, Darren Govoni <darren@ontrenet.com> wrote:
A Deferred is just a convenient way to keep track of callbacks. It's not an active thing, so to "spawn" one doesn't make much sense. Instead, you spawn something else and use a Deferred to keep track of its result.
You probably want twisted.internet.threads.deferToThreadPool (or the older deferToThread). From its docstring: Call the function C{f} using a thread from the given threadpool and return the result as a Deferred. Jean-Paul
![](https://secure.gravatar.com/avatar/41c1ef3ff18edc7ed831ac32f7d6cd42.jpg?s=120&d=mm&r=g)
Hi, Thanks for that suggestion. It looks very close to what I need. I tried doing it _after_ the reactor is run and it did not execute. Here is a code snippet. def convert_to_pdf(arg): print "converting..." def notify_pdf: print "Yay! I'm done" ... d = threads.deferToThread(convert_to_pdf,offset) d.addCallback(notify_pdf) All the examples for threads.deferToThread call reactor.run() after configuring all the threads, but this isn't useful in my app. If I call reactor.run() and _then_ try to deferToThread, it doesn't work. I'm probably missing an argument or something. Thanks for any tips. Much appreciated. Darren On Wed, 2008-12-31 at 11:16 -0500, Jean-Paul Calderone wrote:
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 31 Dec 2008 09:37:26 -0500, Darren Govoni <darren@ontrenet.com> wrote:
Queued/threaded tasks? That do what?
The way to do things after the reactor is running is the same as the way to do things before the reactor is running. Just do them. There's no difference. If you're having trouble figuring out where to put the code that does them, think about when you want the tasks done. You said you want to do things when you "find messages". How do you find messages?
This will call `self.some_method(object)´ soon. If the call is never happening, then you've done something wrong. Perhaps you didn't start the reactor, or perhaps the signature of some_method is incompatible with that invocation so that when the call was attempted, it just raised a TypeError. It's hard to say exactly what's wrong without seeing a complete, self-contained, runnable example which demonstrates the behavior.
I want to create a high-performance threaded scheduler using twist. Any tips?
Think about ways to not use threads. Anything you can do without threads will be higher performance than the threaded equivalent. Jean-Paul
![](https://secure.gravatar.com/avatar/41c1ef3ff18edc7ed831ac32f7d6cd42.jpg?s=120&d=mm&r=g)
Hi Jean-Paul, Thank you for those suggestions. I can post some simple code after I get further. The psuedo code of what I want to do is like this: every few seconds, check a message queue (I see how this is done): if there is a message, spawn a deferred so I don't block: # The deferred should immediately call a method in a separate # thread (wisted kind) to process the message # Allow for 10 or so deferreds to be processing messages # in parallel while the reactor continues to receive them # but block scheduling the deferred if the pool is maxed # out Sorry if this is lacking in detail, i'm trying to test the idea in a simple way with twisted first. If I set up a deferred before calling reactor.run(), then of course, it runs, but after reactor.run() I have to invoke reactor.callLater(...) so its a bit different, but what I'd like is something like reactor.callNow(...) that doesn't block and invokes the generator in a thread pool. Thanks for any tips! Awesome package. Darren On Wed, 2008-12-31 at 10:17 -0500, Jean-Paul Calderone wrote:
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 31 Dec 2008 10:51:38 -0500, Darren Govoni <darren@ontrenet.com> wrote:
A Deferred is just a convenient way to keep track of callbacks. It's not an active thing, so to "spawn" one doesn't make much sense. Instead, you spawn something else and use a Deferred to keep track of its result.
You probably want twisted.internet.threads.deferToThreadPool (or the older deferToThread). From its docstring: Call the function C{f} using a thread from the given threadpool and return the result as a Deferred. Jean-Paul
![](https://secure.gravatar.com/avatar/41c1ef3ff18edc7ed831ac32f7d6cd42.jpg?s=120&d=mm&r=g)
Hi, Thanks for that suggestion. It looks very close to what I need. I tried doing it _after_ the reactor is run and it did not execute. Here is a code snippet. def convert_to_pdf(arg): print "converting..." def notify_pdf: print "Yay! I'm done" ... d = threads.deferToThread(convert_to_pdf,offset) d.addCallback(notify_pdf) All the examples for threads.deferToThread call reactor.run() after configuring all the threads, but this isn't useful in my app. If I call reactor.run() and _then_ try to deferToThread, it doesn't work. I'm probably missing an argument or something. Thanks for any tips. Much appreciated. Darren On Wed, 2008-12-31 at 11:16 -0500, Jean-Paul Calderone wrote:
participants (2)
-
Darren Govoni
-
Jean-Paul Calderone