On Wed, 31 Dec 2008 09:37:26 -0500, Darren Govoni wrote:

Hi, I'm new to twisted and have a simple question. I want to run a threaded server that monitors a remote service and spawns queued/threaded tasks as it finds messages.

Queued/threaded tasks? That do what?

I'm having trouble understanding how to do this after a reactor is running since the tasks are not set up before reactor.run(). I see reactor.callLater(). But I want to spawn the task immediately.

The way to do things after the reactor is running is the same as the way to do things before the reactor is running. Just do them. There's no difference. If you're having trouble figuring out where to put the code that does them, think about when you want the tasks done. You said you want to do things when you "find messages". How do you find messages?

I tried:

reactor.callLater(0,self.some_method,object)

but the method is not immediately invoked. It seems its not invoked at all.

This will call `self.some_method(object)´ soon. If the call is never happening, then you've done something wrong. Perhaps you didn't start the reactor, or perhaps the signature of some_method is incompatible with that invocation so that when the call was attempted, it just raised a TypeError. It's hard to say exactly what's wrong without seeing a complete, self-contained, runnable example which demonstrates the behavior.

I want to create a high-performance threaded scheduler using twist. Any tips?

Think about ways to not use threads. Anything you can do without threads will be higher performance than the threaded equivalent. Jean-Paul

On Wed, 31 Dec 2008 10:51:38 -0500, Darren Govoni wrote:

Hi Jean-Paul, Thank you for those suggestions. I can post some simple code after I get further. The psuedo code of what I want to do is like this: every few seconds, check a message queue (I see how this is done): if there is a message, spawn a deferred so I don't block: # The deferred should immediately call a method in a separate # thread (wisted kind) to process the message # Allow for 10 or so deferreds to be processing messages # in parallel while the reactor continues to receive them # but block scheduling the deferred if the pool is maxed # out

A Deferred is just a convenient way to keep track of callbacks. It's not an active thing, so to "spawn" one doesn't make much sense. Instead, you spawn something else and use a Deferred to keep track of its result.

Sorry if this is lacking in detail, i'm trying to test the idea in a simple way with twisted first. If I set up a deferred before calling reactor.run(), then of course, it runs, but after reactor.run() I have to invoke reactor.callLater(...) so its a bit different, but what I'd like is something like reactor.callNow(...) that doesn't block and invokes the generator in a thread pool.

You probably want twisted.internet.threads.deferToThreadPool (or the older deferToThread). From its docstring: Call the function C{f} using a thread from the given threadpool and return the result as a Deferred. Jean-Paul

It works now. Sorry for the extra post. On Wed, 2008-12-31 at 15:08 -0500, Darren Govoni wrote:

Hi, Thanks for that suggestion. It looks very close to what I need. I tried doing it _after_ the reactor is run and it did not execute. Here is a code snippet.

def convert_to_pdf(arg): print "converting..."

def notify_pdf: print "Yay! I'm done"

... d = threads.deferToThread(convert_to_pdf,offset) d.addCallback(notify_pdf)

All the examples for threads.deferToThread call reactor.run() after configuring all the threads, but this isn't useful in my app.

If I call reactor.run() and _then_ try to deferToThread, it doesn't work. I'm probably missing an argument or something.

Thanks for any tips. Much appreciated.

Darren

On Wed, 2008-12-31 at 11:16 -0500, Jean-Paul Calderone wrote:

...

On Wed, 31 Dec 2008 10:51:38 -0500, Darren Govoni wrote:

...

Hi Jean-Paul, Thank you for those suggestions. I can post some simple code after I get further. The psuedo code of what I want to do is like this: every few seconds, check a message queue (I see how this is done): if there is a message, spawn a deferred so I don't block: # The deferred should immediately call a method in a separate # thread (wisted kind) to process the message # Allow for 10 or so deferreds to be processing messages # in parallel while the reactor continues to receive them # but block scheduling the deferred if the pool is maxed # out

A Deferred is just a convenient way to keep track of callbacks. It's not an active thing, so to "spawn" one doesn't make much sense. Instead, you spawn something else and use a Deferred to keep track of its result.

...

Sorry if this is lacking in detail, i'm trying to test the idea in a simple way with twisted first. If I set up a deferred before calling reactor.run(), then of course, it runs, but after reactor.run() I have to invoke reactor.callLater(...) so its a bit different, but what I'd like is something like reactor.callNow(...) that doesn't block and invokes the generator in a thread pool.

You probably want twisted.internet.threads.deferToThreadPool (or the older deferToThread). From its docstring:

Call the function C{f} using a thread from the given threadpool and return the result as a Deferred.

Jean-Paul

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python