On Fri, 27 Feb 2009 15:26:43 +0100, Marcel Keller <mkeller@cs.au.dk> wrote:
Hi,
I am working on the VIFF project (viff.dk) which uses Twisted. I found out that our code is sometimes inefficient because we are generating many deferreds (maybe about 10000) in a callback. While doing that, no network communication is performed. Therefore, I investigated the possibility of adding a function to the reactor which is called every iteration and from which the iteration could be called safely. Then, we could generate all deferreds in that function and activate the reactor from to time. See the attached patch for details.
So you're doing a ton of work all at once now and you want to split up that ton of work into smaller pieces and do it a little at a time? If that's the case, then you don't need to modify the reactor, you just need to split up the work your code is going. There are a lot of techniques for doing this. coiterate and inlineCallbacks are two solutions which are closest to "cookie cutter" (ie, you have the least flexibility in deciding how to use them).
Some of our code runs twice as fast when using that hack. Are there any chances something similar might be included in Twisted? Or does anyone have a better solution for the described problem?
Give coiterate or inlineCallbacks a try. If you need help applying either of these to your problem, please ask. :) I can't make any specific suggestions now because I can only guess at how you're using the reactor modification you attached. You have a very long, steep, uphill battle to convince me that adding support for re-entrant iteration is a good idea.
Furthermore, I have a question, which is probably related. The documentation of IReactorCore says about the iterate() function:
"The reactor must have been started (via the run() method) prior to any invocations of this method. It must also be stopped manually after the last call to this method (via the stop() method). This method is not re-entrant: you must not call it recursively; in particular, you must not call it while the reactor is running."
This looks to me as if the reactor needs to be running and not running at the same time so that iterate() can be called. Is there an error in my reasoning?
It's very subtle and intentionally written to sound impossible to use (maybe that is a stupid idea, but that was my intent when I wrote it ;). What you're missing is that there is a way to make reactor.run() return without stopping the reactor. Jean-Paul