On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas sflist@ihonk.com wrote:
On Fri, 2008-05-09 at 13:20 +0200, Maarten ter Huurne wrote:
By returning the DeferredList from next_batch(), you are chaining Deferreds:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto11
If you remove the "return" from next_batch(), the problem disappears. If you want to be able to register a callback when all items are processed, it's probably better to create a dedicated Deferred for that instead of chaining the DeferredLists.
Thanks for your reply, Marteen!
I guess I'm a little mystified by what's going wrong here -- I chain deferreds all the time -- I'm not sure what it is about returning a few thousand deferreds wrapped up into a smaller number of DeferredLists that's causing the problem, and I'd like to understand why.
Sitting at the core of Deferred is a loop over callback functions. If you have a chained Deferred, then one of those callbacks is doing to be a function which recursively calls the function which has that loop. If you chain enough Deferreds, then eventually this recursion will fail. It turns out to be about 250 Deferreds (roughly 4 stack frames per level of chaining, with the default limit of 1000 stack frames imposed by CPython) which will trigger this limit.
It may be possible to replace this recursion with iteration, but I'm not sure that solves all problems. After all, if you suddenly have to blow through thousands of levels of chaining in response to one event, then you're paying a pretty hefty price which could be avoided by jumping over all the irrelevant intermediate stuff. Maybe *that* could be implemented in Deferred as well somehow, but it's not totally obvious to me how. :)
Hope this helps,
Jean-Paul