On Fri, 2008-05-09 at 13:40 -0400, Jean-Paul Calderone wrote:
On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas sflist@ihonk.com
wrote:
I guess I'm a little mystified by what's going wrong here -- I chain deferreds all the time -- I'm not sure what it is about returning a few thousand deferreds wrapped up into a smaller number of DeferredLists that's causing the problem, and I'd like to understand why.
Sitting at the core of Deferred is a loop over callback functions. If you have a chained Deferred, then one of those callbacks is doing to be a function which recursively calls the function which has that loop. If you chain enough Deferreds, then eventually this recursion will fail. It turns out to be about 250 Deferreds (roughly 4 stack frames per level of chaining, with the default limit of 1000 stack frames imposed by CPython) which will trigger this limit.
Yep, that helps, Jean-Paul, thanks for the info. I've been chaining deferreds for years, but I never thought there was a limit to how many times I could do it, and even if I had, I'd have never expected the number would be so low. I looked in the docs and found no mention of it.
It may be possible to replace this recursion with iteration, but I'm not sure that solves all problems. After all, if you suddenly have to blow through thousands of levels of chaining in response to one event, then you're paying a pretty hefty price which could be avoided by jumping over all the irrelevant intermediate stuff. Maybe *that* could be implemented in Deferred as well somehow, but it's not totally obvious to me how. :)
I disagree that this would be a problem. If I'm chaining 10,000 deferreds, I'm taking responsibility to handle a blow-up of that magnitude. That's why my original example involved putting 5 deferreds at a time into a DeferredList -- in my real code I'm checking the results from that DeferredList and stopping sanely if there's an issue.
Deferred's got too much mojo for me to consider submitting a patch, but I hope somebody does, since there's no conceptual reason why there should be such a limit.
/me wanders off to Trac to submit a ticket...
Steve