[Twisted-Python] Maximum recursion depth reached
Hi all,
I'm hitting the recursion limit in my code (well, somewhere inside Twisted, that is), and I'd appreciate any insight as to why. I wouldn't think what I'm doing -- returning a callback from method A which calls method B, which then in turn calls method A again -- would be controversial, since I'd assume Python would be letting go of the stack for method A, but I guess there's more to it than that.
Code demonstrating the issue follows. Running it results in: File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded
(This doesn't actually perfectly reflect my problem in the real code I'm working with. I'm seeing "'maximum recursion depth exceeded' in <bound method DebugInfo.__del__ of <twisted.internet.defer.DebugInfo instance at 0xb1d99b6c>>", but I assume this is coming from the same basic problem in my code.)
===================================
from twisted.internet import defer, reactor
def yo_print(s, d): print s d.callback(s) return s
def some_deferred(item): d = defer.Deferred() reactor.callLater(0, lambda: yo_print(item, d)) return d
def start(items): dl = [] for item in items: dl.append(some_deferred(item)) if len(dl) > 4: break items = items[len(dl):] if dl: return defer.DeferredList(dl).addCallback(next_batch, items) reactor.stop()
def next_batch(_, items): return start(items)
if __name__ == '__main__': items = [] for i in range(1651): # 1650 doesn't do it items.append(i) print start(items) reactor.run() ===================================
So I guess the rule is to never, within a callback chain, call a function which was invoked earlier in the callback chain?
Steve
On Friday 09 May 2008, Steve Freitas wrote:
def start(items): dl = [] for item in items: dl.append(some_deferred(item)) if len(dl) > 4: break items = items[len(dl):] if dl: return defer.DeferredList(dl).addCallback(next_batch, items) reactor.stop()
def next_batch(_, items): return start(items)
By returning the DeferredList from next_batch(), you are chaining Deferreds:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto11
If you remove the "return" from next_batch(), the problem disappears. If you want to be able to register a callback when all items are processed, it's probably better to create a dedicated Deferred for that instead of chaining the DeferredLists.
Bye, Maarten
On Fri, 2008-05-09 at 13:20 +0200, Maarten ter Huurne wrote:
By returning the DeferredList from next_batch(), you are chaining Deferreds:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto11
If you remove the "return" from next_batch(), the problem disappears. If you want to be able to register a callback when all items are processed, it's probably better to create a dedicated Deferred for that instead of chaining the DeferredLists.
Thanks for your reply, Marteen!
I guess I'm a little mystified by what's going wrong here -- I chain deferreds all the time -- I'm not sure what it is about returning a few thousand deferreds wrapped up into a smaller number of DeferredLists that's causing the problem, and I'd like to understand why.
Steve
On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas sflist@ihonk.com wrote:
On Fri, 2008-05-09 at 13:20 +0200, Maarten ter Huurne wrote:
By returning the DeferredList from next_batch(), you are chaining Deferreds:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto11
If you remove the "return" from next_batch(), the problem disappears. If you want to be able to register a callback when all items are processed, it's probably better to create a dedicated Deferred for that instead of chaining the DeferredLists.
Thanks for your reply, Marteen!
I guess I'm a little mystified by what's going wrong here -- I chain deferreds all the time -- I'm not sure what it is about returning a few thousand deferreds wrapped up into a smaller number of DeferredLists that's causing the problem, and I'd like to understand why.
Sitting at the core of Deferred is a loop over callback functions. If you have a chained Deferred, then one of those callbacks is doing to be a function which recursively calls the function which has that loop. If you chain enough Deferreds, then eventually this recursion will fail. It turns out to be about 250 Deferreds (roughly 4 stack frames per level of chaining, with the default limit of 1000 stack frames imposed by CPython) which will trigger this limit.
It may be possible to replace this recursion with iteration, but I'm not sure that solves all problems. After all, if you suddenly have to blow through thousands of levels of chaining in response to one event, then you're paying a pretty hefty price which could be avoided by jumping over all the irrelevant intermediate stuff. Maybe *that* could be implemented in Deferred as well somehow, but it's not totally obvious to me how. :)
Hope this helps,
Jean-Paul
On Fri, 2008-05-09 at 13:40 -0400, Jean-Paul Calderone wrote:
On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas sflist@ihonk.com
wrote:
I guess I'm a little mystified by what's going wrong here -- I chain deferreds all the time -- I'm not sure what it is about returning a few thousand deferreds wrapped up into a smaller number of DeferredLists that's causing the problem, and I'd like to understand why.
Sitting at the core of Deferred is a loop over callback functions. If you have a chained Deferred, then one of those callbacks is doing to be a function which recursively calls the function which has that loop. If you chain enough Deferreds, then eventually this recursion will fail. It turns out to be about 250 Deferreds (roughly 4 stack frames per level of chaining, with the default limit of 1000 stack frames imposed by CPython) which will trigger this limit.
Yep, that helps, Jean-Paul, thanks for the info. I've been chaining deferreds for years, but I never thought there was a limit to how many times I could do it, and even if I had, I'd have never expected the number would be so low. I looked in the docs and found no mention of it.
It may be possible to replace this recursion with iteration, but I'm not sure that solves all problems. After all, if you suddenly have to blow through thousands of levels of chaining in response to one event, then you're paying a pretty hefty price which could be avoided by jumping over all the irrelevant intermediate stuff. Maybe *that* could be implemented in Deferred as well somehow, but it's not totally obvious to me how. :)
I disagree that this would be a problem. If I'm chaining 10,000 deferreds, I'm taking responsibility to handle a blow-up of that magnitude. That's why my original example involved putting 5 deferreds at a time into a DeferredList -- in my real code I'm checking the results from that DeferredList and stopping sanely if there's an issue.
Deferred's got too much mojo for me to consider submitting a patch, but I hope somebody does, since there's no conceptual reason why there should be such a limit.
/me wanders off to Trac to submit a ticket...
Steve
participants (3)
-
Jean-Paul Calderone -
Maarten ter Huurne -
Steve Freitas