[Twisted-Python] Strange recursion error with twisted.web
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
I'm getting a "maximum recursion depth exceeded" error that appears to be coming from flatten(). The odd thing is that it only happens sometimes. The HTML that's being flattened does have a few Deferreds in it. Those come from function calls, which cache the results, which might explain why I only see the error on the first visit to the page (as far as I can tell). The system recursion limit is the standard 1000. My HTML is only nested a few tags deep, two orders of magnitude short of that. Is there anything about the way flatten() works that might cause this behaviour? Peter.
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Tue, 9 Mar 2021, at 19:28, Glyph wrote:
Here it is: import sys from twisted.internet import reactor, defer, task from twisted.web.template import flatten def output(stuff): sys.stdout.write(stuff.decode()) def sync(reactor): return flatten(None, [defer.succeed(str(i)+'\n') for i in range(1000)], output) task.react(sync) It fails after printing 197 lines. The same sort of thing using deferLater instead of defer.succeed printed 1000 without error. Peter.
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Wed, 10 Mar 2021, at 23:11, Glyph wrote:
Here is it: https://twistedmatrix.com/trac/ticket/10125#ticket thanks, Peter.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I did indeed have a half-finished experiment to fix this. The fix is now finished and in review here: https://twistedmatrix.com/trac/ticket/10125#comment:1 <https://twistedmatrix.com/trac/ticket/10125#comment:1> . If you could review it thoroughly enough, I could land it. -g
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Sun, 14 Mar 2021, at 02:08, Glyph wrote:
I did indeed have a half-finished experiment to fix this. The fix is now finished and in review here: https://twistedmatrix.com/trac/ticket/10125#comment:1 .
If you could review it thoroughly enough, I could land it.
Excellent, thank you! I think I understand how it works, but one thing puzzles me: how does the queue in the jump() function ever get longer than one element? Is it because it might contain a recursive call, which is activated by the unpause()? Peter.
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
Hello Twisted team, Do we have any idea when the fix for "RecursionError with synchronous deferreds (#1549)" will appear in a release? Peter.
![](https://secure.gravatar.com/avatar/82bbf56e965eb9c44f5f8a40c6958d77.jpg?s=120&d=mm&r=g)
On Mon, Jun 14, 2021 at 05:25:42PM +0100, Peter Westlake wrote:
Do we have any idea when the fix for "RecursionError with synchronous deferreds (#1549)" will appear in a release?
It's been merged to trunk, so it'll be in the next release... whenever that is. Twisted tends to release about once or twice a year but I don't think there's a schedule. Are you able to test against a recent development version? It's rare for trunk to have regressions, so it should be pretty safe to run against a recent version if it fixes a crash you're having.
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Tue, Jun 15, 2021, at 2:46 AM, Peter Westlake wrote:
We don't publish snapshots, but you can install directly from GitHub, like: pip install https://github.com/twisted/twisted/archive/trunk.zip It's probably wise to specify a commit rather than reference trunk. For example, to install the latest commit as of now: pip install https://github.com/twisted/twisted/archive/4edc214ce9ca61950dd613b4d2dbb6fa8... Pip can also install from a Git URL, but using the archive URL is faster because you avoid doing a full clone. ---Tom
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Tue, 9 Mar 2021, at 19:28, Glyph wrote:
Here it is: import sys from twisted.internet import reactor, defer, task from twisted.web.template import flatten def output(stuff): sys.stdout.write(stuff.decode()) def sync(reactor): return flatten(None, [defer.succeed(str(i)+'\n') for i in range(1000)], output) task.react(sync) It fails after printing 197 lines. The same sort of thing using deferLater instead of defer.succeed printed 1000 without error. Peter.
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Wed, 10 Mar 2021, at 23:11, Glyph wrote:
Here is it: https://twistedmatrix.com/trac/ticket/10125#ticket thanks, Peter.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I did indeed have a half-finished experiment to fix this. The fix is now finished and in review here: https://twistedmatrix.com/trac/ticket/10125#comment:1 <https://twistedmatrix.com/trac/ticket/10125#comment:1> . If you could review it thoroughly enough, I could land it. -g
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
On Sun, 14 Mar 2021, at 02:08, Glyph wrote:
I did indeed have a half-finished experiment to fix this. The fix is now finished and in review here: https://twistedmatrix.com/trac/ticket/10125#comment:1 .
If you could review it thoroughly enough, I could land it.
Excellent, thank you! I think I understand how it works, but one thing puzzles me: how does the queue in the jump() function ever get longer than one element? Is it because it might contain a recursive call, which is activated by the unpause()? Peter.
![](https://secure.gravatar.com/avatar/e0114f22fcde3deed8ebe94c70652140.jpg?s=120&d=mm&r=g)
Hello Twisted team, Do we have any idea when the fix for "RecursionError with synchronous deferreds (#1549)" will appear in a release? Peter.
![](https://secure.gravatar.com/avatar/82bbf56e965eb9c44f5f8a40c6958d77.jpg?s=120&d=mm&r=g)
On Mon, Jun 14, 2021 at 05:25:42PM +0100, Peter Westlake wrote:
Do we have any idea when the fix for "RecursionError with synchronous deferreds (#1549)" will appear in a release?
It's been merged to trunk, so it'll be in the next release... whenever that is. Twisted tends to release about once or twice a year but I don't think there's a schedule. Are you able to test against a recent development version? It's rare for trunk to have regressions, so it should be pretty safe to run against a recent version if it fixes a crash you're having.
![](https://secure.gravatar.com/avatar/bcb6ef473ff1644fddee1b4e7c730b01.jpg?s=120&d=mm&r=g)
On Tue, Jun 15, 2021, at 2:46 AM, Peter Westlake wrote:
We don't publish snapshots, but you can install directly from GitHub, like: pip install https://github.com/twisted/twisted/archive/trunk.zip It's probably wise to specify a commit rather than reference trunk. For example, to install the latest commit as of now: pip install https://github.com/twisted/twisted/archive/4edc214ce9ca61950dd613b4d2dbb6fa8... Pip can also install from a Git URL, but using the archive URL is faster because you avoid doing a full clone. ---Tom
participants (4)
-
Glyph
-
Peter Westlake
-
Tom Most
-
Wim Lewis