* glyph@divmod.com glyph@divmod.com [2007-09-07 01:43:19 -0000]:
Use a recursive generator that yields Deferreds for loading each row.
Thanks for the example! Combined with a DeferredQueue to pass the data from the callback to the rendering "loop", this approach seems to work perfectly.
Here's an example - which I even looked at in a web browser :) - that should help to get you started. (The athena stuff is beside the point, it's just the quickest way I know to get a server serving a Nevow page...)
The athena-widget plugin is definitely very handy for quick examples like this.
Warning: I haven't tested this on really large datasets, so it might have disastrous performance consequences. I don't *think* that it will, but I know there are some booby-traps in the rendering pipeline to snare the unwary.
As far as I can tell, nothing particularly nasty is happening when I use this; if the overhead of producing each row is very low, all the indirection through generators / deferreds might be costly, but that's certainly not the case for my code (the database and formatting operations are relatively expensive).
def deferLater(n, result=None): """ Why isn't this in Twisted yet? Is it? """
http://twistedmatrix.com/trac/ticket/1875
I'm not really sure what I can do to drive the ticket along, but if there is something, please let me know.