Mailman 3 Streaming render - Twisted-web

Streaming render

Tristan Seligmann

6 Sep 2007 6 Sep '07

11:59 p.m.

I've run into a curious blank wall here: I'm rendering a page, basically consisting of an HTML table of results plus various fluff. In order to generate these results, I fire off an asynchronous operation to which I pass a callback; the callback is then invoked once per result, corresponding to a row in the resulting table. Currently I'm just firing the operation off in beforeRender, building up a list as an instance attribute on the class in question, and then preceeding with the rendering, using that list in a renderer later. For small result sets, this works fine; however, I'm now running into situations where I have over 10k rows in the resulting table. Under these circumstances, holding all this data in memory and then rendering it all at once incurs a rather large and unnecessary memory overhead; also, there is a large delay between the client's request, and the response document beginning to be streamed. I'd like to start streaming the document as soon as I start receiving the results of the operation, but I'm stumped on how to actually implement this. If I return a deferred from the render method, I cannot fire this deferred until all of the rows have been built up. Any ideas? I'm sure I've overlooked something obvious here, but I'm drawing a complete blank. -- mithrandi, i Ainil en-Balandor, a faer Ambar

Attachments:

signature.asc (application/pgp-signature — 197 bytes)

Show replies by date

Maarten ter Huurne

7 Sep 7 Sep

12:32 a.m.

On Friday 07 September 2007, Tristan Seligmann wrote:

...

I'm rendering a page, basically consisting of an HTML table of results plus various fluff. In order to generate these results, I fire off an asynchronous operation to which I pass a callback; the callback is then invoked once per result, corresponding to a row in the resulting table. Currently I'm just firing the operation off in beforeRender, building up a list as an instance attribute on the class in question, and then preceeding with the rendering, using that list in a renderer later.

I don't know "beforeRender", is a method from Nevow? In the old web module, you can just call "write" on the request object whenever you have rendered part of the page and end by calling "finish". Bye, Maarten

glyph＠divmod.com

1:43 a.m.

On 6 Sep, 11:59 pm, mithrandi-twisted-web@mithrandi.za.net wrote:

...

I'd like to start streaming the document as soon as I start receiving the results of the operation, but I'm stumped on how to actually implement this. If I return a deferred from the render method, I cannot fire this deferred until all of the rows have been built up.

Use a recursive generator that yields Deferreds for loading each row. Here's an example - which I even looked at in a web browser :) - that should help to get you started. (The athena stuff is beside the point, it's just the quickest way I know to get a server serving a Nevow page...) Warning: I haven't tested this on really large datasets, so it might have disastrous performance consequences. I don't *think* that it will, but I know there are some booby-traps in the rendering pipeline to snare the unwary. # -- cut here for slow.py -- # Run quickly: twistd -n athena-widget --element slow.SlowElement from twisted.internet import defer from twisted.internet import reactor from nevow.loaders import stan from nevow.tags import div, directive from nevow.athena import LiveElement from nevow.page import renderer def deferLater(n, result=None): """ Why isn't this in Twisted yet? Is it? """ d = defer.Deferred() reactor.callLater(n, d.callback, result) return d def delayedResults(results): """ Yield results.... slowly..... """ yield div[results.pop(0)] if results: def more(result): return delayedResults(results) yield deferLater(2.0).addCallback(more) class SlowElement(LiveElement): """ Render a list of 1 to 10 really slowly. """ def start(self, ctx, data): """ Give some data, wait for next result... """ return delayedResults(map(str,range(10))) renderer(start) docFactory = stan(div(render=directive("start")))

Tristan Seligmann

3:01 p.m.

* glyph@divmod.com [2007-09-07 01:43:19 -0000]:

...

Use a recursive generator that yields Deferreds for loading each row.

Thanks for the example! Combined with a DeferredQueue to pass the data from the callback to the rendering "loop", this approach seems to work perfectly.

...

Here's an example - which I even looked at in a web browser :) - that should help to get you started. (The athena stuff is beside the point, it's just the quickest way I know to get a server serving a Nevow page...)

The athena-widget plugin is definitely very handy for quick examples like this.

...

Warning: I haven't tested this on really large datasets, so it might have disastrous performance consequences. I don't *think* that it will, but I know there are some booby-traps in the rendering pipeline to snare the unwary.

As far as I can tell, nothing particularly nasty is happening when I use this; if the overhead of producing each row is very low, all the indirection through generators / deferreds might be costly, but that's certainly not the case for my code (the database and formatting operations are relatively expensive).

...

def deferLater(n, result=None): """ Why isn't this in Twisted yet? Is it? """

http://twistedmatrix.com/trac/ticket/1875 I'm not really sure what I can do to drive the ticket along, but if there is something, please let me know. -- mithrandi, i Ainil en-Balandor, a faer Ambar

glyph＠divmod.com

9:38 p.m.

On 03:01 pm, mithrandi-twisted-web@mithrandi.za.net wrote:

...

* glyph@divmod.com [2007-09-07 01:43:19 -0000]:

...
Use a recursive generator that yields Deferreds for loading each row.

Thanks for the example! Combined with a DeferredQueue to pass the data from the callback to the rendering "loop", this approach seems to work perfectly.

No problem! Your Fan Club Dollars At Work!22.

...

...
Here's an example - which I even looked at in a web browser :) - that should help to get you started. (The athena stuff is beside the point, it's just the quickest way I know to get a server serving a Nevow page...)

The athena-widget plugin is definitely very handy for quick examples like this.

We've also been discussing a slightly more general "twistd nevow" plugin. More on this as it develops.

...

...
Warning: I haven't tested this on really large datasets, so it might have disastrous performance consequences. I don't *think* that it will, but I know there are some booby-traps in the rendering pipeline to snare the unwary.

As far as I can tell, nothing particularly nasty is happening when I use this; if the overhead of producing each row is very low, all the indirection through generators / deferreds might be costly, but that's certainly not the case for my code (the database and formatting operations are relatively expensive).

Good to know.

...

http://twistedmatrix.com/trac/ticket/1875

I'm not really sure what I can do to drive the ticket along, but if there is something, please let me know.

I vaguely remembered this ticket when I wrote that comment :-). Some of the things that are blocking it are on my list of stuff to get back to when I get back to Twisted development at all.

6074

Age (days ago)

6075

Last active (days ago)

List overview

Download

4 comments

3 participants

participants (3)

glyph＠divmod.com
Maarten ter Huurne
Tristan Seligmann