<br><br><div class="gmail_quote">On Tue, Jun 29, 2010 at 7:37 AM, Aaron Fransen <span dir="ltr"><<a href="mailto:aaron.fransen@gmail.com">aaron.fransen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div class="h5"><br><br><div class="gmail_quote">On Mon, Jun 28, 2010 at 5:42 PM, Graham Dumpleton <span dir="ltr"><<a href="mailto:graham.dumpleton@gmail.com" target="_blank">graham.dumpleton@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div>On 29 June 2010 05:01, Aaron Fransen <<a href="mailto:aaron.fransen@gmail.com" target="_blank">aaron.fransen@gmail.com</a>> wrote:<br>
> One of the nice things about mod_python is the req.write() function.<br>
<br>
</div>One thing I should warn you about req.write() in Apache is that for<br>
streaming data as you seem to be using it, it will accumulate memory<br>
against a request for each write call and that will not be reused,<br>
albeit it will be released again at the end of the request.<br>
<br>
The problem here isn't actually in mod_python but in the underlying<br>
Apache ap_rwrite() call.<br>
<br>
What this function does is that for each call to it, it creates what<br>
is called a bucket to hold the data to be written. The memory for this<br>
bucket is allocated from the per request memory pool each time. This<br>
bucket is then passed down the Apache output filter chain and<br>
eventually the data gets written out.<br>
<br>
Now, because the code doesn't attempt to reuse the bucket, that memory<br>
then remains unused, but still allocated against the memory pool, with<br>
the memory pool only being destroyed at the end of the request.<br>
<br>
The outcome of this is that if you had a long running request which<br>
continually wrote out response data in small bits using req.write(),<br>
for each call there is a small increase in amount of memory taken from<br>
the per request memory pool with it not being reused. Thus if the<br>
request were running for a very long time, you will see a gradual<br>
increase in overall memory usage of the process. When the request<br>
finishes, the memory is reclaimed and reused, but you have by then<br>
already set the high ceiling on ongoing process memory in use.<br>
<br>
Anyway, thought I should just warn you about this. In part this issue<br>
may even be why mod_python got a reputation for memory bloat in some<br>
situations. That is, the fundamental way of returning response data<br>
could cause unnecessary increase in process size if called many times<br>
for a request.<br>
<font color="#888888"><br>
Graham<br>
</font></blockquote></div><br><br></div></div>Fortunately we're not talking about a huge amount of data here, basically just a couple of notices to keep the user happy (less than 1K usually).<br><br>When using yield, it's as if the module where the yield command is run is completely ignored. The page returned is a "default" page generated by the application. Errors are being trapped, but none are being generated, it's just exiting without any kind of notice.<br>
<br>When using write() without a Content-Length header, nothing shows on the browser.<br><br>When using write() with a Content-Length header, the first update shows (and only after the entire page has been generated), but none of the subsequent ones nor the final page.<br>
<br>When using write() with a Content-Length header set large enough to encompass the entire final result, the final result page shows, but none of the informational messages leading up to the generation of the page appear.<br>
<br>I haven't really done anything to the base wsgi installation; just set it up in daemon mode.<br><br>
</blockquote></div><br>Couple more things I've been able to discern.<br><br>The first happened after I "fixed" the html code. Originally under mod_python, I guess I was cheating more than a little bit by sending <html></html> code blocks twice, once for the incremental notices, once for the final content. Once I changed the code to send a single properly parsed block, the entire document showed up as expected, however it still did not send any part of the html incrementally.<br>
<br>Watching the line with Wireshark, all of the data was transmitted at the same time, so nothing was sent to the browser incrementally.<br><br>(This is using the write() functionality, I haven't tried watching the line with yield yet.)<br>