<br><br><div class="gmail_quote">On Wed, Jun 30, 2010 at 6:26 AM, Graham Dumpleton <span dir="ltr"><<a href="mailto:graham.dumpleton@gmail.com">graham.dumpleton@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div class="h5">On 30 June 2010 21:35, Aaron Fransen <<a href="mailto:aaron.fransen@gmail.com">aaron.fransen@gmail.com</a>> wrote:<br>
><br>
><br>
> On Tue, Jun 29, 2010 at 6:17 PM, Graham Dumpleton<br>
> <<a href="mailto:graham.dumpleton@gmail.com">graham.dumpleton@gmail.com</a>> wrote:<br>
>><br>
>> On 30 June 2010 02:14, Aaron Fransen <<a href="mailto:aaron.fransen@gmail.com">aaron.fransen@gmail.com</a>> wrote:<br>
>> > Couple more things I've been able to discern.<br>
>> ><br>
>> > The first happened after I "fixed" the html code. Originally under<br>
>> > mod_python, I guess I was cheating more than a little bit by sending<br>
>> > <html></html> code blocks twice, once for the incremental notices, once<br>
>> > for<br>
>> > the final content. Once I changed the code to send a single properly<br>
>> > parsed<br>
>> > block, the entire document showed up as expected, however it still did<br>
>> > not<br>
>> > send any part of the html incrementally.<br>
>> ><br>
>> > Watching the line with Wireshark, all of the data was transmitted at the<br>
>> > same time, so nothing was sent to the browser incrementally.<br>
>> ><br>
>> > (This is using the write() functionality, I haven't tried watching the<br>
>> > line<br>
>> > with yield yet.)<br>
>><br>
>> Use a variation of WSGI middleware wrapper in:<br>
>><br>
>><br>
>> <a href="http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response" target="_blank">http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response</a><br>
>><br>
>> using it to 'print' returned data to Apache log and then tail Apache<br>
>> error log to see when that data is output. Alternatively, change the<br>
>> code there to output a time stamp against each chunk of data written<br>
>> to the file recording the response content.<br>
>><br>
>> This will show what data is returned by WSGI application, before<br>
>> mod_wsgi truncates anything greater than content length specified,<br>
>> plus also show whether it is your WSGI application which is delaying<br>
>> output somehow, or whether Apache output filters are doing it.<br>
>><br>
>> Graham<br>
><br>
> I've actually tried a variation on this already using a built-in logging<br>
> facility in the application that writes date/time values to an external log<br>
> file with comments, and in the case of testing wsgi I actually included some<br>
> time.sleep() statements to force a delay in the application.<br>
><br>
> To give you an idea of the flow, here's essentially what's going on:<br>
><br>
> def application(environ,start_response):<br>
> mydict = {}<br>
> mydict['environ']=environ<br>
> mydict['startresponse'] = start_response<br>
> # run program in another .py file that has been imported<br>
> RunTest(mydict)<br>
><br>
> Then in the other module you would have something like:<br>
><br>
> def RunTest(mydict):<br>
> status = '200 OK'<br>
> response_headers = [('Content-type','text/html')]<br>
> writeobj = detail['startresponse'](status,response_headers)<br>
> writeobj('<html><body>Fetching sales for 2009...')<br>
> time.sleep(2)<br>
> writeobj('<br>Fetching sales for 2010...')<br>
><br>
> ...then finally...<br>
><br>
> writeobj('5000 results returned.</body></html>')<br>
> return<br>
><br>
> This is obviously a truncated (and fake) example, but it gives you an idea<br>
> of the flow.<br>
<br>
</div></div>Now go try the following two examples as illustrated instead.<br>
<br>
In both cases, do not use a web browser, instead telnet to the port of<br>
the web server and enter HTTP GET directly. If you are not using<br>
VirtualHost, use something like:<br>
<br>
telnet localhost 80<br>
GET /stream-yield.wsgi HTTP/1.0<br>
<br>
If using a VirtualHost, use something like:<br>
<br>
telnet localhost 80<br>
GET /stream-yield.wsgi HTTP/1.1<br>
Host: <a href="http://tests.example.com" target="_blank">tests.example.com</a><br>
<br>
Ensure additional blank line entered to indicate end of headers.<br>
<br>
First example uses yield.<br>
<br>
# stream-yield.wsgi<br>
<br>
import time<br>
<div class="im"><br>
def application(environ, start_response):<br>
</div> status = '200 OK'<br>
<br>
response_headers = [('Content-type', 'text/plain')]<br>
start_response(status, response_headers)<br>
<br>
for i in range(10):<br>
yield '%d\n' % i<br>
time.sleep(1)<br>
<br>
Second example uses write:<br>
<br>
# stream-write.wsgi<br>
<br>
import time<br>
<div class="im"><br>
def application(environ, start_response):<br>
</div> status = '200 OK'<br>
<br>
response_headers = [('Content-type', 'text/plain')]<br>
write = start_response(status, response_headers)<br>
<br>
for i in range(10):<br>
write('%d\n' % i)<br>
time.sleep(1)<br>
<br>
return []<br>
<br>
For me, using stock standard operating system supplied Apache on Mac<br>
OS X, I see a line returned every second.<br>
<br>
If I use Safari as a web browser, in both cases the browser only shows<br>
the response after all data has been written and the socket connection<br>
closed. If I use Firefox however, they display as data comes in.<br>
<br>
This delay in display is thus possibly just the behaviour of a<br>
specific browser delaying the display until the socket is closed.<br>
<br>
The example for multipart/x-mixed-replace which others mention is:<br>
<br>
import time<br>
<div class="im"><br>
def application(environ, start_response):<br>
</div> status = '200 OK'<br>
<br>
response_headers = [('Content-Type', 'multipart/x-mixed-replace;<br>
boundary=xstringx')]<br>
start_response(status, response_headers)<br>
<br>
yield '--xstrinx\n'<br>
<br>
for i in range(10):<br>
<br>
yield 'Content-type: text/plain\n'<br>
yield '\n'<br>
yield '%d\n' % i<br>
yield '--xstringx\n'<br>
<br>
time.sleep(1)<br>
<br>
With telnet you will see the various sections, but with Safari again<br>
only shows at end, although you will find that it only shows the data<br>
line, ie., the number and not all the other stuff. So, understands<br>
multipart format but doesn't support x-mixed-replace. It was always<br>
the case that only certain browsers supported that mime type. In the<br>
case of Firefox, it doesn't seem to like it at all and seems to give<br>
up and not display anything, not even replacing the previously<br>
displayed page contents.<br>
<br>
What this means is that you cant rely on browsers to handle multipart<br>
mixed replace alone. If you were really going to use that format, you<br>
really want to use JavaScript and AJAX stuff to process it. The same<br>
applies for progressive display of plain text content when streamed<br>
over time.<br>
<br>
In summary, you really want to be using some JavaScript/AJAX stuff on<br>
browser side to get uniform behaviour on all the browsers.<br>
<font color="#888888"><br>
Graham<br>
</font></blockquote></div><br>I can see that this could potentially get very ugly very quickly.<br><br>Using stock Apache on the current Ubuntu server, using yield produced a response error and using write() (over the telnet interface) returned the 0 only and disconnected. Similar behavior in Firefox.<br>
<br>How odd that nobody's come up with a simple streaming/update schema (at
least to my mind).<br>
<br>
It would have been nice to be able to provide some kind of in-stream feedback for long running jobs, but it looks like I'm going to have to abandon that approach. The only issue with either of the other solutions is that each subsequent request depends on data provided by the prior, so the amount of traffic going back & forth could potentially become a problem.<br>
<br>Alternatively I could simply create a session database that saves the required objects then each subsequent request simply fetches the required one from the table and...<br><br>Well, you can see why streaming seemed like such a simple solution! Back to the drawing board, as it were.<br>
<br>Thanks all.<br>