Apologies Graham, I'm not actually trying to appear dense but clearly I'm not one of the world's bright lights when it comes to web interfaces.<br><br>My installation is literally a base installation of the latest Ubuntu server platform. The only configuration at play is this:<br>
<br> WSGIDaemonProcess node9 user=www-data group=www-data processes=2 threads=25<br> WSGIProcessGroup node9<br> WSGIScriptAlias /run /var/www/run/run.py<br><br>The error that occurs when using telnet and yield is:<br>
<br>[Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): Target WSGI script '/var/www/run/run.py' cannot be loaded as Python module.<br>[Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): Exception occurred processing WSGI script '/var/www/run/run.py'.<br>
[Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] SyntaxError: 'return' with argument inside generator (run.py, line 14)<br><br>using this code:<br><br> status = '200 OK'<br> response_headers = [('Content-type','text/plain')]<br>
start_response(status, response_headers)<br> for x in range(0,10):<br> yield 'hey %s' % x<br> time.sleep(1)<br><br>The error occurs when I use "return []" as opposed to simply "return", however I now see that is a result of the yield command itself.<br>
<br>Using this method, the telnet interface returns immediately with:<br><br>HTTP/1.1 200 OK<br>Date: Mon, 05 Jul 2010 12:30:45 GMT<br>Server: Apache/2.2.14 (Ubuntu)<br>Vary: Accept-Encoding<br>Connection: close<br>Content-Type: text/plain<br>
<br>0<br>Connection closed by foreign host.<br><br>In fact, whether using yield or write produces the same result.<br><br>If I'm not getting the results I should be, then obviously I'm doing something wrong.<br><br>
I understand the danger of having a long-running web process (hence the reason I have a lot of virtual machines in the live environment using mod_python right now) but unfortunately it's something I don't seem to be able to work around at the moment.<br>
<br>Thanks to all.<br><br><div class="gmail_quote">On Wed, Jun 30, 2010 at 5:19 PM, Graham Dumpleton <span dir="ltr"><<a href="mailto:graham.dumpleton@gmail.com">graham.dumpleton@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div class="h5">On 30 June 2010 22:55, Aaron Fransen <<a href="mailto:aaron.fransen@gmail.com">aaron.fransen@gmail.com</a>> wrote:<br>
><br>
> I can see that this could potentially get very ugly very quickly.<br>
><br>
> Using stock Apache on the current Ubuntu server, using yield produced a<br>
> response error<br>
<br>
</div></div>What error? If you aren't going to debug it enough to even work out<br>
what the error is in the browser or Apache error logs and post it here<br>
for comment so can say what may be wrong on your system, then we cant<br>
exactly help you much can we.<br>
<div class="im"><br>
> and using write() (over the telnet interface) returned the 0<br>
> only and disconnected. Similar behavior in Firefox.<br>
<br>
</div>All the scripts I provided you are conforming WSGI applications and<br>
work on mod_wsgi. If you are having issues, then it is likely going to<br>
be the way your Apache/Python is setup or how you configured mod_wsgi<br>
to host the scripts. Again, because you are providing no details about<br>
how you configured mod_wsgi we cant help you work out what is wrong<br>
with your system.<br>
<div class="im"><br>
> How odd that nobody's come up with a simple streaming/update schema (at<br>
> least to my mind).<br>
<br>
</div>For response content they have and it can be made to work. Just<br>
because you cant get it working or don't understand what we are saying<br>
about the need to use a JavaScript/AJAX type client (eg. comet style)<br>
to make use of it as opposed to trying to rely on browser<br>
functionality that doesn't exist doesn't change that. Request content<br>
streaming is a different matter as I will explain below but you<br>
haven't even mentioned that as yet that I can see.<br>
<div class="im"><br>
> It would have been nice to be able to provide some kind of in-stream<br>
> feedback for long running jobs, but it looks like I'm going to have to<br>
> abandon that approach. The only issue with either of the other solutions is<br>
> that each subsequent request depends on data provided by the prior, so the<br>
> amount of traffic going back & forth could potentially become a problem.<br>
><br>
> Alternatively I could simply create a session database that saves the<br>
> required objects then each subsequent request simply fetches the required<br>
> one from the table and...<br>
><br>
> Well, you can see why streaming seemed like such a simple solution! Back to<br>
> the drawing board, as it were.<br>
<br>
</div>I'll try one last time to try and summarise a few issues for you,<br>
although based on your attitude so far, I don't think it will change<br>
your opinion or help your understanding.<br>
<br>
1. Streaming of responses from a WSGI application works fine using<br>
either yield or write(). If it doesn't work for a specific WSGI<br>
hosting mechanism then that implementation may not be conforming to<br>
WSGI requirements. Specifically, between a yield and/or write() it is<br>
required that an implicit flush is performed. This should ensure that<br>
the data is written to the HTTP client connection and/or ensure that<br>
the return of such data to the client occurs in parallel to further<br>
actions occurring in that request.<br>
<br>
2. A WSGI middleware that caches response data can stuff this up. One<br>
cant outright prohibit a WSGI middleware holding on to response data,<br>
albeit that for each yield or write() technically it is supposed to<br>
still pass on at least an empty string down the chain so as to allow<br>
control to get back to the underlying WSGI implementation, which may<br>
uses such windows to swap what request context it is operating on so<br>
as to allow a measure of concurrency in situation where threads may<br>
not be getting used.<br>
<br>
3. Where a WSGI adapter on top of an existing web server is used, eg.<br>
various options that exist with Apache and nginx, then an output<br>
filter configured into the web server may also stuff this up. For<br>
example, an output filter that compresses response data may buffer up<br>
response data into large blocks before compressing them and returning<br>
them.<br>
<br>
4. Although response content can be streamed subject to above caveats,<br>
streaming of request content is a totally different matter. First off,<br>
WSGI requires that the request content have a Content-Length<br>
specified. Thus technically a HTTP client cant leave out<br>
Content-Length and instead used chunked request content. Further, the<br>
way in which many web servers and WSGI servers are implemented would<br>
prohibit streaming of request content anyway. This is because many<br>
implementations, especially where proxying occurs, eg. cgi, fastcgi,<br>
scgi, ajp, uwsgi, mod_proxy (??), and mod_wsgi daemon mode, expect<br>
that the whole request content can be read in and written across the<br>
proxy connection before any attempt is made to start reading any data<br>
returned from the web application. The request content therefore<br>
cannot be open ended in length because most implementations will never<br>
switch from reading that content to expecting response from the<br>
application. Thus it isn't possible to use WSGI as both way streaming<br>
mechanism where some request content is written, some response content<br>
returned and then the client sends more request content based on that<br>
etc etc.<br>
<br>
So what does this all mean. First up is that response content<br>
streaming should be able to be made to work, however since request<br>
content streaming isn't technically allowed with WSGI, if you need<br>
that you are out of luck if you want to conform to WSGI specification.<br>
Second, you can however with mod_wsgi embedded mode slightly step<br>
outside of strict WSGI conformance and have request content streaming.<br>
You are then bound to Apache/mod_wsgi, but whether you want to do that<br>
is debatable for reasons below.<br>
<br>
The bigger problem with both way streaming or long polling<br>
applications which use the same HTTP request is that WSGI servers tend<br>
to use processes and threads for concurrency. When you use this<br>
mechanisms they will tie up a process or thread for the whole time.<br>
Thus if you have lots of concurrent request you need huge numbers of<br>
processes and/or threads, which just isn't usually practical because<br>
of resource usage such as memory. For that reason, one would instead<br>
on the server usually use special purpose web servers for these types<br>
of applications and use HTTP directly and avoid WSGI, due to WSGI<br>
blocking nature. Instead these servers would use an event driven<br>
system model or other system which allows concurrency without<br>
requiring a process or thread per application.<br>
<br>
In short, this is what Comet and dedicated servers for that are about.<br>
Allowing large numbers of concurrent long requests with minimal<br>
resources. That they are dedicated systems also allows them to avoid<br>
limitations in other high level web application interfaces such as<br>
CGI, FASTCGI, SCGI, AJP etc which have an expectation that can read<br>
whole request content before trying to deal with any response from a<br>
web application that is handling the requests.<br>
<br>
Anyway, hopefully that explains things better. You can do what you<br>
want, you just need to select the correct tool for the job.<br>
<font color="#888888"><br>
Graham<br>
</font></blockquote></div><br>