From graham.dumpleton at gmail.com Thu Jul 1 01:19:33 2010 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Thu, 1 Jul 2010 09:19:33 +1000 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: On 30 June 2010 22:55, Aaron Fransen wrote: > > > On Wed, Jun 30, 2010 at 6:26 AM, Graham Dumpleton > wrote: >> >> On 30 June 2010 21:35, Aaron Fransen wrote: >> > >> > >> > On Tue, Jun 29, 2010 at 6:17 PM, Graham Dumpleton >> > wrote: >> >> >> >> On 30 June 2010 02:14, Aaron Fransen wrote: >> >> > Couple more things I've been able to discern. >> >> > >> >> > The first happened after I "fixed" the html code. Originally under >> >> > mod_python, I guess I was cheating more than a little bit by sending >> >> > code blocks twice, once for the incremental notices, >> >> > once >> >> > for >> >> > the final content. Once I changed the code to send a single properly >> >> > parsed >> >> > block, the entire document showed up as expected, however it still >> >> > did >> >> > not >> >> > send any part of the html incrementally. >> >> > >> >> > Watching the line with Wireshark, all of the data was transmitted at >> >> > the >> >> > same time, so nothing was sent to the browser incrementally. >> >> > >> >> > (This is using the write() functionality, I haven't tried watching >> >> > the >> >> > line >> >> > with yield yet.) >> >> >> >> Use a variation of WSGI middleware wrapper in: >> >> >> >> >> >> >> >> ?http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response >> >> >> >> using it to 'print' returned data to Apache log and then tail Apache >> >> error log to see when that data is output. Alternatively, change the >> >> code there to output a time stamp against each chunk of data written >> >> to the file recording the response content. >> >> >> >> This will show what data is returned by WSGI application, before >> >> mod_wsgi truncates anything greater than content length specified, >> >> plus also show whether it is your WSGI application which is delaying >> >> output somehow, or whether Apache output filters are doing it. >> >> >> >> Graham >> > >> > I've actually tried a variation on this already using a built-in logging >> > facility in the application that writes date/time values to an external >> > log >> > file with comments, and in the case of testing wsgi I actually included >> > some >> > time.sleep() statements to force a delay in the application. >> > >> > To give you an idea of the flow, here's essentially what's going on: >> > >> > def application(environ,start_response): >> > ??? mydict = {} >> > ??? mydict['environ']=environ >> > ??? mydict['startresponse'] = start_response >> > ??? # run program in another .py file that has been imported >> > ??? RunTest(mydict) >> > >> > Then in the other module you would have something like: >> > >> > def RunTest(mydict): >> > ??? status = '200 OK' >> > ??? response_headers = [('Content-type','text/html')] >> > ??? writeobj = detail['startresponse'](status,response_headers) >> > ??? writeobj('Fetching sales for 2009...') >> > ??? time.sleep(2) >> > ??? writeobj('
Fetching sales for 2010...') >> > >> > ??? ...then finally... >> > >> > ??? writeobj('5000 results returned.') >> > ??? return >> > >> > This is obviously a truncated (and fake) example, but it gives you an >> > idea >> > of the flow. >> >> Now go try the following two examples as illustrated instead. >> >> In both cases, do not use a web browser, instead telnet to the port of >> the web server and enter HTTP GET directly. If you are not using >> VirtualHost, use something like: >> >> ?telnet localhost 80 >> ?GET /stream-yield.wsgi HTTP/1.0 >> >> If using a VirtualHost, use something like: >> >> ?telnet localhost 80 >> ?GET /stream-yield.wsgi HTTP/1.1 >> ?Host: tests.example.com >> >> Ensure additional blank line entered to indicate end of headers. >> >> First example uses yield. >> >> # stream-yield.wsgi >> >> import time >> >> def application(environ, start_response): >> ? ?status = '200 OK' >> >> ? ?response_headers = [('Content-type', 'text/plain')] >> ? ?start_response(status, response_headers) >> >> ? ?for i in range(10): >> ? ? ?yield '%d\n' % i >> ? ? ?time.sleep(1) >> >> Second example uses write: >> >> # stream-write.wsgi >> >> import time >> >> def application(environ, start_response): >> ? ?status = '200 OK' >> >> ? ?response_headers = [('Content-type', 'text/plain')] >> ? ?write = start_response(status, response_headers) >> >> ? ?for i in range(10): >> ? ? ?write('%d\n' % i) >> ? ? ?time.sleep(1) >> >> ? ?return [] >> >> For me, using stock standard operating system supplied Apache on Mac >> OS X, I see a line returned every second. >> >> If I use Safari as a web browser, in both cases the browser only shows >> the response after all data has been written and the socket connection >> closed. If I use Firefox however, they display as data comes in. >> >> This delay in display is thus possibly just the behaviour of a >> specific browser delaying the display until the socket is closed. >> >> The example for multipart/x-mixed-replace which others mention is: >> >> import time >> >> def application(environ, start_response): >> ? ?status = '200 OK' >> >> ? ?response_headers = [('Content-Type', 'multipart/x-mixed-replace; >> boundary=xstringx')] >> ? ?start_response(status, response_headers) >> >> ? ?yield '--xstrinx\n' >> >> ? ?for i in range(10): >> >> ? ? ?yield 'Content-type: text/plain\n' >> ? ? ?yield '\n' >> ? ? ?yield '%d\n' % i >> ? ? ?yield '--xstringx\n' >> >> ? ? ?time.sleep(1) >> >> With telnet you will see the various sections, but with Safari again >> only shows at end, although you will find that it only shows the data >> line, ie., the number and not all the other stuff. So, understands >> multipart format but doesn't support x-mixed-replace. It was always >> the case that only certain browsers supported that mime type. In the >> case of Firefox, it doesn't seem to like it at all and seems to give >> up and not display anything, not even replacing the previously >> displayed page contents. >> >> What this means is that you cant rely on browsers to handle multipart >> mixed replace alone. If you were really going to use that format, you >> really want to use JavaScript and AJAX stuff to process it. The same >> applies for progressive display of plain text content when streamed >> over time. >> >> In summary, you really want to be using some JavaScript/AJAX stuff on >> browser side to get uniform behaviour on all the browsers. >> >> Graham > > I can see that this could potentially get very ugly very quickly. > > Using stock Apache on the current Ubuntu server, using yield produced a > response error What error? If you aren't going to debug it enough to even work out what the error is in the browser or Apache error logs and post it here for comment so can say what may be wrong on your system, then we cant exactly help you much can we. > and using write() (over the telnet interface) returned the 0 > only and disconnected. Similar behavior in Firefox. All the scripts I provided you are conforming WSGI applications and work on mod_wsgi. If you are having issues, then it is likely going to be the way your Apache/Python is setup or how you configured mod_wsgi to host the scripts. Again, because you are providing no details about how you configured mod_wsgi we cant help you work out what is wrong with your system. > How odd that nobody's come up with a simple streaming/update schema (at > least to my mind). For response content they have and it can be made to work. Just because you cant get it working or don't understand what we are saying about the need to use a JavaScript/AJAX type client (eg. comet style) to make use of it as opposed to trying to rely on browser functionality that doesn't exist doesn't change that. Request content streaming is a different matter as I will explain below but you haven't even mentioned that as yet that I can see. > It would have been nice to be able to provide some kind of in-stream > feedback for long running jobs, but it looks like I'm going to have to > abandon that approach. The only issue with either of the other solutions is > that each subsequent request depends on data provided by the prior, so the > amount of traffic going back & forth could potentially become a problem. > > Alternatively I could simply create a session database that saves the > required objects then each subsequent request simply fetches the required > one from the table and... > > Well, you can see why streaming seemed like such a simple solution! Back to > the drawing board, as it were. I'll try one last time to try and summarise a few issues for you, although based on your attitude so far, I don't think it will change your opinion or help your understanding. 1. Streaming of responses from a WSGI application works fine using either yield or write(). If it doesn't work for a specific WSGI hosting mechanism then that implementation may not be conforming to WSGI requirements. Specifically, between a yield and/or write() it is required that an implicit flush is performed. This should ensure that the data is written to the HTTP client connection and/or ensure that the return of such data to the client occurs in parallel to further actions occurring in that request. 2. A WSGI middleware that caches response data can stuff this up. One cant outright prohibit a WSGI middleware holding on to response data, albeit that for each yield or write() technically it is supposed to still pass on at least an empty string down the chain so as to allow control to get back to the underlying WSGI implementation, which may uses such windows to swap what request context it is operating on so as to allow a measure of concurrency in situation where threads may not be getting used. 3. Where a WSGI adapter on top of an existing web server is used, eg. various options that exist with Apache and nginx, then an output filter configured into the web server may also stuff this up. For example, an output filter that compresses response data may buffer up response data into large blocks before compressing them and returning them. 4. Although response content can be streamed subject to above caveats, streaming of request content is a totally different matter. First off, WSGI requires that the request content have a Content-Length specified. Thus technically a HTTP client cant leave out Content-Length and instead used chunked request content. Further, the way in which many web servers and WSGI servers are implemented would prohibit streaming of request content anyway. This is because many implementations, especially where proxying occurs, eg. cgi, fastcgi, scgi, ajp, uwsgi, mod_proxy (??), and mod_wsgi daemon mode, expect that the whole request content can be read in and written across the proxy connection before any attempt is made to start reading any data returned from the web application. The request content therefore cannot be open ended in length because most implementations will never switch from reading that content to expecting response from the application. Thus it isn't possible to use WSGI as both way streaming mechanism where some request content is written, some response content returned and then the client sends more request content based on that etc etc. So what does this all mean. First up is that response content streaming should be able to be made to work, however since request content streaming isn't technically allowed with WSGI, if you need that you are out of luck if you want to conform to WSGI specification. Second, you can however with mod_wsgi embedded mode slightly step outside of strict WSGI conformance and have request content streaming. You are then bound to Apache/mod_wsgi, but whether you want to do that is debatable for reasons below. The bigger problem with both way streaming or long polling applications which use the same HTTP request is that WSGI servers tend to use processes and threads for concurrency. When you use this mechanisms they will tie up a process or thread for the whole time. Thus if you have lots of concurrent request you need huge numbers of processes and/or threads, which just isn't usually practical because of resource usage such as memory. For that reason, one would instead on the server usually use special purpose web servers for these types of applications and use HTTP directly and avoid WSGI, due to WSGI blocking nature. Instead these servers would use an event driven system model or other system which allows concurrency without requiring a process or thread per application. In short, this is what Comet and dedicated servers for that are about. Allowing large numbers of concurrent long requests with minimal resources. That they are dedicated systems also allows them to avoid limitations in other high level web application interfaces such as CGI, FASTCGI, SCGI, AJP etc which have an expectation that can read whole request content before trying to deal with any response from a web application that is handling the requests. Anyway, hopefully that explains things better. You can do what you want, you just need to select the correct tool for the job. Graham From aaron.fransen at gmail.com Mon Jul 5 14:43:44 2010 From: aaron.fransen at gmail.com (Aaron Fransen) Date: Mon, 5 Jul 2010 06:43:44 -0600 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: Apologies Graham, I'm not actually trying to appear dense but clearly I'm not one of the world's bright lights when it comes to web interfaces. My installation is literally a base installation of the latest Ubuntu server platform. The only configuration at play is this: WSGIDaemonProcess node9 user=www-data group=www-data processes=2 threads=25 WSGIProcessGroup node9 WSGIScriptAlias /run /var/www/run/run.py The error that occurs when using telnet and yield is: [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): Target WSGI script '/var/www/run/run.py' cannot be loaded as Python module. [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): Exception occurred processing WSGI script '/var/www/run/run.py'. [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] SyntaxError: 'return' with argument inside generator (run.py, line 14) using this code: status = '200 OK' response_headers = [('Content-type','text/plain')] start_response(status, response_headers) for x in range(0,10): yield 'hey %s' % x time.sleep(1) The error occurs when I use "return []" as opposed to simply "return", however I now see that is a result of the yield command itself. Using this method, the telnet interface returns immediately with: HTTP/1.1 200 OK Date: Mon, 05 Jul 2010 12:30:45 GMT Server: Apache/2.2.14 (Ubuntu) Vary: Accept-Encoding Connection: close Content-Type: text/plain 0 Connection closed by foreign host. In fact, whether using yield or write produces the same result. If I'm not getting the results I should be, then obviously I'm doing something wrong. I understand the danger of having a long-running web process (hence the reason I have a lot of virtual machines in the live environment using mod_python right now) but unfortunately it's something I don't seem to be able to work around at the moment. Thanks to all. On Wed, Jun 30, 2010 at 5:19 PM, Graham Dumpleton < graham.dumpleton at gmail.com> wrote: > On 30 June 2010 22:55, Aaron Fransen wrote: > > > > I can see that this could potentially get very ugly very quickly. > > > > Using stock Apache on the current Ubuntu server, using yield produced a > > response error > > What error? If you aren't going to debug it enough to even work out > what the error is in the browser or Apache error logs and post it here > for comment so can say what may be wrong on your system, then we cant > exactly help you much can we. > > > and using write() (over the telnet interface) returned the 0 > > only and disconnected. Similar behavior in Firefox. > > All the scripts I provided you are conforming WSGI applications and > work on mod_wsgi. If you are having issues, then it is likely going to > be the way your Apache/Python is setup or how you configured mod_wsgi > to host the scripts. Again, because you are providing no details about > how you configured mod_wsgi we cant help you work out what is wrong > with your system. > > > How odd that nobody's come up with a simple streaming/update schema (at > > least to my mind). > > For response content they have and it can be made to work. Just > because you cant get it working or don't understand what we are saying > about the need to use a JavaScript/AJAX type client (eg. comet style) > to make use of it as opposed to trying to rely on browser > functionality that doesn't exist doesn't change that. Request content > streaming is a different matter as I will explain below but you > haven't even mentioned that as yet that I can see. > > > It would have been nice to be able to provide some kind of in-stream > > feedback for long running jobs, but it looks like I'm going to have to > > abandon that approach. The only issue with either of the other solutions > is > > that each subsequent request depends on data provided by the prior, so > the > > amount of traffic going back & forth could potentially become a problem. > > > > Alternatively I could simply create a session database that saves the > > required objects then each subsequent request simply fetches the required > > one from the table and... > > > > Well, you can see why streaming seemed like such a simple solution! Back > to > > the drawing board, as it were. > > I'll try one last time to try and summarise a few issues for you, > although based on your attitude so far, I don't think it will change > your opinion or help your understanding. > > 1. Streaming of responses from a WSGI application works fine using > either yield or write(). If it doesn't work for a specific WSGI > hosting mechanism then that implementation may not be conforming to > WSGI requirements. Specifically, between a yield and/or write() it is > required that an implicit flush is performed. This should ensure that > the data is written to the HTTP client connection and/or ensure that > the return of such data to the client occurs in parallel to further > actions occurring in that request. > > 2. A WSGI middleware that caches response data can stuff this up. One > cant outright prohibit a WSGI middleware holding on to response data, > albeit that for each yield or write() technically it is supposed to > still pass on at least an empty string down the chain so as to allow > control to get back to the underlying WSGI implementation, which may > uses such windows to swap what request context it is operating on so > as to allow a measure of concurrency in situation where threads may > not be getting used. > > 3. Where a WSGI adapter on top of an existing web server is used, eg. > various options that exist with Apache and nginx, then an output > filter configured into the web server may also stuff this up. For > example, an output filter that compresses response data may buffer up > response data into large blocks before compressing them and returning > them. > > 4. Although response content can be streamed subject to above caveats, > streaming of request content is a totally different matter. First off, > WSGI requires that the request content have a Content-Length > specified. Thus technically a HTTP client cant leave out > Content-Length and instead used chunked request content. Further, the > way in which many web servers and WSGI servers are implemented would > prohibit streaming of request content anyway. This is because many > implementations, especially where proxying occurs, eg. cgi, fastcgi, > scgi, ajp, uwsgi, mod_proxy (??), and mod_wsgi daemon mode, expect > that the whole request content can be read in and written across the > proxy connection before any attempt is made to start reading any data > returned from the web application. The request content therefore > cannot be open ended in length because most implementations will never > switch from reading that content to expecting response from the > application. Thus it isn't possible to use WSGI as both way streaming > mechanism where some request content is written, some response content > returned and then the client sends more request content based on that > etc etc. > > So what does this all mean. First up is that response content > streaming should be able to be made to work, however since request > content streaming isn't technically allowed with WSGI, if you need > that you are out of luck if you want to conform to WSGI specification. > Second, you can however with mod_wsgi embedded mode slightly step > outside of strict WSGI conformance and have request content streaming. > You are then bound to Apache/mod_wsgi, but whether you want to do that > is debatable for reasons below. > > The bigger problem with both way streaming or long polling > applications which use the same HTTP request is that WSGI servers tend > to use processes and threads for concurrency. When you use this > mechanisms they will tie up a process or thread for the whole time. > Thus if you have lots of concurrent request you need huge numbers of > processes and/or threads, which just isn't usually practical because > of resource usage such as memory. For that reason, one would instead > on the server usually use special purpose web servers for these types > of applications and use HTTP directly and avoid WSGI, due to WSGI > blocking nature. Instead these servers would use an event driven > system model or other system which allows concurrency without > requiring a process or thread per application. > > In short, this is what Comet and dedicated servers for that are about. > Allowing large numbers of concurrent long requests with minimal > resources. That they are dedicated systems also allows them to avoid > limitations in other high level web application interfaces such as > CGI, FASTCGI, SCGI, AJP etc which have an expectation that can read > whole request content before trying to deal with any response from a > web application that is handling the requests. > > Anyway, hopefully that explains things better. You can do what you > want, you just need to select the correct tool for the job. > > Graham > -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis+ml-python-web at subspacefield.org Tue Jul 6 01:41:20 2010 From: travis+ml-python-web at subspacefield.org (travis+ml-python-web at subspacefield.org) Date: Mon, 5 Jul 2010 16:41:20 -0700 Subject: [Web-SIG] Python Web Application & Programming Pages Message-ID: <20100705234119.GA4544@subspacefield.org> So glad I found this group! Been looking for a relevant community. http://www.subspacefield.org/~travis/python_web_authoring_programming.html Meant to answer questions such as "what tools should I use?" Refers to both HTML generation and web app development. Several weeks of work have gone into it. Comments/contributions VERY welcome. HTH, Travis -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email john at subspacefield.org to get blacklisted. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From graham.dumpleton at gmail.com Tue Jul 6 12:50:00 2010 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Tue, 6 Jul 2010 20:50:00 +1000 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: On 5 July 2010 22:43, Aaron Fransen wrote: > Apologies Graham, I'm not actually trying to appear dense but clearly I'm > not one of the world's bright lights when it comes to web interfaces. > > My installation is literally a base installation of the latest Ubuntu server > platform. The only configuration at play is this: > > ??? WSGIDaemonProcess node9 user=www-data group=www-data processes=2 > threads=25 > ??? WSGIProcessGroup node9 > ??? WSGIScriptAlias /run /var/www/run/run.py > > The error that occurs when using telnet and yield is: > > [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): > Target WSGI script '/var/www/run/run.py' cannot be loaded as Python module. > [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] mod_wsgi (pid=2716): > Exception occurred processing WSGI script '/var/www/run/run.py'. > [Mon Jul 05 06:30:24 2010] [error] [client 127.0.0.1] SyntaxError: 'return' > with argument inside generator (run.py, line 14) > > using this code: > > ??? status??? =??? '200 OK' > ??? response_headers??? =??? [('Content-type','text/plain')] > ??? start_response(status, response_headers) > ??? for x in range(0,10): > ??? ??? yield 'hey %s' % x > ??? ??? time.sleep(1) > > The error occurs when I use "return []" as opposed to simply "return", > however I now see that is a result of the yield command itself. In the code example I posted I never had a 'return' statement in same function as 'yield'. You shouldn't be mixing the two. Graham > Using this method, the telnet interface returns immediately with: > > HTTP/1.1 200 OK > Date: Mon, 05 Jul 2010 12:30:45 GMT > Server: Apache/2.2.14 (Ubuntu) > Vary: Accept-Encoding > Connection: close > Content-Type: text/plain > > 0 > Connection closed by foreign host. > > In fact, whether using yield or write produces the same result. > > If I'm not getting the results I should be, then obviously I'm doing > something wrong. > > I understand the danger of having a long-running web process (hence the > reason I have a lot of virtual machines in the live environment using > mod_python right now) but unfortunately it's something I don't seem to be > able to work around at the moment. > > Thanks to all. > > On Wed, Jun 30, 2010 at 5:19 PM, Graham Dumpleton > wrote: >> >> On 30 June 2010 22:55, Aaron Fransen wrote: >> > >> > I can see that this could potentially get very ugly very quickly. >> > >> > Using stock Apache on the current Ubuntu server, using yield produced a >> > response error >> >> What error? If you aren't going to debug it enough to even work out >> what the error is in the browser or Apache error logs and post it here >> for comment so can say what may be wrong on your system, then we cant >> exactly help you much can we. >> >> > and using write() (over the telnet interface) returned the 0 >> > only and disconnected. Similar behavior in Firefox. >> >> All the scripts I provided you are conforming WSGI applications and >> work on mod_wsgi. If you are having issues, then it is likely going to >> be the way your Apache/Python is setup or how you configured mod_wsgi >> to host the scripts. Again, because you are providing no details about >> how you configured mod_wsgi we cant help you work out what is wrong >> with your system. >> >> > How odd that nobody's come up with a simple streaming/update schema (at >> > least to my mind). >> >> For response content they have and it can be made to work. Just >> because you cant get it working or don't understand what we are saying >> about the need to use a JavaScript/AJAX type client (eg. comet style) >> to make use of it as opposed to trying to rely on browser >> functionality that doesn't exist doesn't change that. Request content >> streaming is a different matter as I will explain below but you >> haven't even mentioned that as yet that I can see. >> >> > It would have been nice to be able to provide some kind of in-stream >> > feedback for long running jobs, but it looks like I'm going to have to >> > abandon that approach. The only issue with either of the other solutions >> > is >> > that each subsequent request depends on data provided by the prior, so >> > the >> > amount of traffic going back & forth could potentially become a problem. >> > >> > Alternatively I could simply create a session database that saves the >> > required objects then each subsequent request simply fetches the >> > required >> > one from the table and... >> > >> > Well, you can see why streaming seemed like such a simple solution! Back >> > to >> > the drawing board, as it were. >> >> I'll try one last time to try and summarise a few issues for you, >> although based on your attitude so far, I don't think it will change >> your opinion or help your understanding. >> >> 1. Streaming of responses from a WSGI application works fine using >> either yield or write(). If it doesn't work for a specific WSGI >> hosting mechanism then that implementation may not be conforming to >> WSGI requirements. Specifically, between a yield and/or write() it is >> required that an implicit flush is performed. This should ensure that >> the data is written to the HTTP client connection and/or ensure that >> the return of such data to the client occurs in parallel to further >> actions occurring in that request. >> >> 2. A WSGI middleware that caches response data can stuff this up. One >> cant outright prohibit a WSGI middleware holding on to response data, >> albeit that for each yield or write() technically it is supposed to >> still pass on at least an empty string down the chain so as to allow >> control to get back to the underlying WSGI implementation, which may >> uses such windows to swap what request context it is operating on so >> as to allow a measure of concurrency in situation where threads may >> not be getting used. >> >> 3. Where a WSGI adapter on top of an existing web server is used, eg. >> various options that exist with Apache and nginx, then an output >> filter configured into the web server may also stuff this up. For >> example, an output filter that compresses response data may buffer up >> response data into large blocks before compressing them and returning >> them. >> >> 4. Although response content can be streamed subject to above caveats, >> streaming of request content is a totally different matter. First off, >> WSGI requires that the request content have a Content-Length >> specified. Thus technically a HTTP client cant leave out >> Content-Length and instead used chunked request content. Further, the >> way in which many web servers and WSGI servers are implemented would >> prohibit streaming of request content anyway. This is because many >> implementations, especially where proxying occurs, eg. cgi, fastcgi, >> scgi, ajp, uwsgi, mod_proxy (??), and mod_wsgi daemon mode, expect >> that the whole request content can be read in and written across the >> proxy connection before any attempt is made to start reading any data >> returned from the web application. The request content therefore >> cannot be open ended in length because most implementations will never >> switch from reading that content to expecting response from the >> application. Thus it isn't possible to use WSGI as both way streaming >> mechanism where some request content is written, some response content >> returned and then the client sends more request content based on that >> etc etc. >> >> So what does this all mean. First up is that response content >> streaming should be able to be made to work, however since request >> content streaming isn't technically allowed with WSGI, if you need >> that you are out of luck if you want to conform to WSGI specification. >> Second, you can however with mod_wsgi embedded mode slightly step >> outside of strict WSGI conformance and have request content streaming. >> You are then bound to Apache/mod_wsgi, but whether you want to do that >> is debatable for reasons below. >> >> The bigger problem with both way streaming or long polling >> applications which use the same HTTP request is that WSGI servers tend >> to use processes and threads for concurrency. When you use this >> mechanisms they will tie up a process or thread for the whole time. >> Thus if you have lots of concurrent request you need huge numbers of >> processes and/or threads, which just isn't usually practical because >> of resource usage such as memory. For that reason, one would instead >> on the server usually use special purpose web servers for these types >> of applications and use HTTP directly and avoid WSGI, due to WSGI >> blocking nature. Instead these servers would use an event driven >> system model or other system which allows concurrency without >> requiring a process or thread per application. >> >> In short, this is what Comet and dedicated servers for that are about. >> Allowing large numbers of concurrent long requests with minimal >> resources. That they are dedicated systems also allows them to avoid >> limitations in other high level web application interfaces such as >> CGI, FASTCGI, SCGI, AJP etc which have an expectation that can read >> whole request content before trying to deal with any response from a >> web application that is handling the requests. >> >> Anyway, hopefully that explains things better. You can do what you >> want, you just need to select the correct tool for the job. >> >> Graham > > From dirkjan at ochtman.nl Tue Jul 6 13:02:32 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 6 Jul 2010 13:02:32 +0200 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: On Tue, Jul 6, 2010 at 12:50, Graham Dumpleton wrote: > In the code example I posted I never had a 'return' statement in same > function as 'yield'. You shouldn't be mixing the two. Well, you can still use bare "return" as a way of raising StopIteration. Cheers, Dirkjan From graham.dumpleton at gmail.com Tue Jul 6 13:22:23 2010 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Tue, 6 Jul 2010 21:22:23 +1000 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: On 6 July 2010 21:02, Dirkjan Ochtman wrote: > On Tue, Jul 6, 2010 at 12:50, Graham Dumpleton > wrote: >> In the code example I posted I never had a 'return' statement in same >> function as 'yield'. You shouldn't be mixing the two. > > Well, you can still use bare "return" as a way of raising StopIteration. True. It was just easier to say not to and avoid any chance of misunderstanding. :-) Graham "There's something very important I forgot to tell you. Don't cross the streams? It would be bad? Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light." ?Egon Spengler on crossing proton streams From aaron.fransen at gmail.com Wed Jul 7 15:20:33 2010 From: aaron.fransen at gmail.com (Aaron Fransen) Date: Wed, 7 Jul 2010 07:20:33 -0600 Subject: [Web-SIG] Emulating req.write() in WSGI In-Reply-To: References:

Message-ID: Well, just in case this helps anyone, I managed to get it working on most of the current browsers. My first problem was a missing import statement (definite "DOH!" moment there), but once that was resolved Firefox steadfastly refused to co-operate. The fix was to use a boundary indicator then specify the content type for every subsequent chunk of data, something along these lines: # during initial converstation status = '200 OK' response_headers = [('Content-type','multipart/x-mixed-replace')] if string.find(environ['HTTP_USER_AGENT'],'Firefox') != -1: response_headers = [('Content-type','multipart/x-mixed-replace;boundary=x0x0x0x')] writer = start_response(status, response_headers) # during subsequent conversations if string.find(environ['HTTP_USER_AGENT'],'Firefox') != -1: writer('Content-type: text/html\r\n\r\n'+text+'\r\n\r\n--x0x0x0x') else: writer(text) Now, in Firefox the behavior is that it completely replaces the previous chunk, while in IE8, Chrome, Safari it simply adds to the existing content. It doesn't work in Opera yet (with either method), I haven't been able to determine why but I'll continue to work on it. Sorry for the hassle everyone! -------------- next part -------------- An HTML attachment was scrubbed... URL: From moneyonlinemail85 at gmail.com Tue Jul 13 17:24:20 2010 From: moneyonlinemail85 at gmail.com (Sunday Ojelabi) Date: Tue, 13 Jul 2010 16:24:20 +0100 Subject: [Web-SIG] Post list Message-ID: Kindly find of what is new on this site........www.moneymakmachine.blogspot.com From faassen at startifact.com Tue Jul 13 18:57:18 2010 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 13 Jul 2010 18:57:18 +0200 Subject: [Web-SIG] hurry.resource Message-ID: Hi there, I thought this might be a nice place to bring hurry.resource to the attention of more people. hurry.resource is a way to package Javascript (and CSS) libraries as Python libraries. It can be compared with client-side javascript-based resource handling libraries such as the YUI loader. hurry.resource is server-based however. hurry.resource is designed to be web-framework agnostic. I've recently released version 0.9 which includes a simple WSGI middleware that can help to integrate this into your application or framework. What can you do with hurry.resource? We've wrapped a whole selection of Javascript libraries using hurry.resource, such as YUI (hurry.yui), jquery (hurry.jquery), jquery-ui (hurry.jquery-ui), ExtJS (hurry.extjs), TinyMC (hurry.tinymce) and more. There are all available on PyPI. To use jquery for instance you could add ``hurry.jquery`` to your project's setup.py. When installing your project hurry.jquery will then be installed. This contains both the jquery library resources as well as a description of them. In your application you can then do: from hurry.jquery import jquery and then, when you generate a page that needs jquery support on it, you say this somewhere in the Python code that gets executed during (or before) page rendering: jquery.need() What this will do is automatically include a