From pje at telecommunity.com Sun Apr 1 06:36:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 31 Mar 2007 23:36:04 -0500 Subject: [Web-SIG] Rack: WSGI on Ruby Message-ID: <5.1.1.6.0.20070331232810.0295dad8@sparrow.telecommunity.com> http://rack.rubyforge.org/ Interestingly, they take the approach we're talking about for WSGI 2.0, as far as a call signature taking one argument and returning three. The spec is brief and to the point, in part because of that simplicity, but also because they're ignoring HTTP 1.1 and issues like our read() problems with the input stream. They're also ignoring flow control, push, asynchronous servers, etc. Actually, the whole thing seems pretty underspecified, but then they have some advantages we didn't. At the time WSGI was created, depending on an external library like wsgiref that wasn't in the stdlib was an almost unheard-of luxury. Anyway, the fact that they can define the thing in terms of a canonical implementation, *and* don't have to worry as much about backward compatibility with a wide variety of web frameworks, means that they can probably actually get away with it. From cce at clarkevans.com Tue Apr 3 00:05:29 2007 From: cce at clarkevans.com (Clark C. Evans) Date: Mon, 2 Apr 2007 18:05:29 -0400 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> Message-ID: <20070402220529.GA44827@personal.applied-epistemics.com> On Fri, Mar 30, 2007 at 11:10:17AM +1000, Graham Dumpleton wrote: | One other issue if aiming at supporting chunked encoding for a request, | is how (if one even can) make available the trailing headers if present | after the final null data block. Personally I am not sure this one is | worth the trouble and may be quite hard to even implement with some web | servers as they don't even provide them as a separate set of headers but | simply merge them on top of the main request headers. In my particular application it'd be quite helpful if I could return Chunked-Body responses (especially ones with additional trailing headers). Since WSGI rules the use of transfer encodings, then it should have a mechanism to request this sort of behavior aka, yield a "flush" object that has optional trailing headers that should be included in the chunk that is returned. Best, Clark From graham.dumpleton at gmail.com Fri Apr 6 04:08:17 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 6 Apr 2007 12:08:17 +1000 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <20070402220529.GA44827@personal.applied-epistemics.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> Message-ID: <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> On 03/04/07, Clark C. Evans wrote: > On Fri, Mar 30, 2007 at 11:10:17AM +1000, Graham Dumpleton wrote: > | One other issue if aiming at supporting chunked encoding for a request, > | is how (if one even can) make available the trailing headers if present > | after the final null data block. Personally I am not sure this one is > | worth the trouble and may be quite hard to even implement with some web > | servers as they don't even provide them as a separate set of headers but > | simply merge them on top of the main request headers. > > In my particular application it'd be quite helpful if I could return > Chunked-Body responses (especially ones with additional trailing > headers). Since WSGI rules the use of transfer encodings, then it should > have a mechanism to request this sort of behavior aka, yield a "flush" > object that has optional trailing headers that should be included in the > chunk that is returned. Do you know of any actual web servers that provide a facility for sending trailers after the final null chunk in a response? Apache for one doesn't provide a way of doing this: /* RFC 2616, Section 3.6.1 * * If there is an EOS bucket, then prefix it with: * 1) the last-chunk marker ("0" CRLF) * 2) the trailer * 3) the end-of-chunked body CRLF * * We only do this if we have not seen an error bucket with * status HTTP_BAD_GATEWAY. We have memorized an * error bucket that we had seen in the filter context. * The error bucket with status HTTP_BAD_GATEWAY indicates that the * connection to the backend (mod_proxy) broke in the middle of the * response. In order to signal the client that something went wrong * we do not create the last-chunk marker and set c->keepalive to * AP_CONN_CLOSE in the core output filter. * * XXX: it would be nice to combine this with the end-of-chunk * marker above, but this is a bit more straight-forward for * now. */ if (eos && !f->ctx) { /* XXX: (2) trailers ... does not yet exist */ e = apr_bucket_immortal_create(ASCII_ZERO ASCII_CRLF /* */ ASCII_CRLF, 5, c->bucket_alloc); APR_BUCKET_INSERT_BEFORE(eos, e); } Note the comment that step 2, sending of trailers doesn't yet exist. So even if WSGI had a way of supplying trailers, I doubt there would be many WSGI adapters for web servers that could actually implement it. FWIW, in mod_wsgi I have now added a directive which allows one to enable within a specific context that chunked transfer encoding should be used for a response when a HTTP/1.1 client is being used. Thus, if you know the content generated from a resource at a particular URL within your application would benefit from chunked transfer encoding on the response, you could use: WSGIOutputChunking On At this stage is probably better than nothing given that WSGI doesn't provide a way of enabling it. Graham From foom at fuhm.net Fri Apr 6 04:17:03 2007 From: foom at fuhm.net (James Y Knight) Date: Thu, 5 Apr 2007 22:17:03 -0400 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> Message-ID: <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> On Apr 5, 2007, at 10:08 PM, Graham Dumpleton wrote: > FWIW, in mod_wsgi I have now added a directive which allows one to > enable within a specific context that chunked transfer encoding should > be used for a response when a HTTP/1.1 client is being used. Thus, if > you know the content generated from a resource at a particular URL > within your application would benefit from chunked transfer encoding > on the response, you could use: > > > WSGIOutputChunking On > > > At this stage is probably better than nothing given that WSGI doesn't > provide a way of enabling it. What's the point of a switch? If the app didn't provide a content- length, and you can't otherwise determine a content-length (because for example the result is a simple list of strings), just do chunked encoding output automatically. James From graham.dumpleton at gmail.com Fri Apr 6 04:52:25 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 6 Apr 2007 12:52:25 +1000 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> Message-ID: <88e286470704051952w639f9656hef0c9c87eae84f83@mail.gmail.com> On 06/04/07, James Y Knight wrote: > > On Apr 5, 2007, at 10:08 PM, Graham Dumpleton wrote: > > FWIW, in mod_wsgi I have now added a directive which allows one to > > enable within a specific context that chunked transfer encoding should > > be used for a response when a HTTP/1.1 client is being used. Thus, if > > you know the content generated from a resource at a particular URL > > within your application would benefit from chunked transfer encoding > > on the response, you could use: > > > > > > WSGIOutputChunking On > > > > > > At this stage is probably better than nothing given that WSGI doesn't > > provide a way of enabling it. > > What's the point of a switch? If the app didn't provide a content- > length, and you can't otherwise determine a content-length (because > for example the result is a simple list of strings), just do chunked > encoding output automatically. To a degree that may make sense, but to my mind, the implementation of a lower level layer beneath the application, which an application writer may have little real control over, should not really be making an arbitrary decision on behalf of the user to impose such a behaviour. It should always be up to the application writer to make the decision. That said, it may be worthwhile to actually implement the directive to honour Off (default), On or Auto. Thus On is always use chunked transfer encoding and Auto would be what you describe. Which approach is used though should still be the application writers choice. One good thing your comment highlights is that when we come to working out how in future WSGI specification an application could control use of chunked transfer encoding, that a simple On and Off may not be enough and that having an Auto like switch may also be advantageous in certain cases. Graham From foom at fuhm.net Fri Apr 6 05:54:05 2007 From: foom at fuhm.net (James Y Knight) Date: Thu, 5 Apr 2007 23:54:05 -0400 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <88e286470704051952w639f9656hef0c9c87eae84f83@mail.gmail.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> <88e286470704051952w639f9656hef0c9c87eae84f83@mail.gmail.com> Message-ID: On Apr 5, 2007, at 10:52 PM, Graham Dumpleton wrote: > On 06/04/07, James Y Knight wrote: >> What's the point of a switch? If the app didn't provide a content- >> length, and you can't otherwise determine a content-length (because >> for example the result is a simple list of strings), just do chunked >> encoding output automatically. > > To a degree that may make sense, but to my mind, the implementation of > a lower level layer beneath the application, which an application > writer may have little real control over, should not really be making > an arbitrary decision on behalf of the user to impose such a > behaviour. It should always be up to the application writer to make > the decision. > That said, it may be worthwhile to actually implement the directive to > honour Off (default), On or Auto. Thus On is always use chunked > transfer encoding and Auto would be what you describe. Which approach > is used though should still be the application writers choice. But you didn't answer the question: what is the _point_? When do you ever want to require a connection close rather than use chunking ("Off")? Why would you ever want to use chunking if you already know the content length ("On"). Those switches seem wholly useless. No addition or extension to WSGI needed... James From graham.dumpleton at gmail.com Fri Apr 6 08:41:29 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 6 Apr 2007 16:41:29 +1000 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> <88e286470704051952w639f9656hef0c9c87eae84f83@mail.gmail.com> Message-ID: <88e286470704052341p5ab576fcqe3f118c2da62ae3e@mail.gmail.com> On 06/04/07, James Y Knight wrote: > > On Apr 5, 2007, at 10:52 PM, Graham Dumpleton wrote: > > > On 06/04/07, James Y Knight wrote: > >> What's the point of a switch? If the app didn't provide a content- > >> length, and you can't otherwise determine a content-length (because > >> for example the result is a simple list of strings), just do chunked > >> encoding output automatically. > > > > To a degree that may make sense, but to my mind, the implementation of > > a lower level layer beneath the application, which an application > > writer may have little real control over, should not really be making > > an arbitrary decision on behalf of the user to impose such a > > behaviour. It should always be up to the application writer to make > > the decision. > > > That said, it may be worthwhile to actually implement the directive to > > honour Off (default), On or Auto. Thus On is always use chunked > > transfer encoding and Auto would be what you describe. Which approach > > is used though should still be the application writers choice. > > But you didn't answer the question: what is the _point_? When do you > ever want to require a connection close rather than use chunking > ("Off")? Why would you ever want to use chunking if you already know > the content length ("On"). Those switches seem wholly useless. > > No addition or extension to WSGI needed... Am I take this then that you believe or are proposing that WSGI 2.0 should require that if no content length is provided in a response and it can' be calculated, that a WSGI adapter should ensure the web server uses chunked transfer encoding? Since the original point of this message thread was to determine what WSGI 2.0 should be, it would help if you are clear on what you believe should be specified by WSGI 2.0 in this respect. Personally I don't understand why having a choice is a problem. Since the default for Apache and probably many other web servers is not to use chunked transfer encoding, it would seem logical that one still has a choice that chunked transfer encoding not be used. The logical opposite to off is for chunked transfer encoding to always be used. As you have pointed out there is also a logical middle ground where the content length is actually provided. Why shouldn't a user have a choice as to how they want it to behave. A user may have very good reasons that they want it to behave in a certain way and as a provider of a piece of middleware software I should be allowing them access to all options. I should not be limiting choices. Thus from where I stand I don't see that there is a need to answer the questions as I am not the end user and can't say why a user would want to do it a specific way nor rule out that any particular way may be invalid. Anyway, the whole point of adding the option in the first place was merely to get around the current shortcoming of WSGI 1.0 that it doesn't define whether chunked transfer encoding should be used nor provide a way for an application to control it. Thus it was convenience for users who felt they needed to be able to control this. I could have just said 'touch luck' and tell users to wait until WSGI 2.0 is out and says something about it. I guess one can't make everyone happy though, but if this is going to be seen as such a big issue, I'd sooner remove the option totally and users can just put up with the limitation as it stands with WSGI 1.0 of not being able to use chunked transfer encoding. Graham From cce at clarkevans.com Fri Apr 6 23:48:14 2007 From: cce at clarkevans.com (Clark C. Evans) Date: Fri, 6 Apr 2007 17:48:14 -0400 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> Message-ID: <20070406214814.GA19998@personal.applied-epistemics.com> On Fri, Apr 06, 2007 at 12:08:17PM +1000, Graham Dumpleton wrote: | On 03/04/07, Clark C. Evans wrote: | >On Fri, Mar 30, 2007 at 11:10:17AM +1000, Graham Dumpleton wrote: | >| One other issue if aiming at supporting chunked encoding for a request, | >| is how (if one even can) make available the trailing headers if present | >| after the final null data block. Personally I am not sure this one is | >| worth the trouble and may be quite hard to even implement with some web | >| servers as they don't even provide them as a separate set of headers but | >| simply merge them on top of the main request headers. | > | >In my particular application it'd be quite helpful if I could return | >Chunked-Body responses (especially ones with additional trailing | >headers). Since WSGI rules the use of transfer encodings, then it should | >have a mechanism to request this sort of behavior aka, yield a "flush" | >object that has optional trailing headers that should be included in the | >chunk that is returned. | | Do you know of any actual web servers that provide a facility for | sending trailers after the final null chunk in a response? Apache for | one doesn't provide a way of doing this I'm starting to add support for 14.46 "Warning" headers [1] to my server application and custom client. Often times a warning might happen *after* the 200 has been sent; buffering isn't an option unfortunately. So, I need to find a way to send this additional information to the client. Most headers are kinda senseless as trailing headers, but this one is particularly useful. [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.46 | Note the comment that step 2, sending of trailers doesn't yet exist. Sounds like Apache isn't a fully-compliant webserver ;) | So even if WSGI had a way of supplying trailers, I doubt there would | be many WSGI adapters for web servers that could actually implement | it. This doesn't mean our API shouldn't support it. It's a valid need and when I get time, I'd like to find a way to update paste's httpserver to support this sort of thing. | FWIW, in mod_wsgi I have now added a directive which allows one to | enable within a specific context that chunked transfer encoding should | be used for a response when a HTTP/1.1 client is being used. That's great ;) | At this stage is probably better than nothing given that WSGI doesn't | provide a way of enabling it. In your implementation, how would I indicate where the chunk boundaries should be? Or is this just a fixed size thing? Best, Clark From foom at fuhm.net Fri Apr 6 23:52:03 2007 From: foom at fuhm.net (James Y Knight) Date: Fri, 6 Apr 2007 17:52:03 -0400 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <88e286470704052341p5ab576fcqe3f118c2da62ae3e@mail.gmail.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> <88e286470704051952w639f9656hef0c9c87eae84f83@mail.gmail.com> <88e286470704052341p5ab576fcqe3f118c2da62ae3e@mail.gmail.com> Message-ID: <54683B8E-E828-4955-8398-10C97E5C0984@fuhm.net> On Apr 6, 2007, at 2:41 AM, Graham Dumpleton wrote: > Am I take this then that you believe or are proposing that WSGI 2.0 > should require that if no content length is provided in a response and > it can' be calculated, that a WSGI adapter should ensure the web > server uses chunked transfer encoding? Since the original point of > this message thread was to determine what WSGI 2.0 should be, it would > help if you are clear on what you believe should be specified by WSGI > 2.0 in this respect. No. I believe nothing about chunked encoding should be specified by WSGI 2.0 (the same as is currently in WSGI 1.0). It's completely up to the server whether and when it wants to used chunked transfer encoding on the output. I believe that a high quality server implementation will do what I outlined above, as a matter of course. But other than the overhead of re-establishing a connection after a forced-close from an undetermined-length response, there is no difference between using chunked or not. > I'd sooner remove the option totally and users can just put up > with the limitation as it stands with WSGI 1.0 of not being able to > use chunked transfer encoding. There is no such limitation. A server can use chunked transfer encoding whenever it wants to. James From graham.dumpleton at gmail.com Sat Apr 7 01:50:36 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 7 Apr 2007 09:50:36 +1000 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <20070406214814.GA19998@personal.applied-epistemics.com> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <20070406214814.GA19998@personal.applied-epistemics.com> Message-ID: <88e286470704061650p223bf9e2y69b5dc1dccaec8a0@mail.gmail.com> On 07/04/07, Clark C. Evans wrote: > On Fri, Apr 06, 2007 at 12:08:17PM +1000, Graham Dumpleton wrote: > | FWIW, in mod_wsgi I have now added a directive which allows one to > | enable within a specific context that chunked transfer encoding should > | be used for a response when a HTTP/1.1 client is being used. > > That's great ;) > > | At this stage is probably better than nothing given that WSGI doesn't > | provide a way of enabling it. > > In your implementation, how would I indicate where the chunk boundaries > should be? Or is this just a fixed size thing? In respect of requirements in WSGI specification for flushing blocks of data, mod_wsgi implements: """1. Send the entire block to the operating system (and request that any O/S buffers be flushed) before returning control to the application,""" In Apache, each time a flush is performed it will send all buffered data as a chunk. Thus, if you are using a generator as the iterable, a flush is performed after each block of data your application returns and so each block of data you yield becomes a chunk. Even were you to return a sequence as the iterable, a flush is performed after outputing each block in the sequence and thus each block in a sequence also becomes a chunk. Graham From graham.dumpleton at gmail.com Sat Apr 7 09:39:40 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 7 Apr 2007 17:39:40 +1000 Subject: [Web-SIG] WSGI 2.0 In-Reply-To: <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> References: <460C6044.2090602@colorstudy.com> <88e286470703291810n56593d3bq98ad7717f06b520e@mail.gmail.com> <20070402220529.GA44827@personal.applied-epistemics.com> <88e286470704051908u111dc057o25ef2b048b895865@mail.gmail.com> <0D772114-AC8B-4088-B603-E7EE1DD4C598@fuhm.net> Message-ID: <88e286470704070039l559ce2d8m193f4978b9d4f2ee@mail.gmail.com> On 06/04/07, James Y Knight wrote: > > On Apr 5, 2007, at 10:08 PM, Graham Dumpleton wrote: > > FWIW, in mod_wsgi I have now added a directive which allows one to > > enable within a specific context that chunked transfer encoding should > > be used for a response when a HTTP/1.1 client is being used. Thus, if > > you know the content generated from a resource at a particular URL > > within your application would benefit from chunked transfer encoding > > on the response, you could use: > > > > > > WSGIOutputChunking On > > > > > > At this stage is probably better than nothing given that WSGI doesn't > > provide a way of enabling it. > > What's the point of a switch? If the app didn't provide a content- > length, and you can't otherwise determine a content-length (because > for example the result is a simple list of strings), just do chunked > encoding output automatically. Okay, I'll freely admit I have been an arse over this issue. Apache does do what you expect but because of disabling lots of my Apache configuration at one point when doing some debugging, I managed to comment out a file that specified the KeepAlive option being set to On. As a result my Apache installation wasn't doing what you figure it should unless I enabled chunking from within the code. Lesson learned. Graham From ben at groovie.org Fri Apr 13 04:47:54 2007 From: ben at groovie.org (Ben Bangert) Date: Thu, 12 Apr 2007 19:47:54 -0700 Subject: [Web-SIG] ANN: Pylons 0.9.5 Released Message-ID: <9329E15F-3694-4BF3-84CF-128EF95EC560@groovie.org> I'm thrilled to announce a new release of Pylons, with one of the largest ticket fix counts of any Pylons release thus far. This release and its associated updates have only been possible with the growing and active Pylons community. A big shout-out to the many contributions by Phil Jenvey, Ian Bicking, James Gardner, Mike Orr, Shannon -jj Behrens, David Smith, Graham Higgins, and Wyatt Baldwin for their work on Pylons, Routes the mail list, and the documentation. Many thanks to the numerous others that have reported bugs and helped out on IRC and the mail list over the past few months. Also, as Pylons uses so many contributed projects, there's work going on to better integrate documentation across the smaller projects that make up the whole. A first stab at this is being taken with the Python Web Documentation effort which we realize not every single 'python web' project will be using; but its our hope that smaller WSGI based web components will put their docs up so they can all be utilized in a single place. The new documentation wiki: http://docs.pythonweb.org/ And onto update news for 0.9.5! There's big change logs for both Pylons and Routes, with some important updates in WebHelpers and Paste as well. First, the big improvements in 0.9.5, Unicode and i18n: - Routes now translates utf-8 characters during both URL generation, and URL recognition, many thanks to David Smith for numerous patches - The Request now has an option so that all GET/POST arguments come in as unicode, with the decoding you prefer - i18n now uses code contributed from Aquarium by Shannon -jj Behrens for translation fall-backs and has browser language matching code - Optional lazy translation options contributed by David Smith - Routes implicit defaults (action=index, id=None) can be turned off per route with _explicit=True - Routes memory can also be disabled by setting the mapper to explicit=True, which will also disable the implicit defaults Next, some backward compatibility changes: Webhelpers: * WARNING: paginate now takes arguments intended for the collection object as query_args. This could affect backwards compatibility. This fixes a common issue that non-keyword arguments passed into paginate get eaten by paginate's keyword arguments instead of being in *args to go on to the collection. * WARNING: Due to a typo, the Text helper highlight function no longer highlights text with the CSS class name 'hilight' by default: it now uses the CSS class name 'highlight' instead. The function's 'hilighter' keyword argument has also been deprecated, use 'highlighter' instead. Pylons: * WARNING: Pylons now requires the decorator module: it no longer packages it as pylons.decorator. Code relying on the pylons.decorator.decorator function will trigger a deprecation warning and should be changed to use decorator.decorator. * WARNING: pylons.h was deprecated for using projects' lib.helpers module directly in 0.9.3. pylons.h is now formally deprecated (emits DeprecationWarnings). Projects still accessing pylons.h must change the following import: from pylons import h to: import MYPROJ.lib.helpers as h And finally, the full changelog of the various parts that make Pylons what it is: PYLONS * Fixed paster shell breaking for projects where the base package was not the first package listed in top_level.txt. Patch from Alberto Valverde. Fixes #229. * Fixed doc references to config['app_conf']. Fixes #116. * Changed `get_engine_conf` to properly evaluate sqlalchemy echo statement when its 'debug'. Fixes #226. * make_session and create_engine now accept keyword arguments to pass to SQLAlchemy's create_engine. * make_session now accepts the keyword argument 'session_kwargs' to pass to SQLAlchemy's create_session. * Fixed _inspect_call to call function with keyword arguments instead of list args. Corrects issue with action defaults that caused the value for the latter args to be in the wrong spots. Spotted by Topher. Fixes #223. * Added the allow_none option (passed to xmlrpc.dumps) to XMLRPCController. Suggested by Jaroslaw Zabiello. * Updated XMLRPC Controller with patch for name lookup and additional unit tests for the patch. Fixes #216. * Updated docs for validate decorator to more clearly illustrate what the post_only args apply to. Fixes #221. * Added ability to return strings in the WSGIController. Fixes #218. * Added lazy i18n translation functions. Patch from David Smith. Fixes #181. * Added fix for XMLRPCController system.methodHelp function and unit test. Patch and unit test submitted by Graham Higgins. * Fixed bug in validate decorator with new UnicodeMultiDict response content not properly retaining form content as unicode for formencode's htmlfill. * Fixed bug in XMLRPC Controller with xmlrpclib Faults not being properly transformed into a WSGI response within the controller. * WARNING: Pylons now requires the decorator module: it no longer packages it as pylons.decorator. Code relying on the pylons.decorator.decorator function will trigger a deprecation warning and should be changed to use decorator.decorator. * WARNING: pylons.h was deprecated for using projects' lib.helpers module directly in 0.9.3. pylons.h is now formally deprecated (emits DeprecationWarnings). Projects still accessing pylons.h must change the following import: from pylons import h to: import MYPROJ.lib.helpers as h * pylons.jsonify and pylons.Controller references have been deprecated (they are misplaced references). They continue to be available at pylons.decorators.jsonify and pylons.controllers.Controller, as they always have been. * Updated templating Buffet to recognize format parameter and properly pass it to the template engine. * Updated LICENSE for new year and to indicate license covering templates generated. Fixes #188. * Interactive debugger now supports Mako. After r1780 if you are using a custom theme you will need to change '%(myghty_data)s' to '%(template_data)s' in your template. If you are using JavaScript the tab id is now "template_data". * Fixed bug in WSGIController with private function attempts not returning a valid WSGI response. * Added full unit test coverage of cache decorator. * Adding messages binary file, enabling i18n unit tests. Updating pylons.i18n to import LanguageError. Fixes #193. * Adding i18n tests, not active yet as they're waiting on a binary file from a patch. Refs #193. * Updated tests so that they now work with nose, removing py.test requirement. * Switching config setup to load keys into main config dict with app_conf and global_conf keys set for any code looking for those keys. Fixes #116. * PylonsInstaller is now the default paste.app_install entry point for new projects: this makes Cheetah no longer required for the paster make-config command. (Thanks Alexander Schremmer, Ian Bicking) * Added custom redirect_to function in pylons.helpers that will take an optional _response arg to pull headers and cookies out for preservation during a redirect. Fixes #136. * Changed config.Config.__init__ to take all options as keyword args so unused args can be skipped. Fixes #162. * The request object can now automatically decode GET/POST/params vars to unicode, when its charset attribute is set. * Added a new request_settings keyword arg to Config's constructor. Allows setting the default charset and errors values of of the request object. * Deprecated Config constructor's default_charset keyword arg. Use Config's response_settings keyword arg instead. * Fixed paster controller to test for lib.base and only add that import statement when its present. This fixes the controller template when used with minimal Pylons project templates. Fixes #140 and fixes #139. * Fixed the paster shell error: KeyError: 'pylons.routes_dict' when calling app.get and app.post. * Fixed paster shell not working on projects with names containing hyphens. * Fixed the config directive 'sqlalchemy.echo' set to False being interpreted as True. Patch by Alex Conrad. * Fixed paster shell not restoring CONFIG['global_conf']. ROUTES * Fixed matching so that an attempt to match an empty path raises a RouteException. Fixes #44. * Added ability to use characters in URL's such as '-' and '_' in map.resource. Patch by Wyatt Baldwin. Fixes #45. * Updated Mapper.resource handling with name_prefix and path_prefix checking to specify defaults. Also ensures that should either of them be set, they override the prefixes should parent_resource be specified. Patch by Wyatt Baldwin. Fixes #42. * Added utf-8 decoding of incoming path arguments, with fallback to ignoring them in the very rare cases a malformed request URL is sent. Patch from David Smith. * Fixed treatment of '#' character as something that can be left off and used in route paths. Found by Mike Orr. * Added ability to specify parent resource to map.resource command. Patch from Wyatt Baldwin. * Fixed formatted route issue with map.resource when additional collection methods are specified. Added unit tests to verify the collection methods work properly. * Updated URL parsing to properly use HTTP_HOST for hostname + port info before falling back to SERVER_PORT and SERVER_NAME. Fixes #43. * Added member_name and collection_name setting to Route object when made with map.resource. * Updated routes.middleware to make the Routes matched accessible as environ['routes.route']. * Updating mapper object to use thread local for request data (such as environ) and middleware now deletes environ references at the end of the request. * Added explicit option to Routes and Mapper. Routes _explicit setting will prevent the Route defaults from being implicitly set, while setting Mapper to explicit will prevent Route implicit defaults and stop url_for from using Route memory. Fixes #38. * Updated config object so that the route is attached if possible. * Adding standard logging usage with debug messages. * Added additional test for normal '.' match and fixed new special matching to match it properly. Thanks David Smith. * Fixed hanging special char issue with 'special' URL chars at the end of a URL that are missing the variable afterwards. * Changed Routes generation and recognition to handle other 'special' URL chars , . and ; as if they were /. This lets them be optionally left out of the resulting generated URL. Feature requested by David Smith. * Fixed lookahead assertion in regexp builder to properly handle two grouped patterns in a row. * Applied patch to generation and matching to handle Unicode characters properly. Reported with patch by David Smith. WEBHELPERS * WARNING: paginate now takes arguments intended for the collection object as query_args. This could affect backwards compatibility. This fixes a common issue that non-keyword arguments passed into paginate get eaten by paginate's keyword arguments instead of being in *args to go on to the collection. * Added environ checking with Routes so that page will be automatically pulled out of the query string, or from the Routes match dict if available. * Added ability for paginate to check for objects that had SQLAlchemy's assign_mapper applied to them. * Added better range checking to paginator to require a positive value that is less than the total amount of pages available for a page. * WARNING: Due to a typo, the Text helper highlight function no longer highlights text with the CSS class name 'hilight' by default: it now uses the CSS class name 'highlight' instead. The function's 'hilighter' keyword argument has also been deprecated, use 'highlighter' instead. * Fixed the broken markdown function. * Upgraded markdown from 1.5 to 1.6a. * Sync'd Prototype helper to 6057. * Sync'd Urls helper to 6070. * Sync'd Text helper to 6096. * Sync'd Date helper to 6080. * Sync'd Tags helper to 5857. * Sync'd Asset tag helper to 6057. * Sync'd Rails Number helper to 6045. * Updated Ajax commands to internally use 'with_' to avoid name conflicts with Python 2.5 and beyond. Reported by anilj. Fixes #190. * Applied patch from David Smith to decode URL parts as Routes does. Fixes #186. * Changed pagination to give better response if its passed an invalid object. Patch from Christoph Haas. * Fixed scriptaculous helper docs example. Fixes #178. * Updated scriptaculous/prototype to Prototype 1.5.0 and Scriptaculous 1.7.0. * Updated scriptaculous javascripts to 1.6.5. Fixes #155. * Updated remote_function doc-string to more clearly indicate the arguments it can receive. * Synced Rails Javascript helper to 5245 (escape_javascript now escaping backslashes and allow passing html_options to javascript_tag). PASTE * In ``paste.httpserver`` remove the reverse DNS lookup to set ``REMOTE_HOST`` * In ``paste.fileapp``, if the client sends both If-None-Match and If-Modified-Since, prefer If-None-Match. Make ETags include the size as well as last modified timestamp. Make it possible to override how mimetypes are guessed. * ``HTTPException`` objects now have a ``exc.response(environ)`` method that returns a ``WSGIResponse`` object. * ``egg:Paste#watch_threads`` will show tracebacks of each thread under Python 2.5. * Made ``paste.util.template`` trim whitespace around statements that are on their own line. * ``paste.fileapp.DataApp`` now accepts ``allowed_headers=[...]`` to specify the allowed headers. By default only ``GET`` and ``HEAD`` are allowed. * Added ``paste.util.import_string.try_import_module``, which imports modules and catches ``ImportError``, but only if it's an error importing the specific module, not an uncaught ``ImportError`` in the module being imported. PASTESCRIPT 1.3.3 ----- * Fixed problem with ``paster serve`` on Windows. Also on Windows, fixed issue with executables with spaces in their names (this case requires the ``win32all`` module). * You can use ``+dot+`` in your project template filenames, specifically so that you can use leading dots in the filename. Usually leading dots cause the file to be ignored. So if you want to have new projects contain a ``.cvsignore`` file, you can put a ``+dot+cvsignore`` file in your template. * Relatedly, ``+plus+`` has been added so you can include pluses. 1.3.2 ----- * ``paster`` was largely broken under Windows; fixed. 1.3.1 ----- * Fix related to Python 2.5 (when there are errors creating files, you could get infinite recursion under Python 2.5). * Use ``subprocess`` module in ``paster serve`` command. Added ``--monitor`` option which will restart the server if it exits. * The ``exe`` command now does % substitution in keys (e.g., ``pid_file=%(here)s/paste.pid``). * Some import problems with Cheetah should be improved. 1.3 --- * Fixed an exception being raised when shutting down flup servers using sockets. * Fixed the CherryPy 3 WSGI server entry point's handling of SIGHUP and SIGTERM. * The CherryPy wsgiserver is now available at ``paste.script.wsgiserver`` (no longer requiring CherryPy to be installed). * Added entry point for twisted server. * Made ``paste.script.pluginlib:egg_info_dir`` work with packages that put the ``Package.egg-info/`` directory in a subdirectory (typically ``src/``). * Remove Cheetah requirement. Packages using Cheetah templates should require Cheetah themselves. If you are using ``paster make-config`` and you *don't* want to use Cheetah, you must add ``use_cheetah = False`` to your ``Installer`` subclass (it defaults to true for backward compatibility). * Make scripts work when there is no ``setup.py`` (if you aren't making a Python/setuptools package). * When using ``paste.script.copydir.copy_dir`` (as with most ``paster create`` templates), you can raise ``SkipTemplate`` (or call the ``skip_template()`` function) which will cause the template to be skipped. You can use this to conditionally include files. * When using ``paster serve c:/...``, it should no longer confuse ``c:`` with a scheme (such as ``config:`` or ``egg:``). * More careful about catching import errors in ``websetup``, so if you have a bug in your ``app.websetup`` module it won't swallow it. Cheers, Ben From bwhiteley at novell.com Wed Apr 25 23:55:15 2007 From: bwhiteley at novell.com (Bart Whiteley) Date: Wed, 25 Apr 2007 15:55:15 -0600 Subject: [Web-SIG] httplib.HTTPConnection and Broken Pipe Message-ID: <1177538115.4553.26.camel@lap.provo.novell.com> I have an issue with the following code in httplib.HTTPConnection.send(): # send the data to the server. if we get a broken pipe, # then close the socket. we want to reconnect when # somebody tries to send again. # # NOTE: we DO propagate the error, though, because we # cannot simply ignore the error... the caller will # know if they can retry. if self.debuglevel > 0: print "send:", repr(str) try: self.sock.sendall(str) except socket.error, v: if v[0] == 32: # Broken pipe self.close() raise I'm sending a POST request, with a somewhat large (3K) request body. The server requires authentication. If the request doesn't have the proper credentials, the server immediately sends a 401 response with an authentication challenge, and closes the connection without reading the request body. The server does this to protect against DoS attacks. When the client continues sending after the server closes the socket, the server sends a TCP reset, and the client usually gets a socket error of 104 (connection reset), but sometimes gets an error of 32 (broken pipe) instead. When I get a 104, I'm able to call getresponse(), and read the 401 response. Then I can retry the request with an appropriate response to the authentication challenge. When I get a socket error of 32, getresponse() throws an exception, so I don't have an authentication challenge to respond to. If I remove the "try:... except:" code above, so that self.close() does not get called on a broken pipe error, the http client code works nicely. In spite of the broken pipe, I can retrieve the response, and can retry the request with an appropriate authentication header. Is anyone aware of a way to work around this problem? From sidnei at enfoldsystems.com Thu Apr 26 04:30:04 2007 From: sidnei at enfoldsystems.com (Sidnei da Silva) Date: Wed, 25 Apr 2007 23:30:04 -0300 Subject: [Web-SIG] httplib.HTTPConnection and Broken Pipe In-Reply-To: <1177538115.4553.26.camel@lap.provo.novell.com> References: <1177538115.4553.26.camel@lap.provo.novell.com> Message-ID: Subclass HTTPConnection and implement your own send()? On 4/25/07, Bart Whiteley wrote: > I have an issue with the following code in > httplib.HTTPConnection.send(): > > # send the data to the server. if we get a broken pipe, > # then close the socket. we want to reconnect when > # somebody tries to send again. > # > # NOTE: we DO propagate the error, though, because we > # cannot simply ignore the error... the caller will > # know if they can retry. > if self.debuglevel > 0: > print "send:", repr(str) > try: > self.sock.sendall(str) > except socket.error, v: > if v[0] == 32: # Broken pipe > self.close() > raise > > I'm sending a POST request, with a somewhat large (3K) request body. > The server requires authentication. If the request doesn't have the > proper credentials, the server immediately sends a 401 response with an > authentication challenge, and closes the connection without reading the > request body. The server does this to protect against DoS attacks. > When the client continues sending after the server closes the socket, > the server sends a TCP reset, and the client usually gets a socket error > of 104 (connection reset), but sometimes gets an error of 32 (broken > pipe) instead. When I get a 104, I'm able to call getresponse(), and > read the 401 response. Then I can retry the request with an appropriate > response to the authentication challenge. When I get a socket error of > 32, getresponse() throws an exception, so I don't have an authentication > challenge to respond to. > > If I remove the "try:... except:" code above, so that self.close() does > not get called on a broken pipe error, the http client code works > nicely. In spite of the broken pipe, I can retrieve the response, and > can retry the request with an appropriate authentication header. > > Is anyone aware of a way to work around this problem? > > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/sidnei%40enfoldsystems.com > -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 From bwhiteley at novell.com Thu Apr 26 21:21:57 2007 From: bwhiteley at novell.com (Bart Whiteley) Date: Thu, 26 Apr 2007 13:21:57 -0600 Subject: [Web-SIG] httplib.HTTPConnection and Broken Pipe In-Reply-To: References: <1177538115.4553.26.camel@lap.provo.novell.com> Message-ID: <1177615317.7424.9.camel@lap.provo.novell.com> On Wed, 2007-04-25 at 23:30 -0300, Sidnei da Silva wrote: > Subclass HTTPConnection and implement your own send()? Yes. That seemed to be the best (or only) solution. I believe anyone using HTTPConnection for a POST where authentication may be required will have to do the same thing. > > On 4/25/07, Bart Whiteley wrote: > > I have an issue with the following code in > > httplib.HTTPConnection.send(): > > > > # send the data to the server. if we get a broken pipe, > > # then close the socket. we want to reconnect when > > # somebody tries to send again. > > # > > # NOTE: we DO propagate the error, though, because we > > # cannot simply ignore the error... the caller will > > # know if they can retry. > > if self.debuglevel > 0: > > print "send:", repr(str) > > try: > > self.sock.sendall(str) > > except socket.error, v: > > if v[0] == 32: # Broken pipe > > self.close() > > raise > > > > I'm sending a POST request, with a somewhat large (3K) request body. > > The server requires authentication. If the request doesn't have the > > proper credentials, the server immediately sends a 401 response with an > > authentication challenge, and closes the connection without reading the > > request body. The server does this to protect against DoS attacks. > > When the client continues sending after the server closes the socket, > > the server sends a TCP reset, and the client usually gets a socket error > > of 104 (connection reset), but sometimes gets an error of 32 (broken > > pipe) instead. When I get a 104, I'm able to call getresponse(), and > > read the 401 response. Then I can retry the request with an appropriate > > response to the authentication challenge. When I get a socket error of > > 32, getresponse() throws an exception, so I don't have an authentication > > challenge to respond to. > > > > If I remove the "try:... except:" code above, so that self.close() does > > not get called on a broken pipe error, the http client code works > > nicely. In spite of the broken pipe, I can retrieve the response, and > > can retry the request with an appropriate authentication header. > > > > Is anyone aware of a way to work around this problem? > > > > _______________________________________________ > > Web-SIG mailing list > > Web-SIG at python.org > > Web SIG: http://www.python.org/sigs/web-sig > > Unsubscribe: http://mail.python.org/mailman/options/web-sig/sidnei%40enfoldsystems.com > > > > From fumanchu at amor.org Thu Apr 26 22:53:41 2007 From: fumanchu at amor.org (Robert Brewer) Date: Thu, 26 Apr 2007 13:53:41 -0700 Subject: [Web-SIG] httplib.HTTPConnection and Broken Pipe In-Reply-To: <1177615317.7424.9.camel@lap.provo.novell.com> Message-ID: <435DF58A933BA74397B42CDEB8145A860B5DE84A@ex9.hostedexchange.local> Bart Whiteley wrote: > On Wed, 2007-04-25 at 23:30 -0300, Sidnei da Silva wrote: > > Subclass HTTPConnection and implement your own send()? > > Yes. That seemed to be the best (or only) solution. I believe anyone > using HTTPConnection for a POST where authentication may be required > will have to do the same thing. Sounds like an excellent candidate for a ticket and patch to Python. Since socket error numbers are platform-specific, CherryPy's HTTP server does this: import errno socket_errors_to_ignore = [] # Not all of these names will be defined for every platform. for _ in ("EPIPE", "ETIMEDOUT", "ECONNREFUSED", "ECONNRESET", "EHOSTDOWN", "EHOSTUNREACH", "WSAECONNABORTED", "WSAECONNREFUSED", "WSAECONNRESET", "WSAENETRESET", "WSAETIMEDOUT"): if _ in dir(errno): socket_errors_to_ignore.append(getattr(errno, _)) # de-dupe the list socket_errors_to_ignore = dict.fromkeys(socket_errors_to_ignore).keys() socket_errors_to_ignore.append("timed out") ...the stdlib might benefit from a similar (overridable) list. Robert Brewer System Architect Amor Ministries fumanchu at amor.org