[Web-SIG] WSGI for Python 3
ianb at colorstudy.com
Mon Aug 30 18:00:28 CEST 2010
Just to narrow in on one case, URLs, there are a few pieces of information
that make up the URL:
wsgi.url_scheme: this is *not* present in the request, it's inferred somehow
(e.g., by the port the client connected to)
HTTP_HOST: this is a header. It typically contains both the hostname and
the port. The encoding is generally idna, though you have to split the port
off first. The unicode version of the hostname is not widely supported in
client libraries (it's usually applied at the UI level).
SCRIPT_NAME/PATH_INFO: these represent a portion of the request path (before
?). As submitted these are generally ASCII (URL-quoted). After unquoting,
they are typically UTF-8, but may be of any or no encoding. If an unsafe
character is present in the URL-quoted version of the path, it may be quoted
at the byte level. The '?' character is effectively a byte-oriented marker
and encodings cannot affect it.
QUERY_STRING: this is also generally ASCII (URL-quoted). Unsafe characters
could be quoted at the byte level.
Generally I'm unaware of any reasonable situation where quoting unsafe
characters in an HTTP request would be improper, or even lose any meaningful
information. Mostly because I don't know of any clients that actually would
expect unsafe characters to work. Quoting HTTP_HOST is difficult, as it's
not a byte-oriented quoting, it's a fairly complex encoding. But I'm also
not sure where in a stack you could actually handle unsafe characters in
HTTP_HOST -- it seems like simply an invalid request, and deferring the
error won't give another part of the stack the opportunity to do the right
In their quoted form all these values (at least including the quoted path,
not the unquoted SCRIPT_NAME/PATH_INFO) *should* be ASCII, and I believe a
WSGI server could ensure they were all ASCII without any loss of useful
information (either by simply rejecting the request or by applying
quoting). I don't see any place where bytes are advantageous. Representing
invalid requests does not seem particularly helpful -- *some* invalid
requests are useful to handle (e.g., weird cookies) but in the case of the
URL variables I don't see any benefit.
IMHO all the tricky encoding issues are in the request and response bodies,
and I'm pretty sure we have consensus that those should be bytes.
Reiterating other encoding issues I'm aware of:
Cookie encodings, but parsing cookies as bytes or Latin1 is basically
equivalent, and I don't believe that, for instance, they should ever be
parsed as UTF-8. Parsing as bytes might avoid an unnecessary
encoding/decoding, but it's all tricky enough that libraries should do it
anyway, and the encoding overhead alone isn't very important.
Another example is the Atom Title header (
but that's supposed to be Latin1 with RFC2047 encodings, and I don't believe
anyone is proposing that RFC2047 encodings be handled generally at the WSGI
layer (I think CherryPy does or used to handle these, but there were many
objections at least on this list about it, in part due to security
concerns). A 2047 encoding is like "Title:
Response headers are equivalent to request headers. Response status is
constrained by the spec to Latin1, and there are no use cases I know of
(even really obscure ones) where it would be necessary to use other
And that's it! HTTP has a fairly finite amount of surface area.
Ian Bicking | http://blog.ianbicking.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Web-SIG