[Python-Dev] PEP 3333: wsgi_string() function

[And Clover]

On Tue, 2011-01-04 at 03:44 +0100, Victor Stinner wrote:
> What is this horrible encoding "bytes-as-unicode"?

It is a unicode string decoded from bytes using ISO-8859-1. ISO-8859-1
is the encoding specified by the HTTP RFC, as well as having the happy
property of preserving every input byte.

> os.environ is supposed to be correctly decoded and contain valid unicode characters.

Nope. It is not possible to ‘correctly’ decode to unicode for os.environ
because that decoding happens long before the web application gets a
look in. Maybe the web application is using UTF-8, maybe it's using
cp1252, but if we let the server/gateway decide and do that decoding
before the application can do anything about it, we will get the wrong
encoding in *many* cases and the result will be permanent, unrecoverable
mangling of non-ASCII characters in submitted headers.

> If WSGI uses another encoding than the locale encoding (which is a bad idea),

It's an absolutely necessary idea. The locale encoding is nothing to do
with the web application's encoding. Windows applications need to be
able to use UTF-8 (which is never the ANSI code page), and web
applications in general need to be deployable to any server without
having to worry about the server's locale.

The locale-dependent status quo is that non-ASCII characters in URL
paths and other HTTP headers don't work for Python apps.

The recoding dances present in wsgiref's CGIHandler for 3.2 are
distasteful but completely necessary to normalise differences in
encodings used by various servers and platforms to generate their CGI
environment.

>  it should use os.environb and decodes keys and values using its
> own encoding.

Well yes, but:

(a) os.environb doesn't exist in previous Python 3.1, making it
impossible to implement WSGI before 3.2;
(b) there are also non-HTTP-related environment variables, which may
contain native Unicode strings (eg, very commonly, Windows pathnames),
so you have to have both environ *and* environb.

The bytes-or-bytes-in-Unicode argument is something that has been
bounced around Web-SIG for literally *years*; this is what we ended up
with. Although I personally like bytes, frankly, a re-run of this
argument *again* whilst WSGI remains in perpetual stalemate does not
appeal. WSGI and wsgiref in Python 3.0-3.1 simply not work at all. This
has been an embarrassing situation for what is supposed to be a leading
web language. Let's not perpetuate this sorry story to 3.2 as well.

-- 
And Clover
mailto:and at doxdesk.com http://www.doxdesk.com
skype:uknrbobince gtalk:chat?jid=bobince at gmail.com