[Web-SIG] bytes, strings, and Unicode in Jython, IronPython,
and CPython 3.0
py-web-sig at xhaus.com
Wed Sep 15 16:28:14 CEST 2004
[Phillip J. Eby]
> I've reviewed last month's Python-Dev discussion about the future
> Python 'bytes()' type, and the eventual transition away from Python's
> current 8-bit strings.
> Mainly, the impression I get is that significant change in this
> respect really can't happen until Python 3.0, because too many
> things have to change at once for it to work.
> So, here's what I propose to do about the open issue in PEP 333.
> Servers and gateways that run under Python implementations where all
> strings are Unicode (e.g. Jython) *may*:
Encoding issues? "Oh no", screams Alan, turning tail and sprinting away!
Before starting my response, I just want to point out two things:
1. I'm no bot when it comes to python and character encodings.
2. that the text below may come across a little cold. I've spent a few
hours thinking through the issues, checking code, rewriting text,
rewriting, rewriting, .... I think the below is the most accurate
picture I can present: it won't win any poetry competitions.
Before getting into the WSGI parameter encoding issues, just a quick
overview of character strings vs. binary strings in jython.
Strings in jython: textual vs. binary
Java stores all textual strings as unicode strings, i.e. sequences of
2-byte characters. These strings can be transcoded to any encoding: when
they are so transcoded, that delivers a sequence of bytes.
Java keeps the concept of textual unicode strings and byte sequences
separate, through the use of (rigidly enforced) method signatures. This
ensures both static type correctness and memory efficiency.
Jython blends the two concepts, by using java.lang.String's to store
both python text strings and python binary strings, i.e. byte arrays. It
stores the latter by the trick of only using the lower byte of each
two-byte unicode character to store data, leaving the upper byte unused.
You can see this by running this code on jython.
s = u'\u00E1\u00E9\u00ED\u00F3\u00FA'
u8 = s.encode('utf-8')
u16 = s.encode('utf-16')
for x in [s, u8, u16]:
print "%d:%s:%s" % (len(x), str(type(x)), `x`)
The only way to create binary strings in jython is to create them
explicitly, for example, by transcoding text strings as above, or by
reading from a byte-oriented stream like a socket, or binary file. These
binary strings do not have their encoding metadata associated with them,
in common with cpython: the programmer must know the encoding of the
byte-array/binary-string they're handling.
When these binary strings are created, and stored as textual unicode
strings, they look like latin-1 textual strings, since all of the
upper-bytes of the characters are zero. So on jython, a binary encoded
latin-1 string and a unicode string containing only latin-1 characters
are represented identically.
In jython, any other time a string is created, by assignment to a string
literal ('', "", """ """), or by reading from a text file, text stream,
etc, the result is always a textual unicode string.
So, on to WSGI
[Phillip J. Eby]
> * accept Unicode statuses and headers, so long as they properly encode
> them for transmission (latin-1 + RFC 2047)
String parameters in jython are always passed as unicode strings,
containing either textual strings or the binary-string/byte-arrays
described above. So the strings received by the jython
start_response_callable will be either textual or binary unicode strings.
The start_response_callable has to be able to operate on these strings
regardless, i.e. transform them using standard python functions, e.g.
.split(' '), int(), etc. If these functions fail to operate correctly on
a binary string, then there is little the start_response_callable can
do, without knowing the encoding of the binary string so that it can
decode to a textual string. If the operations fail on a textual string,
it is because the string contains invalid data for the operation.
Note that this is common with cpython, under which code must also simply
assume that .split() and int() will simply work on the string passed,
without knowing its encoding.
So, in the case of the http status value, as long as
int(status_str.split(' ')) returns an integer, that's fine. Which should
be the case all of the time, as long as what was passed really was a
string containing an ascii integer followed by a space.
In the case of the header list, both header names and header values
could also be passed as either textual or binary strings. There are
three scenarios for the content of those strings
1. They are binary strings, i.e. have zero upper-bytes, and are
presumably suitable (application knows best) for use as http headers
2. They are latin-1 strings, i.e. have zero upper-bytes, and are thus
suitable for use as http headers without transformation.
3. They are non latin-1 strings, i.e. have non-zero upper-bytes, and so
will have to be encoded before transmission, according to RFC 2047.
What jython should do
So any jython middleware, gateway or server that receives a Unicode
string for a header value must
A: Send it without transformation if all upper-bytes are zero.
B: Encode it according to RFC 2047 if there are non-zero upper-bytes,
then send it.
In the case of B, how should the jython code know which iso-8859-X
charset to use for RFC 2047? Is there library code? Is mimify the right
module to use?
A couple of notes about J2EE
1. Under J2EE, the HttpServletResponse method signatures specify that a
java.lang.String, i.e. 2-byte unicode, value must be given for header
names and values (although see next point).
2. The most recent 2.4 version of the servlet specification now permits
header strings to be an "octet string ... encoded according to RFC
2047". This was not specified in previous versions of the spec, i.e. 2.3
3. Which indicates to me that J2EE expects that you have completely
taken care of encoding yourself, i.e. that you will have RFC-2047
encoded your header, if required, before passing it to J2EE.
4. So if a jython start_response_callable receives a binary string, it
should simply transmit it directly. If it receives a unicode string with
non-zero upper-bytes, it should attempt to encode it in RFC-2047 before
transmission. This could be done like so
unicode_header = "my value"
wire_string = unicode_header.encode('latin-1')
wire_string = encode_in_rfc2047(unicode_header)
Standalone pure jython server
When running a standalone pure jython WSGI server, jython code will be
writing header values directly to the client socket. In this case, the
jython start_response_callable/server needs latin-1/RFC2047 strings to
transmit down the socket. The same rules as J2EE above apply to the
treatment of strings in this case.
So, in regards to the WSGI requirement above, the application *must*
transmit Unicode statuses and headers to a jython
start_response_callable, which will attempt to appropriately RFC-2047
encode the strings if they contain anything other than latin-1 characters.
Which I think completely agrees with your requirement as stated, just
with different wording.
[Phillip J. Eby]
> * accept Unicode for response body segments, so long as each segment
> may be encoded as latin-1 (i.e. only uses chars 0-255)
I would say "jython servers can *only* accept unicode strings for
response body segments", since this is the jython mechanism for passing
As you (kind-of) specify, the response body segment is not really a
latin-1 encoded textual string, it is really a binary string of varying
encoding, depending on the application. But treating it as a latin-1
string has the effect of preserving its content as a binary string.
So again, I think that this meets with your requirement, except stated
If WSGI response bodies "crossed over" somehow from a cpython
application to a jython application, through either swig-style linkage
or through some form of http relay protocol such as FastCGI, the jython
receiving end of that would have to produce a response body encoded as a
jython binary string. Which is exactly what jython socket operations,
etc, produce. So pure python middleware code that distributes WSGI
requests over, say a network socket, should run identically between
jython and cpython. Which is nice to know.
And which would probably true for IronPython too: That Jim Hugunin is a
clever lad. Jython really does all this stuff pretty seamlessly in
relation to cpython.
[Phillip J. Eby]
> * produce Unicode input headers and body strings by decoding from
> latin-1, as long as the produced values are considered type 'str' for
> that Python implementation.
On jython, there is no point in decoding latin-1 strings to unicode
strings, because their representations are identical: both are
types.StringType, both take 2 bytes per character/byte, with the upper
byte as zero.
If the recipient is another jython component, all string types will be
If the recipient is a cpython component, then it will still receive the
correct string, because whatever interface lies between the cpython and
the jython will have correctly converted the data (if it was latin-1 data).
So perhaps this requirement could be stated as "jython
components/applications must produce unicode input headers and body
strings, which must only contain latin-1 characters"?
Whew! That turned out to be not so bad after all! (Alan crosses his
fingers behind his back :-)
More information about the Web-SIG