[Python-ideas] Python 3.x and bytes

[Stephen J. Turnbull]

Terry Reedy writes:

 > As far as I noticed, Ethan did not explain why he was extracting single
 > bytes and comparing to a constant, so it is hard to know if he was even
 > using them properly.

It doesn't really matter whether Ethan is using them properly.  It's
clear there are such uses, though I don't know how important they are,
so we may as well assume Ethan's is one such.

 > > Japanese mail is transmitted via SMTP, and the control function
 > > "hello" is still spelled "EHLO" in Japanese mail.
 > 
 > I am not familiar with that control function, but if it is part of
 > the SMTP protocol, it has nothing to do with the language of the
 > payload.

Precisely my point.  Therefore a payload represented as bytes should
be treated as *uninterpreted* bytes, except where interpretations are
defined for those bytes.  This works for SMTP, because RFC 822
*deliberately* specifies headers to be encoded in ASCII (not
"ASCII-compatible") in order that the payload (header) manipulations
specified by RFC 821 and friends be guaranteed correct.

Nevertheless, people frequently request mail processing features that
require manipulations of MIME part bodies and even plain RFC 822
message bodies.  These cannot be guaranteed correct unless done by
decoding and reencoding, but bytes-oriented manipulations generally
"work" in monolingual contexts (or seem to, and any problems can
always be blamed on MS Outlook).  There are several such features that
come up over and over again on Mailman lists and sometimes in the
Python Email SIG, and I'm sure the same is true for web protocols.

 > > Farsi web pages are formatted by HTML, and the control
 > > function "new line" is spelled "<BR>" in Farsi, of course.
 > 
 > When writing the html *text* body, sure. But I presume browsers decode
 > encoded bytes to unicode *before* parsing the text. If so, it does not
 > really matter that '<br>' gets encoded to b'<br>'.

HTML is not exclusively processed by browsers.  It is often processed
by servers and middleware that don't know they're speaking HTML, and
according to several experts' testimony, they're in a freakin' hurry
to push bytes out the door, there's no time for Unicode (decoding and
encoding, OMG how inefficient!)

Such developers want to write their libraries using bytes *and*
literals that can be used both for binary protocols and for text
protocols (urlparse seems to be the canonical example).

The convenience of using bytes in a string-like way (eg, the b''
literal) in manipulating many binary protocols is clear.  That
convenience is just as great for people who are at substantial risk of
mojibake if bytes are used to do text manipulations on the encoded
form, as well as for people who face little risk (eg, those who use
only American English).

The question is how far to go with polymorphism, etc.  I think that
Nick's urlparse work gets the balance about right, and see only danger
in more stringlike bytes (eg, by returning b'b' for b'bytes'[0]).
OTOH, there are some changes that might be useful but seem very
low-risk, such as a c'b' literal that means 98, not b'b'.