[Python-3000] email libraries: use byte or unicode strings?

[Glenn Linderman]

On approximately 11/6/2008 3:59 AM, came the following characters from 
the keyboard of Stephen J. Turnbull:
> Glenn Linderman writes:
> 
>  > There is no reference to the word emacs or types in any of the messages 
>  > you've posted in this thread, maybe you are referring to another thread 
>  > somewhere?  Sorry, I'm new to this party, but I have read the whole 
>  > thread... unless my mail reader has missed part of it.
> 
> I'm sorry, you are right; the relevant message was never sent.  Here
> it is; I've looked it over briefly and it seems intelligible, but from
> your point of view it may seem out of context now.


Stuff happens.  Apology accepted.  The goal here isn't to make points or 
  play one-up, the goal is to figure out if making a more complex 
interface (having both bytes and Unicode interfaces) is beneficial to 
life.  I'm certain that I don't see all the issues yet; but if the 
issues can be stated clearly, and the alternative solutions outlined, 
then I would get educated, which is good for me, but perhaps annoying 
for you.  Progress gets made faster if we stay out of the flame-fanning.

I've read the other responses received to date, but choose to compose my 
response to this message, as it is the most meaty.  The others discuss 
only particular (interesting) details.

Summary of issues is at the end.  Skip directly to the summary before 
reading the interspersed comments, if you wish.  Search for "summarize".


Comment on general data handling.  It is good to follow the rules, of 
course, but not everyone does.  When they don't, it is not clear a 
program can cure the problem by itself.

1) If the data is already corrupted by using the wrong encoding, 
potentially it could be reversed if the proper encoding could be intuited.

1a) If it is returned as bytes, then once the proper encoding is 
intuited, the data can be decoded properly into Unicode.

1b) If it is returned as Latin-1 decoded Unicode, then once the proper 
encoding is intuited, the Unicode data can be reencoded as bytes using 
Latin-1 (this is a fully reversible, no data loss reencoding), and then 
decoded properly into Unicode.

The hard part here is intuiting the proper encoding; 1b is less 
efficient than 1a, but no less possible.  Intuiting the proper encoding 
is most likely done by human choice (iterating over: try this encoding, 
does it look better?)

2) If the data is already corrupted by using multiple encodings when 
only one is claimed, then again it could be reversed if the proper 
encodings, as well as the boundaries between them, could be intuited.

The same parts a) and b) apply as in #1, but extremely complexified by 
the boundary selections.  Again it seems that human choice is required. 
  Select a range of text, and try displaying it in a different encoding 
to see if it makes more sense.

For both 1 & 2, the user interaction is much more time consuming than 
the 3-stage decoding, encoding, and redecoding process, I would expect.

More below.


> Glenn Linderman writes:
> 
>  > This is where you use the Latin-1 conversion.  Don't throw an error
>  > when in doesn't conform, but don't go to heroic efforts to provide
>  > bytes alternatives... just convert the bytes to Unicode, and the
>  > way the mail RFCs are written, and the types of encodings used, it
>  > is mostly readable.  And if it isn't encoded, it is even more
>  > readable.
> 
> This is what XEmacs/Mule does.  It's a PITA for everybody (except the
> Mule implementers, whose life is dramatically simplified by punting
> this way).  For one thing, what's readable to a human being may be
> death to a subprogram that expects valid MIME.  GNU Emacs is even
> worse; it does provide both a bytes-like type and a unicode-like type,
> but then it turns around and provides a way to "cast" unicodes to
> bytes and vice-versa, thus exposing implementation in an unclean (and
> often buggy) way.
> 
>  > And so how much is it a problem?  What are the effects of the problem?
> 
> In Emacs, the problem is that strings that are punted get concatenated
> with strings that are properly decoded, and when reencoding is
> attempted, you get garbage or a coding error.  


Uh-huh.  Garbage (wrongly decoded, then re-encoded), I would expect. 
Coding errors, I would not, since Latin-1 codepoints are certainly 
reencodable to Unicode (creating legal looking garbage OUt of originally 
illegal garbage).  Can you give me an example of a coding error, or is 
this just FUD?


> Since Mule discarded
> the type (punt vs. decode) information, the app loses.  


This is precisely the problem that was faced for "fake unicode file 
handling" that was the topic of a thread a few weeks ago.  While the 
Latin-1 transform (or UTF-8b, or others mentioned there), can provide a 
round-trip decode/encode, it is only useful and usable if the knowledge 
that the transform was performed is retained.  The choice there was to 
have a binary interface, and build a Unicode interface on top of it that 
  can't see the binaries that do not conform to UTF-8.  The problem 
there is that existing programs expect to be able to manipulate file 
names as text, but existing operating systems provide bytes interfaces.


> There's no way to recover.  


Not automatically.  Point 2) above addresses this.  It would require 
human intelligence to attempt to recover, and even the human would find 
it extremely painstaking to assist in the recovery process.


> The apps most at risk are things like MUAs (which Emacs
> does well) and web browsers (which it doesn't), and even AUCTeX (a
> mode for handling LaTeX documents---TeX is not Unicode-aware so its
> error messages are frequently truncated in the middle of a UTF-8
> character) and they go to great lengths to keep track of what is valid
> and what is not in the app.  They don't always succeed.  I think Emacs
> should be doing this for them, somehow (and I'm an XEmacs implementer,
> not an MUA implementer!)


So your belief that Emacs should be doing this for them somehow is nice, 
perhaps it should.  However, it doesn't sound like you have a solution 
for emacs...  How should it keep track?  How is it helpful?  If TeX is 
not Unicode aware, what is it doing dealing with UTF-8 data?  Or it is 
dealing with Latin-1 transformed UTF-8 garbage?


> The situation in Python will be strongly analogous, I believe.


And so are you proposing that a binary interface to the data, rather 
than a Unicode interface to the Latin-1 transformed data, will be more 
usable by the Python solution that might be able to be similar to the 
Emacs solution, that hasn't been figured out yet?

Once the boundaries and encoding has been lost by the original buggy MUA 
that has injected the data into the email message, only human 
intelligence has a chance of recreating the original message in all 
cases, and even then it may take more than one human to achieve it.

There may be cases where heuristics can be applied, when human 
intelligence figures out the type of bugs in the original MUA, and can 
recognize patterns that allow it to rediscover the boundaries.  This is 
unlikely to work in all cases, but could perhaps work in some cases.

Even in the cases where it can work with some measurable success, I 
claim that the heuristics could be coded based on the Latin-1 
transformed Unicode equally effectively as based on the bytes.


>  > I'm not suggesting making it worse than what it already is, in
>  > bytes form; just to translate the bytes to Unicode codepoints so
>  > that they can be returned on a Unicode interface.
> 
> Which *does* make it worse, unless you enforce a type difference so
> that punted strings can't be mixed with decoded strings without
> effort.  That type difference may as well be bytes vs. Unicode as some
> subclass of Unicode vs. Unicode.


138 is still 138 whether it is a byte or a Unicode codepoint.  Yes, 
concatenating stuff that is transformed with stuff that is properly 
decoded would be stupid.  Enforcing a type difference is purely an 
application thing, though.  Each piece of data retrieved would have a 
consistent decoding provided... either the proper decoding as specified 
in the message, or the Latin-1 or current code page decode if no 
encoding is specified.  Either is reversible if the application doesn't 
like the results, and wants to try a different encoding.  The APIs could 
have optional parameters and results that specify the encoding to use, 
or the encoding that was used, to decode the results.  If the app wishes 
to keep that separate, and convert it to a different type to help it 
stay separate that is the app's privilege.  If the app wishes to 
concatenate with other data, that is the app's choice (and having the 
interface define a bunch of different types for different decodings 
wouldn't really help the ignorant app, which would simply convert the 
different types back to strings and then concatenate, or the smart app, 
which could do its own type encapsulations if it thinks that would help).


> "Why would you mix strings?"  Well, for one example there are multiple
> address headers which get collected into an addressee list for purpose
> of constructing a reply.  If one of the headers is broken and another
> is not, you get mixed mode.  


Sure.  Now you have mixed mode.  Try to send the reply message... if the 
email address part is OK, then it gets sent, with a gibberish name.  If 
the email address part is not OK, that destination bounces.

Now what?  Seriously, what else could be done?  You could try a bunch of 
different encodings to attempt to resolve the broken email address or 
name... requires human intelligence to decide which is correct... when 
the bounce message comes, the human will get involved.  If the bounce 
message doesn't come, then all is well (problem only affected the name 
part, not the email address part).


> The same thing can happen for
> multilingual message bodies: they get split into a multipart with
> different charsets for different parts, and if one is broken but
> another is not, you get mixed mode.


First, if the multilingual message bodies are know to be multilingual 
when they are encoded, and are in different multiparts, what are the 
chances that an application that knows to correctly keep the 
multilingual parts separate is dumb enough to encode one correctly and 
one incorrectly?  Is this a real scenario?  What software/version does this?

If it is real scenario, it still requires human intelligence to 
resolve... to choose different encodings, and decide which one "looks 
right".  Since it is in separate parts, the boundaries are not lost, so 
this is case 1 above.

If the boundaries are lost, the human can direct the program to go back 
to the original message, which still has its boundaries, and start over 
from there, with different encodings.  If the app wants to be smart 
enough to provide such features.  You might write such an app just for 
fun; I might or might not, depending on if someone pays me, or I have 
other incentive.

Given boundaries, it is case 1) above.  If the boundaries are lost, it 
is case 2).  How is it easier if the bytes are preserved, vs translated 
via Latin-1 to a Unicode string?



>  > So they'll use the Unicode API for text, and the bytes APIs for binary 
>  > attachments, because that is what is natural.
> 
> Well, as I see it there won't be bytes APIs for text.  The APIs will
> return Unicode text if they succeed, and raise an error if not.  If
> the error is caught, the offending object will be available as bytes.


Sure; I'd proposed a way to get a whole messages as bytes for archiving, 
logging, message store, etc.  I'd proposed a way to get a particular 
MIME part as bytes for binary parts.

You seem to be proposing a way to get text MIME parts as binary if they 
fail to decode.  I have no particular problem with the API providing 
that ability.

I have a specific question here: what encodings, when the attempt is 
made to decode to Unicode, will ever fail?

For 8-bit encodings, the answer is none.  You may get gibberish, but not 
a failure, because every 8-bit encoding has every byte value used, and 
Unicode contains all those characters.

So you've mentioned Asian encodings, and certainly these could fail to 
convert to Unicode if the decoder finds inappropriate sequences.  I 
don't know enough about all the multi-byte encodings to know if all of 
them can fail, or if applying a particular decoding might produce 
gibberish, but not fail.  The ones I know about use a particular range 
of characters to represent "first byte" of a pair, but what I don't know 
is whether any byte can follow the first byte, or if only certain bytes 
can follow the first byte.  I do that for some multi-byte encodings, the 
first byte can be followed by second bytes in the ASCII range; I don't 
know if it is illegal to be followed by another byte in the "first byte" 
range.  Certainly there could be 2-byte pairs that don't have an 
associated character, although I don't know that that exists for any 
particular encoding.

Can you cite a particular multi-byte encoding that has byte sequences 
that are illegal, and can be used to detect failure?  Or can failure 
only be detected by the human determining that it is gibberish?


>  > If improperly encoded messages are received, and appropriate 
>  > transliterations are made so that the bytes get converted (default code 
>  > page) or passed through (Latin-1 transformation), then the data may be 
>  > somewhat garbled for characters in the non-ASCII subset.  But that is 
>  > not different than the handling done by any 8-bit email client, nor, I 
>  > suspect (a little uncertainty here) different than the handling done by 
>  > Python < 3.0 mail libraries.
> 
> Which is exactly how we got to this point.  Experience with GNU
> Mailman and other such applications indicate that the implementation
> in the existing Python email module needs work, and Barry Warsaw and
> others who have tried to work on it say that it's not that easy, and
> that the API may need to change to accomodate needed changes in the
> implementation.


So let me try to summarize.  I could have reached some inappropriate 
issues or conclusions.  I'm willing to be corrected.  But I'd much 
prefer to be corrected by specific cases that can be detected and 
corrected via a bytes interface that cannot be detected and corrected by 
using a bytes-translitered-to-Unicode interface, complete with specific 
encodings that are used, properly or improperly, to arrive at the case, 
and specific APIs that must be changed to achieve the goal.

A) An attempt to decode text to Unicode may fail.
A1) doesn't apply to 8-bit encodings.
A2) doesn't apply to some multi-byte encodings
A3) applies to UTF-8
A4) may apply to some other multi-byte encodings

B) User sees gibberish because of decoding problems.  What can be done? 
  Can the app provide features to help? Do any of the features depend on 
API features?  Let's assume that the app wants to help, and provides 
features.  User must also get involved, because the app/API can't tell 
the difference between gibberish and valid text.

B1) User can see a map of the components of the email, and their 
encodings, and whether they were provided by the email message, or were 
the default for the app.  User chooses a different decoding for a 
component, and the app reprocesses that component.  API requirement: a 
way for the user/app to specify an override to the decoding for a component.

B2) User chooses binary for a particular component.  App reprocesses the 
component, and asks what file to store the binary in.  API requirement: 
a way for the user/app to specify an override to the decoding for a 
component.


I've now looked briefly at the email module APIs.  They seem quite 
flexible to me.  I don't know what happens under the covers.  It seems 
that the API is already set up flexibly enough to handle both bytes and 
Unicode!!!  Perhaps it is just the implementation that should be 
adjusted.  (I'm not saying that might not be too big a job for 3.0, I 
haven't read the code.)

It seems that get_/set_payload might want to be able to return/accept 
either string or bytes, depending on the other parameters involved.

Let's talk again about creation of messages first.

If a string is supplied, it is Unicode.  The encoding parameter 
describes what encoding should be applied to convert the message to 
wire-protocol bytes.  The data should be saved as Unicode until the 
request is made to convert it to wire protocol, so that set_charset can 
be called a few dozen times if desired (not clear why that would be 
done, though) to change the encoding.  Perhaps it is appropriate to 
verify that the encoding can happen without using the substitution 
character, or perhaps that should be the user's responsibility.  This 
choice should be documented.

If bytes are supplied, an encoding must also be supplied.  The data 
should be saved in this encoding until the request is made to convert it 
to wire-protocol.  This encoding should be used if possible, otherwise 
converted to an encoding that is acceptable to the wire protocol. 
Perhaps it is appropriate to verify that the translation, if necessary, 
can happen without using the substitution character, or perhaps that 
should be the user's responsibility.  This choice should be documented.
It seems that charset None implies ASCII, for historical reasons; 
perhaps that can be overloaded to alternately mean binary, as the 
handling would be roughly the same, but perhaps a new 'binary' charset 
should be created to make it clear that charset changes don't make 
sense, and to reject attempts to convert binary data to character data.

For an incoming message, the wire-protocol format should be used as the 
primary data store.  Cached pointers and lengths to various MIME parts 
and subparts (individual headers, body, preamble, epilogue) would be 
appropriate.  get_ operations would find the data, and interpret it 
according to the current (defaults to message content, overridden by 
set_ operations) charset and encoding.  Requesting a Unicode charset 
would imply decoding the part to Unicode from the current charset and 
would return a string; requesting other character sets would imply 
converting from the message charset to the specified charset and 
returning bytes; requesting binary (or possibly 'None', see above) would 
return the wire-protocol bytes unchanged.  Then the application could do 
what it wants to attempt to decode that data to text using other 
encodings (i.e. not starting the conversion from the encoding declared 
explicitly or implicitly in the message part).

The as_string() method becomes a misnomer in Python 3.0; since it is 
Python 3.0, that can be changed, no?  It should become as_wire_protocol, 
and would default to returning bytes of binary data, which is what the 
wire-protocol APIs need.  A variety that returns the bytes as Unicode 
codepoints could be implemented, for the purpose of "View source" type 
operations on the wire-protocol form... but that would and should only 
be a direct Latin-1 transliteration to Unicode.

Now that I've looked at the API, I don't see why it should be changed 
significantly for Python 3.0.  I have no clue how much of the guts would 
have to be changed to achieve the equivalent of what I described above. 
  I do believe that what I outlined above would use the present API to 
achieve both the "I want Unicode only" philosophy that you ascribe to 
me, and the "I want to do bit-flipping" (whatever that means) philosophy 
that you claimed for yourself.

Setting headers via the msg['Subject'] syntax to Unicode values is no 
problem.  Just make sure that they get converted to ASCII encoded 
properly at the end.  msg['Subject'] and msg[b'Subject'] could be made 
equivalent, but I'd never use the latter, it has an annoying b character 
to distract from the meaning.  The syntax should permit the use of 
Unicode, in other words, but:

* to encode non-ASCII data with full control over what parts get encoded 
and how, the Header API is still appropriate
* as an alternative, the API could be extended to include a default 
header encoding
* Strings supplied via the msg['Subject'] = 'some string' interface, are 
handled as follows: if 'some string' is in the ASCII subset, no problem. 
If not, and if the default header encoding has not been set, then an 
exception is raised.  Otherwise, the default header encoding is used to 
encode the Unicode string as necessary.


So I see the API as quite robust, although its current implementation 
may not be as described above, and I can't scope the effort to achieve 
the above.

I'd like to see a "headers_as_wire_protocol" API added for generating 
bounce messages.  It is easy enough to extract from as_wire_protocol, 
but common enough to be useful, methinks, and avoids allocating space 
for a huge message just to get its headers.


What specific problems are perceived, that the present API can't handle?

Are there areas in which it behaves differently than I outline above?

If so, is my outline an improvement, or confusing, and why?

Are there other issues?

Barry said:
 > Yes, Python 2.x's email package handles broken messages, and email-ng
 > must too.  "Handling it" means:
 >
 > 1) never throw an exception
 > 2) record defects in a usable way for upstream consumers of the message
 > to handle
 >
 > it currently also means
 >
 > 3) ignore idempotency for defective messages.

I'm not sure what "ignore idempotency" means in this context...


If the above outline is perceived as a useful set of semantics for the 
3.0 email library, I might be able to find a little time (don't tell my 
wife) to help work on them, assuming that they are mostly implemented in 
Python and/or C.  But I'd need a bit of hand-holding to get started, 
since I haven't yet figured out how to compile my own Python.


-- 
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking