[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces

[Paul Moore]

2009/4/27 Stephen J. Turnbull <stephen at xemacs.org>:
> I believe there are solutions that don't have that problem.
> Specifically, if the return values were bytes, or (better for 2.x,
> where bytes are strings as far as most programmers are concerned) as a
> new data type, to indicate that they're not text until the client
> acknowledges them as such.  EIBTI.

I think you're ignoring the fact that under Windows, it's the *bytes*
APIs that are lossy.

Can I at least assume that you aren't recommending that only the bytes
API exists on Unix, and only the Unicode API on Windows?

So what's your suggestion?

> Unfortunately, Martin clearly doesn't intend to make such a change to
> the PEP.  I don't have the time or the Python expertise to generate an
> alternative PEP. :-(  I do have long experience with the pain of
> dealing with encoding issues caused by APIs that are intended to DTRT,
> conveniently.  Martin's is better than most, but I just don't think
> convenience and robustness can be combined in this area.

The *only* "robust" solution is to completely separate the 2
platforms. Which helps no-one, and is at least as bad as the 2.x
situation. (Probably worse).

> BTW, I'm on record as +0 on the PEP.  I don't think the better
> proposals have a chance, because most people *want* the non-solution
> that they can just use as a habit, allowing Python to make decisions
> that should be made by the application, and not have to do
> "unnecessary" conversions and the like.  It's not obvious to me that
> it should not be given to them, but I don't much like it.

People *want* a solution that doesn't require every application
developer to sweat blood to write working code, simply to cover corner
cases that they don't believe will happen. Not every application is a
24x7 server, and all that. Similarly, not every application is a
backup program. Such applications have unique issues, which the
developers should (but don't always, admittedly!) understand. The rest
of us don't want to be made to care.

It's not sloppiness. It's a realistic appreciation of the requirements
of the application. (And an acceptance that not every bug must be
fixed before release).

Paul.