[Python-ideas] Support WHATWG versions of legacy encodings

[Soni L.]

On 2018-01-18 04:12 PM, Stephen J. Turnbull wrote:
> Soni L. writes:
>
>   > ISO-8859-1 explicitly defines control characters in the \x80-\x9F range,
>   > IIRC.
>
> You recall incorrectly.  You're probably thinking of RFC 1345.  But
> I've never seen that cited except in the IANA registry.
>
> All of ISO 2022, ISO 4873, ISO 8859, and Unicode suggest the ISO 6429
> primary and supplementary control sets as good choices.  (Unicode goes
> so far as to use ISO 6429's names for the supplementary set for C1
> code points while explicitly denying them *any* semantics.)  But
> none specifies a default, and as far as I know there is no widespread
> agreement on what control codes are good for, except for a handful of
> "whitespace" characters in C0, and a couple of C1 controls that are
> used by (and reserved to) ISO 2022.  In fact, Python ISO-8859 codecs
> do pass them through (both C0 and C1), and the UTF-8 codec passes
> through C0 and allows encoding and decoding of C1 code points.
>
> On the other hand, the ISO standards forbid use of unassigned graphic
> code points as characters (graphic or control), and codecs quite
> reasonably treat unassigned graphic code points as errors.  In Python,
> that practice is extended to the windows-* sets, which seems
> reasonable to me.  But the windows-* encodings do not support C1
> controls.  Instead the entire right half of the code page is graphic
> (per Microsoft's IANA registrations), and that, I suppose, is why
> Python does not allow fallthrough of unassigned code points 0x80-0x9F
> in windows-* codecs.
>
>   > I think python should follow the (de-facto) standard. This is it.
>
> WHAT-WG encoding isn't a "de facto" standard, it's a published
> standard by a recognized (though forked) standards body.  However,
> different standards are designed for different contexts, and WHAT-WG's
> encoding standard is clearly specifically aimed at browsers.  It also
> may be useful for more specialized UI applications such as your IRC
> client, although IMO that's asking for trouble.  Note also that the
> WHAT-WG standard is in a peculiar limbo between informative and
> normative.  The standard encoding is UTF-8, end-of-story.  What we're
> talking about here is best practices for UIs that are faced with
> non-conformant "legacy" documents, and want to display something
> anyway.
>
> But Python is a general-purpose programming language, and should
> cleave to the most generally-accepted, well-defined standards, which
> are the ISO standards themselves in the case of ISO-defined coded
> character sets.  Aliasing the ISO character sets (and ASCII! oh, my
> aching RFC 822 header!) to the corresponding windows-* as a *general*
> practice is pretty abominable, though it makes some sense in the case
> of browsers.  For windows-* character sets, ISTM that the WHAT-WG
> repertoires of graphic characters are improvements of Microsoft's
> (assuming that WHAT-WG version their standards).
>
> Applications can do what they want, of course, and I'm all for a PyPI
> package to make it easier to do that, whether by providing additional
> codecs, additional error handlers, or by post-processing surrogate-
> escaped bytes.  I still don't think the WHAT-WG approach is a good fit
> for most use cases, nor should it be included in the stdlib.  Most of
> the use cases I've seen proposed so far are well-served by existing
> Python features like errors='surrogateescape'.

I'm just glad I *always* use bytestrings when dealing with network 
protocols, I guess. It's the only reasonable option.

>
> Steve
>
>