Nathaniel Smith writes:

It's also nice to be able to parse some HTML data, make a few changes in memory, and then serialize it back to HTML. Having this crash on random documents is rather irritating, esp. if these documents are standards-compliant HTML as in this case.

This example doesn't make sense to me. Why would *conformant* HTML crash the codec? Unless you're saying the source is non-conformant and *lied* about the encoding? Then errors=surrogateescape should do what you want here, no? If not, new codecs won't help you---the "crash" is somewhere else. Similarly, Soni's use case of control characters for formatting in an IRC client. If they're C0, then AFAICT all of the ASCII-compatible codecs do pass all of those through.[1] If they're C1, then you've got big trouble because the multibyte encodings will either error due to a malformed character or produce an unintended character (except for UTF-8, where you can encode the character in UTF-8). The windows-* encodings are quite inconsistent about the graphics they put in C1 space as well as where they leave holes, so this is not just application-specific, it's even encoding-specific behavior. The more examples of claimed use cases I see, the more I think most of them are already addressed more safely by Python's existing mechanisms, and the less I see a real need for this in the stdlib, with the single exception that WHAT-WG may be a better authority to follow than Microsoft for windows-* codecs. Footnotes: [1] I don't like that much, I'd rather restrict to the ones that have universally accepted semantics including CR, LF, HT, ESC, BEL, and FF. But passthrough is traditional there, a few more are in somewhat common use, and I'm not crazy enough to break backward compatibility.

Random832 writes:

I think his point is that the WHATWG standard is the one that governs HTML and therefore HTML that uses these encodings (including the C1 characters) are conformant to *that* standard,

I don't think that is a tenable interpretation of this standard. The WHAT-WG standard encoding for HTML is UTF-8. This is what https://encoding.spec.whatwg.org/#names-and-labels says: Authors must use the UTF-8 encoding and must use the ASCII case-insensitive "utf-8" label to identify it. New protocols and formats, as well as existing formats deployed in new contexts[1], must use the UTF-8 encoding exclusively. If these protocols and formats need to expose the encoding’s name or label, they must expose it as "utf-8". Non-UTF-8 *documents* do not conform. There's nothing anywhere that says you may use other encodings, with the single exception of implied permission when encoding form input to send to the server (and that's not even HTML!) Even there you're encouraged to use UTF-8. The rest of the standard provides for how *processes* should handle encodings in purported HTML documents that fail the requirement to encode in UTF-8. That doesn't mean such documents conform; it simply *gives permission* to a conformant process to try to deal with them, and rules for doing that. Yes, it's true that WHAT-WG processing probably would have saved Nathaniel some aggravation with his manipulations of HTML. It's equally likely that errors='surrogateescape' would do so, and a better job on encodings like Hebrew that leave code points in graphic regions undefined. Footnotes: [1] I take this to mean that when I take an EUC-JP HTML document and move it from my legacy document tree to my new Django static resource collection, I *must* transcode it to UTF-8.