Soni L. writes:
This is surprising to me because I always took those encodings to have those fallbacks [to raw control characters].
ISO-8859-1 implementations do, for historical reasons AFAICT. And they frequently produce mojibake and occasionally wilder behavior. Most legacy encodings don't, and their standards documents frequently leave the behavior undefined for control character codes (which means you can error on them) and define use of unassigned codes as an error.
It's pretty wild to think someone wouldn't want them.
In what context? WHAT-WG's encoding standard is *all about browsers*. If a codec is feeding text into a process that renders them all as glyphs for a human to look at, that's one thing. The codec doesn't want to fatal there, and the likely fallback glyph is something from the control glyphs block if even windows-125x doesn't have a glyph there. I guess it sort of makes sense. If you're feeding a program (as with JSON data, which I believe is "supposed" to be UTF-8, but many developers use the legacy charsets they're used to and which are often embedded in the underlying databases etc, ditto XML), the codec has no idea when or how that's going to get interpreted. In one application I've maintained, an editor, it has to deal with whatever characters are sent to it, but we preferred to take charset designations seriously because users were able to flexibly change those if they wanted to, so the error handler is some form of replacement with a human-readable representation (not pass-through), except for the usual HT, CR, LF, FF, and DEL (and ESC in encodings using ISO 2022 extensions). Mostly users would use the editor to remove or replace invalid codes, although of course they could just leave them in (and they would be converted from display form to the original codes on output). In another, a mailing list manager, codes outside the defined repertoires were a recurring nightmare that crashed server processes and blocked queues. It took a decade before we sealed the last known "leak" and I am not confident there are no leaks left. So I don't actually have experience of a use case for control character pass-through, and I wouldn't even automate the superset substitutions if I could avoid it. (In the editor case, I would provide a dialog saying "This is supposed to be iso-8859-1, but I'm seeing C1 control codes. Would you like me to try windows-1252, which uses those codes for graphic characters?") So to my mind, the use case here is relatively restricted (writing user display interfaces) and does not need to be in the stdlib, and would constitute an attractive nuisance there (developers would say "these users will stop complaining about inability to process their dirty data if I use a WHAT-WG version of a codec, then they don't have to clean up"). I don't have an objection to supporting even that use case, but I don't see why that support needs to be available in the stdlib.