[Python-ideas] Support WHATWG versions of legacy encodings
steve at pearwood.info
Sun Jan 21 05:43:44 EST 2018
On Fri, Jan 19, 2018 at 06:35:30PM +0000, Rob Speer wrote:
> > It depends on what you want to achieve. You may want to fail, assign a
> code point from a private area or use a surrogate escape approach.
> And the way to express that is with errors='replace',
> errors='surrogateescape', or whatever, which Python already does. We do not
> need an explosion of error handlers. This problem can be very
> straightforwardly solved with encodings, and error handlers can keep doing
> their usual job on top of encodings.
> > You could also add a "latin1replace" error handler which simply passes
> through everything that's undefined as-is.
> Nobody asked for this.
Actually, Soni L. seems to have suggested a similar idea in the thread
titled "Chaining coders" (codecs).
But what does it matter whether someone asked for it? Until this thread,
nobody had asked for support for WHATWG encodings either.
The question to my mind is whether or not this "latin1replace" handler,
in conjunction with existing codecs, will do the same thing as the
WHATWG codecs. If I have understood you correctly, I think it will. Have
I missed something?
> > I just don't want to have people start using "web-1252" as encoding
> simply because they they are writing out text for a web application - they
> should use "utf-8" instead.
> I did ask for input on the name. If the problem is that you think my
> working name for the encoding is misleading, you could help with that
> instead of constantly trying to replace the proposal with something
Rob, you've come here with a proposal based on an actual problem (web
pages with mojibake and broken encodings), an existing solution (a third
party library) you dislike, and a suggested new solution you will like
(move the encodings into the std lib). That's great, and we need more
suggestions like this: concrete use-cases and concrete solutions.
But you cannot expect that we're going to automatically agree that:
- the problem is something that Python the language has to solve
(it seems to be a *browser* problem, not a general programming
- the existing solution is not sufficient; and
- your proposal is the right solution.
All of these things need to be justified, and counter-proposals are part
When we make a non-trivial proposal on Python-Ideas, it is very rare
that they are so clearly the right solution for the right problem that
they get instant approval and you can go straight to the PR. Often there
are legitimate questions about all three steps. That's why I suggested
earlier that (in my opinion) there needs to be a PEP to summarise the
issue, justify the proposal, and counter the arguments against it.
(Even if the proposal is agreed upon by everyone, if it is sufficiently
non-trivial, we sometimes require a PEP summarising the issue for future
As the author of one PEP myself, I know how frustrating this process can
seem when you think that this is a bloody obvious proposal with no
downside that all right-thinking people ought to instantly recognise as
a great idea *wink* but nevertheless, in *my opinion* (I don't speak for
anyone else) I think a PEP would be a good idea.
> Guido had some very sensible feedback just a moment ago. I am wondering now
> if we lost Guido because I broke python-ideas etiquette (is a pull request
> not the next step, for example? I never got a good answer on the process),
> or because this thread is just constantly being derailed.
I don't speak for Guido, but it might simply be he isn't invested
enough in *this specific issue* to spend the time wading through a
long thread. (That's another reason why a PEP is sometimes valuable.)
Perhaps he's still on holiday and only has limited time to spend on
If I were in your position, my next step would be to write a new
post summarising the thread so far:
- a brief summary of the nature of the problem;
- why you think a solution (whatever that solution turns out to be)
should be in the stdlib rather than a third-party library;
- what you think the solution should be;
- and give a fair critique of the alternatives suggested so far and
why you thik that they aren't suitable.
That's the same sort of information given in a PEP, but without having
to go through the formal PEP process. That might be enough to gain
consensus on what happens next -- and maybe even agreement that a formal
and more detailed PEP is not needed.
Oh, and in case you're thinking this is all a great PITA, it might help
if you read these to get an understanding of why things are as they are:
More information about the Python-ideas