[Python-ideas] Support WHATWG versions of legacy encodings

[Steven D'Aprano]

On Fri, Jan 19, 2018 at 06:35:30PM +0000, Rob Speer wrote:
> > It depends on what you want to achieve. You may want to fail, assign a
> code point from a private area or use a surrogate escape approach.
> 
> And the way to express that is with errors='replace',
> errors='surrogateescape', or whatever, which Python already does. We do not
> need an explosion of error handlers. This problem can be very
> straightforwardly solved with encodings, and error handlers can keep doing
> their usual job on top of encodings.
> 
> > You could also add a "latin1replace" error handler which simply passes
> through everything that's undefined as-is.
> 
> Nobody asked for this.

Actually, Soni L. seems to have suggested a similar idea in the thread 
titled "Chaining coders" (codecs).

But what does it matter whether someone asked for it? Until this thread, 
nobody had asked for support for WHATWG encodings either.

The question to my mind is whether or not this "latin1replace" handler, 
in conjunction with existing codecs, will do the same thing as the 
WHATWG codecs. If I have understood you correctly, I think it will. Have 
I missed something?


> > I just don't want to have people start using "web-1252" as encoding
> simply because they they are writing out text for a web application - they
> should use "utf-8" instead.
> 
> I did ask for input on the name. If the problem is that you think my
> working name for the encoding is misleading, you could help with that
> instead of constantly trying to replace the proposal with something
> different.

Rob, you've come here with a proposal based on an actual problem (web 
pages with mojibake and broken encodings), an existing solution (a third 
party library) you dislike, and a suggested new solution you will like 
(move the encodings into the std lib). That's great, and we need more 
suggestions like this: concrete use-cases and concrete solutions.

But you cannot expect that we're going to automatically agree that:

- the problem is something that Python the language has to solve 
  (it seems to be a *browser* problem, not a general programming
  problem);

- the existing solution is not sufficient; and

- your proposal is the right solution.


All of these things need to be justified, and counter-proposals are part 
of that.

When we make a non-trivial proposal on Python-Ideas, it is very rare 
that they are so clearly the right solution for the right problem that 
they get instant approval and you can go straight to the PR. Often there 
are legitimate questions about all three steps. That's why I suggested 
earlier that (in my opinion) there needs to be a PEP to summarise the 
issue, justify the proposal, and counter the arguments against it.

(Even if the proposal is agreed upon by everyone, if it is sufficiently 
non-trivial, we sometimes require a PEP summarising the issue for future 
reference.)

As the author of one PEP myself, I know how frustrating this process can 
seem when you think that this is a bloody obvious proposal with no 
downside that all right-thinking people ought to instantly recognise as 
a great idea *wink* but nevertheless, in *my opinion* (I don't speak for 
anyone else) I think a PEP would be a good idea.

 
> Guido had some very sensible feedback just a moment ago. I am wondering now
> if we lost Guido because I broke python-ideas etiquette (is a pull request
> not the next step, for example? I never got a good answer on the process),
> or because this thread is just constantly being derailed.

I don't speak for Guido, but it might simply be he isn't invested 
enough in *this specific issue* to spend the time wading through a 
long thread. (That's another reason why a PEP is sometimes valuable.) 
Perhaps he's still on holiday and only has limited time to spend on 
this.

If I were in your position, my next step would be to write a new 
post summarising the thread so far:

- a brief summary of the nature of the problem;
- why you think a solution (whatever that solution turns out to be)
  should be in the stdlib rather than a third-party library;
- what you think the solution should be;
- and give a fair critique of the alternatives suggested so far and
  why you thik that they aren't suitable.

That's the same sort of information given in a PEP, but without having 
to go through the formal PEP process. That might be enough to gain 
consensus on what happens next -- and maybe even agreement that a formal 
and more detailed PEP is not needed.

Oh, and in case you're thinking this is all a great PITA, it might help 
if you read these to get an understanding of why things are as they are:

https://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html

https://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html


Good luck!


-- 
Steve