On Fri, Jan 19, 2018 at 06:35:30PM +0000, Rob Speer wrote:
It depends on what you want to achieve. You may want to fail, assign a code point from a private area or use a surrogate escape approach.
And the way to express that is with errors='replace', errors='surrogateescape', or whatever, which Python already does. We do not need an explosion of error handlers. This problem can be very straightforwardly solved with encodings, and error handlers can keep doing their usual job on top of encodings.
You could also add a "latin1replace" error handler which simply passes through everything that's undefined as-is.
Nobody asked for this.
Actually, Soni L. seems to have suggested a similar idea in the thread titled "Chaining coders" (codecs). But what does it matter whether someone asked for it? Until this thread, nobody had asked for support for WHATWG encodings either. The question to my mind is whether or not this "latin1replace" handler, in conjunction with existing codecs, will do the same thing as the WHATWG codecs. If I have understood you correctly, I think it will. Have I missed something?
I just don't want to have people start using "web-1252" as encoding simply because they they are writing out text for a web application - they should use "utf-8" instead.
I did ask for input on the name. If the problem is that you think my working name for the encoding is misleading, you could help with that instead of constantly trying to replace the proposal with something different.
Rob, you've come here with a proposal based on an actual problem (web pages with mojibake and broken encodings), an existing solution (a third party library) you dislike, and a suggested new solution you will like (move the encodings into the std lib). That's great, and we need more suggestions like this: concrete use-cases and concrete solutions. But you cannot expect that we're going to automatically agree that: - the problem is something that Python the language has to solve (it seems to be a *browser* problem, not a general programming problem); - the existing solution is not sufficient; and - your proposal is the right solution. All of these things need to be justified, and counter-proposals are part of that. When we make a non-trivial proposal on Python-Ideas, it is very rare that they are so clearly the right solution for the right problem that they get instant approval and you can go straight to the PR. Often there are legitimate questions about all three steps. That's why I suggested earlier that (in my opinion) there needs to be a PEP to summarise the issue, justify the proposal, and counter the arguments against it. (Even if the proposal is agreed upon by everyone, if it is sufficiently non-trivial, we sometimes require a PEP summarising the issue for future reference.) As the author of one PEP myself, I know how frustrating this process can seem when you think that this is a bloody obvious proposal with no downside that all right-thinking people ought to instantly recognise as a great idea *wink* but nevertheless, in *my opinion* (I don't speak for anyone else) I think a PEP would be a good idea.
Guido had some very sensible feedback just a moment ago. I am wondering now if we lost Guido because I broke python-ideas etiquette (is a pull request not the next step, for example? I never got a good answer on the process), or because this thread is just constantly being derailed.
I don't speak for Guido, but it might simply be he isn't invested enough in *this specific issue* to spend the time wading through a long thread. (That's another reason why a PEP is sometimes valuable.) Perhaps he's still on holiday and only has limited time to spend on this. If I were in your position, my next step would be to write a new post summarising the thread so far: - a brief summary of the nature of the problem; - why you think a solution (whatever that solution turns out to be) should be in the stdlib rather than a third-party library; - what you think the solution should be; - and give a fair critique of the alternatives suggested so far and why you thik that they aren't suitable. That's the same sort of information given in a PEP, but without having to go through the formal PEP process. That might be enough to gain consensus on what happens next -- and maybe even agreement that a formal and more detailed PEP is not needed. Oh, and in case you're thinking this is all a great PITA, it might help if you read these to get an understanding of why things are as they are: https://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.ht... https://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python... Good luck! -- Steve