[Python-Dev] PEP 263 considered faulty (for some Japanese)

[M.-A. Lemburg]

Guido van Rossum wrote:
> 
> > My position on this is *not* to introduce more defaults -- explicit
> > is better than implicit and in this particular case (encodings)
> > it'll result in a net win.
> 
> I'd like to believe you.  But the fact that apparently there are
> Japanese users who are willing to give up part of the language and
> library just so they can have an certain default, suggests that
> the need for defaults is a strong force.  Maybe we would've been
> better off leaving sys.setdefaultencoding() enabled -- then those
> people might have put sys.setdefaultencoding("utf-16") at the top of
> their program rather than hacking site.py... :-(

(Actually, they'll tweak sitecustomize.py.) It's good that they
can only apply the change in this one location. Placing it inside
the various modules would cause a maintenance nightmare, removing
one of the great advantages of Python over other languages.

Anyway, I tend to believe that changes to the default encoding
are only rarely needed and then only to overcome problems with
occasional use of Unicode.

It's a myth that you can port a program to Unicode by tweaking
the default encoding to fit your environment alone. 

A true port will have to follow the Unicode object through the 
complete processing chain and apply the needed changes along the 
chain (which is much work, but certainly possible). A different
strategy would be treating text data as binary data and not
using Unicode at all. It all depends on the application scope.

> > > In the light of the post by Atsuo Ishimoto and the responses from both
> > > Marc-Andre Lemburg and Martin von Loewis, however, I'm not sure
> > > whether Suziki Hisao's response represents the Japanese community, and
> > > it's possible that nothing needs to be done.
> >
> > Well, users using non-ASCII coding in their source files
> > should start to be explicit about the encoding (in phase 1
> > they'll get a warning printed which makes them aware of the
> > problem), but other than that, I don't see a need for
> > changes to the strategy.
> 
> Suzuki won't get the warning, because his source files are pure ASCII
> -- but his Unicode string literals will be interpreted as utf-16,
> which will break his programs.  The question is, do we care about him
> and others like him, or do we decide that their habits are bad for
> them and they have to change them?

We do care (after all, the PEP was designed for non-ASCII
users), but it was never intended that we allow encodings
like UTF-16 to be used for Python source code. 

I'm afraid there's nothing much we can do. For UTF-8 they would 
just have to add a single coding comment to all source files, 
but there's nothing we can offer them for UTF-16.

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
______________________________________________________________________
Company & Consulting:                           http://www.egenix.com/
Python Software:                   http://www.egenix.com/files/python/