UTF-8

[Eric Brunel]

On Sat, 10 Mar 2007 15:00:04 +0100, Olivier Verdier <chtito at gmail.com>  
wrote:
[snip]
> The default encoding i wish to set is UTF-8 since it encodes unicode and  
> is nowadays the standard encoding.

I can't agree with that: there are still many tools completely ignoring  
the encoding problem, and just saving sequences of bytes for any text.  
This results in letting the current OS deciding what encoding is used, and  
this is still not always UTF-8. Just consider how many web pages are still  
encoded in a local encoding without even any specification of what it may  
be in the HTML header, or how many times you see a [?] or a black square  
replacing a character in newsgroups or mailing lists. UTF-8 *should* be  
the standard encoding, but certainly isn't...

> Having ascii as a default encoding is a pretty strange choice since  
> ascii encodes only 127 characters (suitable only for English, without  
> fancy symbols like €) whereas utf-8 encodes virtually infinitely many  
> characters (suitable for all languages). Besides, utf-8 is a superset of  
> ascii so having it by default won't harm anything.

On the contrary, it's IMHO a very sensible thing to do: any character with  
a code > 127 could be in any encoding. So supposing an encoding would be  
guessing, and part of the Python Zen is: "in the face of ambiguity, refuse  
the temptation to guess". The problem is here that if the encoding is not  
UTF-8, things can seem to work, but give completely wrong results. So the  
decision was apparently to use the minimal common set for everything,  
which is ASCII; this seems quite logical to me.

(BTW, one could even argue that the default encoding should be  
non-existent, as I think there are some non-latin encodings that do not  
even include ASCII [I think I saw a Chinese encoding like that one  
day...]. So even supposing that ASCII will work is sometimes wrong...)

> At the very least there should be a configuration file that allows to  
> set a default encoding (although setting anything else than utf-8 should  
> be advised against). Such a feature would be greatly appreciated by all  
> of us non native English speakers around the world.

Well, I *am* a non-native English speaker, and after the first "why should  
I bother about all this encoding stuff?" period, which seems to happen to  
everybody, I finally understood the logic behind Python choices, and why I  
*should* bother about encodings. Whatever the default encoding is, and  
whatever you do, encodings will still be a problem for some time. The  
transition phase from local encodings to universal ones is likely to last  
a few years more, if not a few decades...

HTH
-- 
python -c "print ''.join([chr(154 - ord(c)) for c in  
'U(17zX(%,5.zmz5(17l8(%,5.Z*(93-965$l7+-'])"