[Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

[Michael Foord]

On 28/06/2011 18:06, Terry Reedy wrote:
> On 6/28/2011 10:46 AM, Paul Moore wrote:
>
>> I use Windows, and come from the UK, so 99% of my text files are
>> ASCII. So the majority of my code will be unaffected. But in the
>> occasional situation where I use a £ sign, I'll get encoding errors,
>
> I do not understand this. With utf-8 you would never get a string 
> encoding error.
>

I assumed he meant that files written out as utf-8 by python would then 
be read in using the platform encoding (i.e. not utf-8 on Windows) by 
the other applications he is inter-operating with. The error would not 
be in Python but in those applications.

>> where currently things will "just work".
>
> As long as you only use the machine-dependent restricted character set.
>

Which is the situation he is describing. You do go into those details 
below, and which choice is "correct" depends on which trade-off you want 
to make.

For the sake of backwards compatibility we are probably stuck with the 
current trade-off however - unless we deprecate using open(...) without 
an explicit encoding.

All the best,

Michael

> > And the failures will be data dependent, and hence intermittent
> > (the worst type of problem).
>
> That is the situation now, with platform/machine dependencies added in.
> Some people share code with other machines, even locally.
>
>> So, in effect, you propose making the default favour writing
>> multiplatform portable code at the expense of quick and dirty scripts?
>
> Let us frame it another way. Should Python installations be compatible 
> with other Python installations, or with the other apps on the same 
> machine? Part of the purpose of Python is to cover up platform 
> differences, to the extent possible (and perhaps sensible -- there is 
> the argument). This was part of the purpose of writing our own io 
> module instead of using the compiler stdlib. The evolution of floating 
> point math has gone in the same direction. For instance, float now 
> expects uniform platform-independent Python-dependent names for 
> infinity and nan instead of compiler-dependent names.
>
> As for practicality. Notepad++ on Windows offers ANSI, utf-8 (w,w/o 
> BOM), utf-16 (big/little endian). I believe that ODF documents are 
> utf-8 encoded xml (compressed or not). My original claim for this 
> proposal was/is that even Windows apps are moving to uft-8 and that 
> someday making that the default for Python everywhere will be the 
> obvious and sensible thing.
>


-- 
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html