[Python-Dev] Quick sum up about open() + BOM

"Martin v. Löwis" martin at v.loewis.de
Sat Jan 9 02:23:07 CET 2010

>>> Antoine would like to check BOM by default, because both options
>>> (system locale vs checking for BOM) is the same thing.
>> To be clear, I am not saying it is the same thing. What I think is 
>> that it would be a mistake to use a mildly unreliable heuristic by
>> default (the locale + device encoding heuristic) but refuse to
>> trust a more reliable heuristic (the BOM-based detection
>> algorithm).
> I concur. On Windows both UTF-8 and signature are very common, yet
> the platform default is the truly awful CP1252.

While I would support combining BOM detection in the case where a file
is opened for reading and no encoding is specified, I see two problems:
a) if a seek operations is performed before having looked at the BOM,
   no determination would have been made
b) what encoding should it use on writing?


More information about the Python-Dev mailing list