[New-bugs-announce] [issue7651] Python3: guess text file charset using the BOM
report at bugs.python.org
Thu Jan 7 04:03:57 CET 2010
New submission from STINNER Victor <victor.stinner at haypocalc.com>:
If the file starts with a BOM, open(filename) should be able to guess the charset. It would be helpful for many high level modules:
- #7519: ConfigParser
- #7185: csv
- and any module using open() to read a text file
Actually, the user have to choose between UTF-8 and UTF-8-SIG to skip the UTF-8 BOM. For UTF-16, the user have to specify UTF-16-LE or UTF-16-BE, even if the file starts with a BOM (which should be the case most the time).
The idea is to delay the creation of the decoder and the encoder. Just after reading the first chunk: try to guess the charset by searching for a BOM (if the charset is unknown). If the BOM is found, fallback to current guess code (os.device_charset() or locale.getpreferredencoding()).
Concerned charsets: UTF-8, UTF-16-LE, UTF-16-BE, UTF-32-LE, UTF-32-BE. Binary files are not concerned. If the encoding is specified to open(), the behaviour is unchanged.
I wrote a proof of concept, but there are still open issues:
- append mode: should we seek at zero to read the BOM?
old=tell(); seek(0); bytes=read(4); seek(old); search_bom(bytes)
- read+write: should we guess the charset using the BOM if the first action is a write? or only search for a BOM if the first action is a read?
title: Python3: guess text file charset using the BOM
versions: Python 2.7, Python 3.2
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce