Nicholas Bastin wrote:
On Apr 5, 2005, at 6:19 AM, M.-A. Lemburg wrote:
Note that the UTF-16 codec is strict w/r to the presence of the BOM mark: you get a UnicodeError if a stream does not start with a BOM mark. For the UTF-8-SIG codec, this should probably be relaxed to not require the BOM.
I've actually been confused about this point for quite some time now, but never had a chance to bring it up. I do not understand why UnicodeError should be raised if there is no BOM. I know that PEP-100 says:
'utf-16': 16-bit variable length encoding (little/big endian)
Note: 'utf-16' should be implemented by using and requiring byte order marks (BOM) for file input/output.
But this appears to be in error, at least in the current unicode standard. 'utf-16', as defined by the unicode standard, is big-endian in the absence of a BOM:
3.10.D42: UTF-16 encoding scheme: ...
- The UTF-16 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian.
The problem is "in the absence of a higher level protocol": the codec doesn't know anything about a protocol - it's the application using the codec that knows which protocol get's used. It's a lot safer to require the BOM for UTF-16 streams and raise an exception to have the application decide whether to use UTF-16-BE or the by far more common UTF-16-LE.
Unlike for the UTF-8 codec, the BOM for UTF-16 is a configuration parameter, not merely a signature.
In terms of history, I don't recall whether your quote was already in the standard at the time I wrote the PEP. You are the first to have reported a problem with the current implementation (which has been around since 2000), so I believe that application writers are more comfortable with the way the UTF-16 codec is currently implemented. Explicit is better than implicit :-)
The current implementation of the utf-16 codecs makes for some irritating gymnastics to write the BOM into the file before reading it if it contains no BOM, which seems quite like a bug in the codec.
The codec writes a BOM in the first call to .write() - it doesn't write a BOM before reading from the file.
I allow for the possibility that this was ambiguous in the standard when the PEP was written, but it is certainly not ambiguous now.