Unicode BOM marks

["Martin v. Löwis"]

Francis Girard wrote:
> If I understand well, into the UTF-8 unicode binary representation, some 
> systems add at the beginning of the file a BOM mark (Windows?), some don't.
> (Linux?). Therefore, the exact same text encoded in the same UTF-8 will 
> result in two different binary files, and of a slightly different length. 
> Right ?

Mostly correct. I would prefer if people referred to the thing not as
"BOM" but as "UTF-8 signature", atleast in the context of UTF-8, as
UTF-8 has no byte-order issues that a "byte order mark" would deal with.
(it is correct to call it "BOM" in the context of UTF-16 or UTF-32).

Also, "some systems" is inadequate. It is not so much the operating
system that decides to add or leave out the UTF-8 signature, but much
more the application writing the file. Any high-quality tool will accept
the file with or without signature, whether it is a tool on Windows
or a tool on Unix.

I personally would write my applications so that they put the signature
into files that cannot be concatenated meaningfully (since the
signature simplifies encoding auto-detection) and leave out the 
signature from files which can be concatenated (as concatenating the
files will put the signature in the middle of a file).


> I guess that this leading BOM mark are special marking bytes that can't be, in 
> no way, decoded as valid text.
> Right ?

Wrong. The BOM mark decodes as U+FEFF:

 >>> codecs.BOM_UTF8.decode("utf-8")
u'\ufeff'

This is what makes it a byte order mark: in UTF-16, you can tell the
byte order by checking whether it is FEFF or FFFE. The character U+FFFE
is an invalid character, which cannot be decoded as valid text
(although the Python codec will decode it as invalid text).

> I also guess that this leading BOM mark is silently ignored by any unicode 
> aware file stream reader to which we already indicated that the file follows 
> the UTF-8 encoding standard.
> Right ?

No. It should eventually be ignored by the application, but whether the
stream reader special-cases it or not is depends on application needs.

> If so, is it the case with the python codecs decoder ?

No; the Python UTF-8 codec is unaware of the UTF-8 signature. It reports
it to the application when it finds it, and it will never generate the
signature on its own. So processing the UTF-8 signature is left to the
application in Python.

> In python documentation, I see theseconstants. The documentation is not clear 
> to which encoding these constants apply. Here's my understanding :
> 
> BOM : UTF-8 only or UTF-8 and UTF-32 ?

UTF-16.

> BOM_BE : UTF-8 only or UTF-8 and UTF-32 ?
> BOM_LE : UTF-8 only or UTF-8 and UTF-32 ?

UTF-16

> Why should I need these constants if codecs decoder can handle them without my 
> help, only specifying the encoding ?

Well, because the codecs don't. It might be useful to add a
"utf-8-signature" codec some day, which generates the signature on
encoding, and removes it on decoding.

Regards,
Martin