what to do with multiple BOMs

[Richard Damon]

By the rules of Unicode, that character, if not the very first character of the file, should be treated as a “zero-width non-breaking space”, it is NOT a BOM character there.

It’s presence in the files is almost certainly an error, and being caused by broken software or software processing files in a manner that it wasn’t designed for.

> On Aug 19, 2021, at 1:48 PM, Robin Becker <robin at reportlab.com> wrote:
> 
> Channeling unicode text experts and xml people:
> 
> I have xml entity with initial bytes ff fe ff fe which the file command says is
> UTF-16, little-endian text.
> 
> I agree, but what should be done about the additional BOM.
> 
> A test output made many years ago seems to keep the extra BOM. The xml context is
> 
> 
> xml file 014.xml
> <!DOCTYPE doc [
> <!ELEMENT doc (#PCDATA)>
> <!ENTITY e SYSTEM "014.ent">
> ]>
> <doc>&e;</doc
> 
> the entitity file 014.ent is bombomdata
> 
> b'\xff\xfe\xff\xfed\x00a\x00t\x00a\x00'
> 
> The old saved test output of processing is
> 
> b'<doc>\xef\xbb\xbfdata</doc>'
> 
> which implies seems as though the extra BOM in the entity has been kept and processed into a different BOM meaning utf8.
> 
> I think the test file is wrong and that multiple BOM chars in the entiry should have been removed.
> 
> Am I right?
> --
> Robin Becker
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list