23 Jan
2021
23 Jan
'21
11:10 p.m.
On Sat, Jan 23, 2021, at 08:00, Stephen J. Turnbull wrote:
I see very little use in detecting the BOMs. I haven't seen a UTF-16 BOM in the wild in a decade (as usual for me, that's Japan-specific, and may be limited to the academic community as well), and the UTF-8 BOM is a no-op if the default is UTF-8 anyway.
It's not *entirely* a no-op, you'd want the decoder to consume the leading BOM rather than returning '\ufeff' on the first read. And AIUI they're much more common on Windows (being able to detect UTF-16 *without* BOMs might be useful as well, but has historically been a source of problems on Windows) - until recently all UTF-8 or UTF-16 files saved with notepad would have them.