# [IronPython] Weird issue with codecs.BOM_UTF8

Michael Foord fuzzyman at voidspace.org.uk
Tue Nov 10 23:30:26 CET 2009

Leonides Saguisag wrote:
> Hi Michael,
>
> I am using SharpDevelop 3.1 which comes with "2.5.0 (IronPython 2.0.2 (2.0.0.0) on .NET 2.0.50727.3603)".
>
>
Yeah, that's IronPython 2.0 - which is fine but not as good as
IronPython 2.6. ;-)

> So this issue is resolved with IronPython 2.6, then?
>
>

No idea. I can't reproduce the problem with IronPython 2.6 though. Try
installing IronPython 2 and seeing what happens from the interactive
interpreter (whether you can reproduce the problem or not). It is
*possible* that it's caused by the way SharpDevelop generates its
executables, but that's highly unlikely to be the cause of the problem.

All the best,

Michael Foord

> Thanks!
>
> -- Leo
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
> Sent: 2009?11?10? 14:05
> To: Discussion of IronPython
> Subject: Re: [IronPython] Weird issue with codecs.BOM_UTF8
>
> Leonides Saguisag wrote:
>
>> Hi Michael,
>>
>> I just verified the empty string theory that you mentioned and Python25\lib\codecs.py (comes with the standard library in Python 2.5) has the following defined:
>>
>>
>>
> You're using IronPython 2.0 then?
>
> If I use IronPython 2.6 it correctly reports a text file as not starting with the BOM:
>
>  >>> import codecs
>  >>> codecs.BOM_UTF8
> u'\xef\xbb\xbf'
>  >>> lines = open('foo.txt').readlines()  >>> lines ['foo']  >>> lines[0].startswith(codecs.BOM_UTF8)
> False
>
>
> All the best,
>
> Michael Foord
>
>> # UTF-8
>> BOM_UTF8 = '\xef\xbb\xbf'
>>
>>
>> So it is not an empty string.
>>
>> Maybe I am approaching this wrong and you guys can provide me with an alternative way of doing this.  I am trying to read a file and determine if the file is encoded in UTF-8 or not.  The approach I took was to use python's built-in open function to read the text file into an array of strings and check if the first line starts with the UTF-8 byte order mark by using line.startswith(codecs.BOM_UTF8).  As I noted below, this works fine in Python 2.5 but in IronPython it just keeps saying it found a UTF-8 BOM even though there is none present.
>>
>> Thanks!
>>
>> -- Leo
>>
>> -----Original Message-----
>> From: users-bounces at lists.ironpython.com
>> [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael Foord
>> Sent: 2009?11?10? 13:32
>> To: Discussion of IronPython
>> Subject: Re: [IronPython] Weird issue with codecs.BOM_UTF8
>>
>> Leonides Saguisag wrote:
>>
>>
>>> Thank you for taking the time to reply.  Any idea why this would happen in IronPython but not with the standard Python interpreter?  What is weirding me out is that the exact same script behaves differently depending on whether I use IronPython or the standard Python interpreter.
>>>
>>>
>>>
>> Well, if codecs.BOM_UTF8 is set to the empty string (you didn't say if you have tried this yet?) then it would be due to a bug in IronPython somewhere - but at least you would know what was causing it.
>>
>> If it is the empty string, purely speculating, it could be due to the way the .NET framework treats the BOM at the start of strings. Pure speculation though - that might not be the problem at all or it could be caused by something entirely different.
>>
>> In .NET it would be more normal to check for the BOM with bytes, as by the time you have a string you have (usually) decoded already.
>> IronPython 2.X is a bit odd for the .NET framework in this respect.
>>
>> Michael
>>
>>
>>
>>> Thanks!
>>>
>>> -- Leo
>>>
>>> -----Original Message-----
>>> From: users-bounces at lists.ironpython.com
>>> [mailto:users-bounces at lists.ironpython.com] On Behalf Of Michael
>>> Foord
>>> Sent: 2009?11?10? 13:17
>>> To: Discussion of IronPython
>>> Subject: Re: [IronPython] Weird issue with codecs.BOM_UTF8
>>>
>>> Leonides Saguisag wrote:
>>>
>>>
>>>
>>>> Hi everyone,
>>>>
>>>> I am encountering a weird issue with getting to codecs.BOM_UTF8 to work correctly.  I am using SharpDevelop 3.1.
>>>>
>>>> Here is the test script that I put together:
>>>>
>>>>
>>>> import sys
>>>> sys.path.append(r'D:\Python25\Lib')
>>>> import codecs
>>>>
>>>> print sys.version
>>>> myfile = open(r'D:\Temp\text_file_with_utf8_bom.txt', 'r') lines =
>>>> myfile.close()
>>>> if lines[0].startswith(codecs.BOM_UTF8):
>>>> 	print ('UTF-8 BOM detected!')
>>>> else:
>>>> 	print ('UTF-8 BOM not detected!')
>>>>
>>>> myfile = open(r'D:\Temp\text_file_without_utf8_bom.txt', 'r') lines
>>>> =
>>>> myfile.close()
>>>> if lines[0].startswith(codecs.BOM_UTF8):
>>>> 	print ('UTF-8 BOM detected!')
>>>> else:
>>>> 	print ('UTF-8 BOM not detected!')
>>>>
>>>>
>>>> If I run the executable that I get from SharpDevelop this is what I get:
>>>> bin\Debug> Test.exe
>>>> 2.5.0 ()
>>>> UTF-8 BOM detected!
>>>> UTF-8 BOM detected!
>>>>
>>>>
>>>> But if I run the same script using the standard python interpreter, this is what I get:
>>>> bin\Debug> D:\Python25\python.exe ..\..\Program.py
>>>> 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
>>>> (Intel)]
>>>> UTF-8 BOM detected!
>>>> UTF-8 BOM not detected!
>>>>
>>>>
>>>> The script works correctly with the standard python interpreter but for some reason is not working right with IronPython.
>>>>
>>>> Any ideas what is going wrong?
>>>>
>>>>
>>>>
>>>>
>>> I'm not in a position to check right now, but this could happen if codes.UTF8_BOM is set to the empty string.
>>>
>>> Michael
>>>
>>>
>>>
>>>
>>>> Thanks!
>>>>
>>>> Best regards,
>>>> -- Leo
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.ironpython.com
>>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>>
>>>>
>>>>
>>>>
>>> --
>>> http://www.ironpythoninaction.com/
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>>>
>>>
>> --
>> http://www.ironpythoninaction.com/
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
>>
>
>
> --
> http://www.ironpythoninaction.com/
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>

--
http://www.ironpythoninaction.com/