<div class="gmail_quote">On Thu, Dec 22, 2011 at 11:30 AM, Rami Chowdhury <span dir="ltr"><<a href="mailto:rami.chowdhury@gmail.com">rami.chowdhury@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>Could you try using the 'open' function from the 'codecs' module?</div></div></blockquote><div><br></div><div>I believe this is what you meant:</div><div><br></div><div><div>file = codecs.open(p + "2.txt", "r", "utf-8")</div>
<div>for line in file:</div><div> print line</div><div><br></div><div>but got this error:</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td><font color="#909090"><tt><small> 141</small> file = codecs.open(p + "2.txt", "r", "utf-8")<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 142</small> for line in file:<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 143</small> print line<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 144</small> <br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><strong>line</strong> = '<font color="#c040c0">\r\n</font>', <strong>file</strong> = <open file 'index2.txt', mode 'rb'></font></small></td></tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>next</strong>(self=<open file 'index2.txt', mode 'rb'>)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 492</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 493</small> """ Return the next decoded line from the input stream."""<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 494</small> return self.reader.next()<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 495</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 496</small> def __iter__(self):<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>reader</strong> = <open file 'index2.txt', mode 'rb'>, self.reader.<strong>next</strong> = <bound method StreamReader.next of <open file 'index2.txt', mode 'rb'>></font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>next</strong>(self=<open file 'index2.txt', mode 'rb'>)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 429</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 430</small> """ Return the next decoded line from the input stream."""<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 431</small> line = self.readline()<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 432</small> if line:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 433</small> return line<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090">line <em>undefined</em>, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>readline</strong> = <bound method StreamReader.readline of <open file 'index2.txt', mode 'rb'>></font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>readline</strong>(self=<open file 'index2.txt', mode 'rb'>, size=None, keepends=True)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 344</small> # If size is given, we call read() only once<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 345</small> while True:<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 346</small> data = self.read(readsize, firstline=True)<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 347</small> if data:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 348</small> # If we're at a "\r" read one extra character (which might<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090">data <em>undefined</em>, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>read</strong> = <bound method StreamReader.read of <open file 'index2.txt', mode 'rb'>>, <strong>readsize</strong> = 72, firstline <em>undefined</em>, <em>builtin</em> <strong>True</strong> = True</font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>read</strong>(self=<open file 'index2.txt', mode 'rb'>, size=72, chars=-1, firstline=True)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 291</small> data = self.bytebuffer + newdata<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 292</small> try:<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 293</small> newchars, decodedbytes = self.decode(data, self.errors)<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 294</small> except UnicodeDecodeError, exc:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 295</small> if firstline:<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><strong>newchars</strong> = u'', <strong>decodedbytes</strong> = 0, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>decode</strong> = <built-in function utf_8_decode>, <strong>data</strong> = '<font color="#c040c0">\xe1</font> intentado para ellos bastante sabios para discernir lo obvio. Tales perso', self.<strong>errors</strong> = 'strict'</font></small></td>
</tr></tbody></table><p><strong>UnicodeDecodeError</strong>: 'utf8' codec can't decode bytes in position 0-2: invalid data
<br><tt><small> </small> </tt>args =
('utf8', '<font color="#c040c0">\xe1</font> intentado para ellos bastante sabios para discernir lo obvio. Tales perso', 0, 3, 'invalid data')
<br><tt><small> </small> </tt>encoding =
'utf8'
<br><tt><small> </small> </tt>end =
3
<br><tt><small> </small> </tt>object =
'<font color="#c040c0">\xe1</font> intentado para ellos bastante sabios para discernir lo obvio. Tales perso'
<br><tt><small> </small> </tt>reason =
'invalid data'
<br><tt><small> </small> </tt>start =
0</p><p>which is the letter á (a with accent). </p></div></div><div>So I tried with utf-16 and got this error:</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td><font color="#909090"><tt><small> 141</small> file = codecs.open(p + "2.txt", "r", "utf-16")<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 142</small> for line in file:<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 143</small> print line<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 144</small> <br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><strong>line</strong> = '<font color="#c040c0">\r\n</font>', <strong>file</strong> = <open file 'index2.txt', mode 'rb'></font></small></td></tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>next</strong>(self=<open file 'index2.txt', mode 'rb'>)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 492</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 493</small> """ Return the next decoded line from the input stream."""<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 494</small> return self.reader.next()<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 495</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 496</small> def __iter__(self):<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>reader</strong> = <open file 'index2.txt', mode 'rb'>, self.reader.<strong>next</strong> = <bound method StreamReader.next of <open file 'index2.txt', mode 'rb'>></font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>next</strong>(self=<open file 'index2.txt', mode 'rb'>)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 429</small> <br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 430</small> """ Return the next decoded line from the input stream."""<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 431</small> line = self.readline()<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 432</small> if line:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 433</small> return line<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090">line <em>undefined</em>, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>readline</strong> = <bound method StreamReader.readline of <open file 'index2.txt', mode 'rb'>></font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>readline</strong>(self=<open file 'index2.txt', mode 'rb'>, size=None, keepends=True)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 344</small> # If size is given, we call read() only once<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 345</small> while True:<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 346</small> data = self.read(readsize, firstline=True)<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 347</small> if data:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 348</small> # If we're at a "\r" read one extra character (which might<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090">data <em>undefined</em>, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>read</strong> = <bound method StreamReader.read of <open file 'index2.txt', mode 'rb'>>, <strong>readsize</strong> = 72, firstline <em>undefined</em>, <em>builtin</em> <strong>True</strong> = True</font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/codecs.py">/usr/lib64/python2.4/codecs.py</a> in <strong>read</strong>(self=<open file 'index2.txt', mode 'rb'>, size=72, chars=-1, firstline=True)</td>
</tr>
<tr><td><font color="#909090"><tt><small> 291</small> data = self.bytebuffer + newdata<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 292</small> try:<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 293</small> newchars, decodedbytes = self.decode(data, self.errors)<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 294</small> except UnicodeDecodeError, exc:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 295</small> if firstline:<br>
</tt></font></td></tr>
<tr><td><small><font color="#909090">newchars <em>undefined</em>, decodedbytes <em>undefined</em>, <strong>self</strong> = <open file 'index2.txt', mode 'rb'>, self.<strong>decode</strong> = <bound method StreamReader.decode of <open file 'index2.txt', mode 'rb'>>, <strong>data</strong> = '<span class="text"><font color="#c040c0">\r\n</font><i>Noticia: Este sitio web entre este portal est<font color="#c040c0">\xe1</font> i', self.<strong>errors</strong> = 'strict'</font></small></td>
</tr></tbody></table>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td bgcolor="#d8bbff"><big> </big><a href="file:///usr/lib64/python2.4/encodings/utf_16.py">/usr/lib64/python2.4/encodings/utf_16.py</a> in <strong>decode</strong>(self=<open file 'index2.txt', mode 'rb'>, input='<span class="text"><font color="#c040c0">\r\n</font><i>Noticia: Este sitio web entre este portal est<font color="#c040c0">\xe1</font> i', errors='strict')</td>
</tr>
<tr><td><font color="#909090"><tt><small> 47</small> self.decode = codecs.utf_16_be_decode<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 48</small> elif consumed>=2:<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt><small> 49</small> raise UnicodeError,"UTF-16 stream does not start with BOM"<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt><small> 50</small> return (object, consumed)<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt><small> 51</small> <br>
</tt></font></td></tr>
<tr><td><small><font color="#909090"><em>builtin</em> <strong>UnicodeError</strong> = <class exceptions.UnicodeError></font></small></td></tr></tbody></table><p><strong>UnicodeError</strong>: UTF-16 stream does not start with BOM
<br><tt><small> </small> </tt>args =
('UTF-16 stream does not start with BOM',) </p></div></div>