[issue10370] py3 readlines() reports wrong offset for UnicodeDecodeError

[Brian Warner]

Brian Warner <warner at users.sourceforge.net> added the comment:

> Use .readline() to locate an invalid byte is not the right algorithm. If
> you would like to do that, you should open the file in binary mode and
> decodes the content yourself, chunk by chunk. Or if you manipulate small
> files, you can use .read() as you wrote.

Oh, I agree that readline() is inappropriate as a validation tool. My
specific complaint is that the error message is misleading. I hit a message
like this:

 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 26: ordinal not in range(128)

and wanted to know if the file was UTF-8, or latin-1, or some other encoding,
so I wanted to see that 0xe2 in context. The message said to look at offset
26, but the actual problem might be at 4122, or 8218, etc. It took me several
minutes (and hexdump and grepping for ' e2 ') to find the character and
figure out what was going on.

Perhaps, if the error message cannot report a correct offset, then it
shouldn't be reporting an offset at all.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10370>
_______________________________________