What happens when python seeks a text file
Cameron Simpson
cs at zip.com.au
Wed Jul 29 03:58:55 EDT 2015
On 29Jul2015 07:52, dieter <dieter at handshake.de> wrote:
>"=?GBK?B?wO68zsX0?=" <lijpbasin at 126.com> writes:
>> Hi, I tried using seek to reverse a text file after reading about the
>> subject in the documentation:
>> https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects
>> https://docs.python.org/3/library/io.html#io.TextIOBase.seek
>> ...
>> However, an exception is raised if a file with the same content encoded in
>> GBK is provided:
>> $ ./reverse_text_by_seek3.py Moon-gbk.txt
>> [0, 7, 8, 19, 21, 32, 42, 53, 64]
>> µÍͷ˼¹ÊÏç
>> ¾ÙÍ·ÍûÃ÷ÔÂ
>> Traceback (most recent call last):
>> File "./reverse_text_by_seek3.py", line 21, in <module>
>> print(f.readline(), end="")
>> UnicodeDecodeError: 'gbk' codec can't decode byte 0xaa in position 8: illegal multibyte sequence
>
>The "seek" works on byte level while decoding works on character level
>where some characters can be composed of several bytes.
>
>The error you observe indicates that you have "seeked" somewhere
>inside a character, not at a legal character beginning.
>
>That you get an error for "gbk" and not for "utf-8" is a bit of
>an "accident". The same problem can happen for "utf-8" but the probability
>might by sligtly inferior.
>
>Seek only to byte position for which you know that they are also
>character beginnings -- e.g. line beginnings.
You may also keep in mind that while you can't do arithmetic on these things
without knowning the length of the _encoded_ text, what you can do is note the
value returned by f.tell() whenever you like. If you are reading a text file
(== an encoding of the text in a specific character set, be it GBK or UTF8)
then after any read you will be on a character boundary, and can return there.
Actually, on reflection, there may be some character encodings where this is
not true; I think some encodings of Japanese employ some kind of mode shift
sequence, so you might need knowledge of those - a plain seek() might not be
enough. But for any encoding where the character encoded at a spot is
everything needed then a seek() to any position obtained by tell() should be
reliable.
In short: line beginnings are not the only places where you can safely seek.
Though they may be conveniently available.
Cheers,
Cameron Simpson <cs at zip.com.au>
More information about the Python-list
mailing list