jaroslav.dobrek at gmail.com
Thu Jul 26 09:46:27 CEST 2012
On Jul 25, 8:50 pm, Dave Angel <d... at davea.name> wrote:
> On 07/25/2012 08:09 AM, jaroslav.dob... at gmail.com wrote:
> > On Wednesday, July 25, 2012 1:35:09 PM UTC+2, Philipp Hagemeister wrote:
> >> Hi Jaroslav,
> >> you can catch a UnicodeDecodeError just like any other exception. Can
> >> you provide a full example program that shows your problem?
> >> This works fine on my system:
> >> import sys
> >> open('tmp', 'wb').write(b'\xff\xff')
> >> try:
> >> buf = open('tmp', 'rb').read()
> >> buf.decode('utf-8')
> >> except UnicodeDecodeError as ude:
> >> sys.exit("Found a bad char in file " + "tmp")
> > Thank you. I got it. What I need to do is explicitly decode text.
> > But I think trial and error with moving files around will in most cases be faster. Usually, such a problem occurs with some (usually complex) program that I wrote quite a long time ago. I don't like editing old and complex programs that work under all normal circumstances.
> > What I am missing (especially for Python3) is something like:
> > try:
> > for line in sys.stdin:
> > except UnicodeDecodeError:
> > sys.exit("Encoding problem in line " + str(line_number))
> > I got the point that there is no such thing as encoding-independent lines. But if no line ending can be found, then the file simply has one single line.
> i can't understand your question. if the problem is that the system
> doesn't magically produce a variable called line_number, then generate
> it yourself, by counting
> in the loop.
That was just a very incomplete and general example.
My problem is solved. What I need to do is explicitly decode text when
reading it. Then I can catch exceptions. I might do this in future
I dislike about this solution that it complicates most programs
unnecessarily. In programs that open, read and process many files I
don't want to explicitly decode and encode characters all the time. I
just want to write:
for line in f:
or something like that. Yet, writing this means to *implicitly* decode
text. And, because the decoding is implicit, you cannot say
for line in f: # here text is decoded implicitly
This isn't possible for syntactic reasons.
The problem is that vast majority of the thousands of files that I
process are correctly encoded. But then, suddenly, there is a bad
character in a new file. (This is so because most files today are
generated by people who don't know that there is such a thing as
encodings.) And then I need to rewrite my very complex program just
because of one single character in one single file.
More information about the Python-list