Need debugging knowhow for my creeping Unicodephobia
clp2 at rebertia.com
Wed Feb 10 22:43:14 CET 2010
On Wed, Feb 10, 2010 at 1:03 PM, kj <no.email at please.post> wrote:
> In <402ac982-0750-4977-adb2-602b19149d81 at m24g2000prn.googlegroups.com>
Jonathan Gardner <jgardner at jonathangardner.net> writes:
>>It sounds like someone, probably beautiful soup, is trying to turn
>>your strings into unicode. A full stacktrace would be useful to see
>>who did what where.
> Unfortunately, there's not much in the stacktrace:
> Traceback (most recent call last):
> File "./download_tt.py", line 427, in <module >
> x = "%s %s" % (table['id'], table.tr.renderContents())
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 41:
ordinal not in range(128)
Think I've found the problem. According to the BeautifulSoup docs,
a (by default, UTF-8-encoded) str [i.e. byte sequence] as opposed to
unicode[i.e. abstract code point sequence]
Thus, as was said previously, you're combining the unicode from
str from renderContents(),
so Python tries to automatically+implicitly convert the str to unicode by
decoding it as ASCII.
However, it's not ASCII but UTF-8, hence you get the error about it having
Solution: Convert the output of renderContents() back to unicode.
x = u"%s %s" % (table['id'], table.tr.renderContents().decode('utf8'))
Now only unicode objects are being combined.
Your problem is particularly ironic considering how well BeautifulSoup
handles Unicode overall;
I was unable to locate a renderContents() equivalent that returned unicode.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list