What the \xc2\xa0 ?!!

Brian D briandenzer at gmail.com
Tue Sep 7 19:01:01 CEST 2010


In an HTML page that I'm scraping using urllib2, a  \xc2\xa0
bytestring appears.

The page's charset = utf-8, and the Chrome browser I'm using displays
the characters as a space.

The page requires authentication:
https://www.nolaready.info/myalertlog.php

When I try to concatenate strings containing the bytestring, Python
chokes because it refuses to coerce the bytestring into ascii.

wfile.write('|'.join(valueList))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
163: ordinal not in range(128)

In searching for help with this issue, I've learned that the
bytestring *might* represent a non-breaking space.

When I scrape the page using urllib2, however, the characters print
as     in a Windows command prompt (though I wouldn't be surprised if
this is some erroneous attempt by the antiquated command window to
handle something it doesn't understand).

If I use IDLE to attempt to decode the single byte referenced in the
error message, and convert it into UTF-8, another error message is
generated:

>>> weird = unicode('\xc2', 'utf-8')

Traceback (most recent call last):
  File "<pyshell#72>", line 1, in <module>
    weird = unicode('\xc2', 'utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc2 in position 0:
unexpected end of data

If I attempt to decode the full bytestring, I don't obtain a human-
readable string (expecting, perhaps, a non-breaking space):

>>> weird = unicode('\xc2\xa0', 'utf-8')
>>> par = ' - '.join(['This is', weird])
>>> par
u'This is - \xa0'

I suspect that the bytestring isn't UTF-8, but what is it? Latin1?

>>> weirder = unicode('\xc2\xa0', 'latin1')
>>> weirder
u'\xc2\xa0'
>>> 'This just gets ' + weirder
u'This just gets \xc2\xa0'

Or is it a Microsoft bytestring?

>>> weirder = unicode('\xc2\xa0', 'mbcs')
>>> 'This just gets ' + weirder
u'This just gets \xc2\xa0'

None of these codecs seem to work.

Back to the original purpose, as I'm scraping the page, I'm storing
the field/value pair in a dictionary with each iteration through table
elements on the page. This is all fine, until a value is found that
contains the offending bytestring. I have attempted to coerce all
value strings into an encoding, but Python doesn't seem to like that
when the string is already Unicode:

valuesDict[fieldString] = unicode(value, 'UTF-8')
TypeError: decoding Unicode is not supported

The solution I've arrived at is to specify the encoding for value
strings both when reading and writing value strings.

for k, v in valuesDict.iteritems():
    valuePair = ':'.join([k, v.encode('UTF-8')])
    [snip] ...
    wfile.write('|'.join(valueList))

I'm not sure I have a question, but does this sound familiar to any
Unicode experts out there?

How should I handle these odd bytestring values? Am I doing it
correctly, or what could I improve?

Thanks!





More information about the Python-list mailing list