soupparser raises a lot of ValueErrors
Hi Mailinglist, I'm writing a feed reader and wanted to make sure, even incorrect xml and html will be parsed without disturbances, so I chose lxml.html.soupparser to do this. Bad thing is, that about the half of regular xml-feeds won't parse since soupparser raises ValueErrors like that: ... File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1186, in _feed SGMLParser.feed(self, markup) File "/usr/lib/python2.7/sgmllib.py", line 104, in feed self.goahead(0) File "/usr/lib/python2.7/sgmllib.py", line 186, in goahead self.handle_charref(name) File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1395, in handle_charref data = unichr(int(ref)) ValueError: invalid literal for int() with base 10: 'x202a' When searching the web for this issue, I saw, it can happen if the feed is malformed. But shouldn't that *not* matter? Greets
I think you should post your code block to the mail-list for having a more concrete response. It seems a encoding issue to me. Encoding proper handling is irrespective to the package or module you're using. In python the encoding is used for i/o and internally depends on the python interpreter implementation/ and environment is running. For example when your using shell, python output might be in ascii encoding or other than utf-8 or even utf-8, but in eclipse IDE python output encoding is always in utf-8. So in general use codecs built in module for having a literal encoding handling, making your code run smoothly independently to the running environment. From any string to utf-8 use codecs.decode() and from utf-8 to any encoding string use codecs.encode(). This applies for output in a file or shell. I hope that helped a bit Cheers! Dimitrios On 06/05/2011 11:47 AM, Peter Schwede wrote:
Hi Mailinglist,
I'm writing a feed reader and wanted to make sure, even incorrect xml and html will be parsed without disturbances, so I chose lxml.html.soupparser to do this.
Bad thing is, that about the half of regular xml-feeds won't parse since soupparser raises ValueErrors like that:
... File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1186, in _feed SGMLParser.feed(self, markup) File "/usr/lib/python2.7/sgmllib.py", line 104, in feed self.goahead(0) File "/usr/lib/python2.7/sgmllib.py", line 186, in goahead self.handle_charref(name) File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1395, in handle_charref data = unichr(int(ref)) ValueError: invalid literal for int() with base 10: 'x202a'
When searching the web for this issue, I saw, it can happen if the feed is malformed. But shouldn't that *not* matter?
Greets _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml@garetjax.info https://mailman-mail5.webfaction.com/listinfo/lxml
Le 05/06/2011 12:16, Dimitrios Pritsos a écrit :
It seems a encoding issue to me.
Assuming it is, the feedparser documentation has a section about character encoding: http://www.feedparser.org/docs/character-encoding.html The almost-last-resort thing it tries is using the chardet library: http://chardet.feedparser.org/ Feedparser also has a ... feed parser which tries hard to parse invalid feeds, but you may have reasons to write your own. However, this particular issue looks more like BeautifulSoup is trying to read an XML entity as decimal while it is hexadecimal. Peter, can you give a sample of the feed you are trying to parse? In particular, the context of where 'x202a' appears. Regards, -- Simon Sapin
On 06/05/2011 02:11 PM, Simon Sapin wrote:
Le 05/06/2011 12:16, Dimitrios Pritsos a écrit :
It seems a encoding issue to me. Assuming it is, the feedparser documentation has a section about character encoding: http://www.feedparser.org/docs/character-encoding.html
The almost-last-resort thing it tries is using the chardet library: http://chardet.feedparser.org/
Feedparser also has a ... feed parser which tries hard to parse invalid feeds, but you may have reasons to write your own.
However, this particular issue looks more like BeautifulSoup is trying to read an XML entity as decimal while it is hexadecimal. Peter, can you give a sample of the feed you are trying to parse? In particular, the context of where 'x202a' appears.
Regards, Nice lib Simon, have you checked it for bugs such as memory leakage? a critical factor for choosing it when need to be employed in multi-processing code. I might use it soon to compare its results to my custom html2vector code. THX!
c = '\x202a' c.decode() u' 2a'
this is what I get in ipython in ubuntu shell. I think Simon is right Petter, you should provide some sample data that is leading to this error. cheers Dimitrios On 06/05/2011 11:47 AM, Peter Schwede wrote:
Hi Mailinglist,
I'm writing a feed reader and wanted to make sure, even incorrect xml and html will be parsed without disturbances, so I chose lxml.html.soupparser to do this.
Bad thing is, that about the half of regular xml-feeds won't parse since soupparser raises ValueErrors like that:
... File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1186, in _feed SGMLParser.feed(self, markup) File "/usr/lib/python2.7/sgmllib.py", line 104, in feed self.goahead(0) File "/usr/lib/python2.7/sgmllib.py", line 186, in goahead self.handle_charref(name) File "/usr/lib/python2.7/dist-packages/BeautifulSoup.py", line 1395, in handle_charref data = unichr(int(ref)) ValueError: invalid literal for int() with base 10: 'x202a'
When searching the web for this issue, I saw, it can happen if the feed is malformed. But shouldn't that*not* matter?
Greets _________________________________________________________________ Mailing list for the lxml Python XML toolkit -http://lxml.de/ lxml@garetjax.info https://mailman-mail5.webfaction.com/listinfo/lxml
Mailing list for the lxml Python XML toolkit -http://lxml.de/ lxml@garetjax.info https://mailman-mail5.webfaction.com/listinfo/lxml
Le 05/06/2011 13:48, Dimitrios Pritsos a écrit :
Nice lib Simon, have you checked it for bugs such as memory leakage? a critical factor for choosing it when need to be employed in multi-processing code. I might use it soon to compare its results to my custom html2vector code. THX!
Both libs seem to be well documented, well tested and (somewhat) widely used but I have used neither extensively so I can’t say about memory leaks or general quality. If you do find a memory leak however, you should probably report a bug to the lib.
c = '\x202a' c.decode() u' 2a'
You should not confuse Python literal strings and XML character reference. In a Python string, \x followed by two hexadecimal digits represents the byte with that number. In your example \x20 is a space, 2 and a are normal characters. Unicode strings also have \u followed by four hexadecimal digits, so you example should be '\u202a'. However that particular character is a control character for text direction, so printing it won’t show much: http://www.fileformat.info/info/unicode/char/202a/index.htm In XML, '&#x' followed by an hexadecimal number followed by ';' (eg. for the space) or '&#' followed by a decimal number followed by ';' (eg. for the same space) is a character reference and should be replaced by parsers such as BeautifulSoup by the Unicode character with that number. See the reference for both: http://docs.python.org/reference/lexical_analysis.html#strings http://www.w3.org/TR/xml/#sec-references Going back to Peter’s traceback, it seems that BeautifulSoup is trying to read an hexadecimal character reference as decimal. It could be a bug in BeautifulSoup, but we need a test case to be sure. Peter: try the latest version of BeautifulSoup if it’s not the one you have already. Regards, -- Simon Sapin http://exyr.org
participants (3)
-
Dimitrios Pritsos -
Peter Schwede -
Simon Sapin