[Python-bugs-list] [Bug #122587] xmllib unable to parse "german scharfes ß" in UTF8 format

[noreply@sourceforge.net]

Bug #122587, was updated on 2000-Nov-16 06:10
Here is a current snapshot of the bug.

Project: Python
Category: XML
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Summary: xmllib unable to parse "german scharfes ß" in UTF8 format

Details: The xmllib.XMLParser seems to be unable to parse 
an XML file that contains 0xc3 0x9f (UTF8 representation
of the german ß).

Python 2.0 (Linux i386) always give me the following 
traceback:

suxlap@/tmp/xx(2)% python test.py test.xml
<?xml version="1.0" encoding="UTF-8" ?>

<test>Ã&#376;üöä</test>

Traceback (most recent call last):
  File "test.py", line 20, in ?
    pp.feed(data)
  File "/opt/python-2.0/lib/python2.0/xmllib.py", line 165, in feed
    self.goahead(0)
  File "/opt/python-2.0/lib/python2.0/xmllib.py", line 261, in goahead
    self.syntax_error('illegal character in content')
  File "/opt/python-2.0/lib/python2.0/xmllib.py", line 786, in syntax_error
    raise RuntimeError, 'Syntax error at line %d: %s' % (self.lineno, message)
RuntimeError: Syntax error at line 3: illegal character in content         



Other UTF8 characters seem to work

Follow-Ups:

Date: 2000-Nov-16 10:58
By: Nobody

Comment:
works with accept_utf=1 as additional parameter,
but using xml package instead.
 
-------------------------------------------------------

Date: 2000-Nov-24 01:47
By: sjoerd

Comment:
The problem here is the character reference &#376;. xmllib is from before Python support for Unicode, so it doesn't support any characters that are not representable in 8 bits, and it only really supports iso-8859-1 (latin1), and not even the utf-8 encoding of latin1. It also doesn't do the right thing for character references outside of the ASCII range, although it'll accept characters references in the range 0 - 255 (decimal).

It is too much work to fix this.

I will make available a rewrite of xmllib that has full Unicode support and what's more, is a validating XML parser. The main problem with this rewrite is that it is pretty slow (it uses many, big regular expressions, and compiling those re's is a time consuming task).  Mail me if you want a copy.
-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=122587&group_id=5470