Mailman 3 November 2009 - lxml - The Python XML Toolkit

[lxml-dev] etree Parser corrupting HTML
by Clif Swiggett Nov. 16, 2009

Nov. 16, 2009

I've had two mysteries come up recently using etree.parse() (lxml version 2.2.2). Can anyone shed some light on how to work around these? Start with this input ======input.html========= <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <body> <h2 class="speechTitle"> <p>Foo</p> </h2> </body> </html> ====================== ... then parse using ... htmlTree = etree.parse(open("input.html"), parser=etree.HTMLParser()) open("out.xml", "wb").write(etree.tostring(htmlTree)) ... the resulting XML is corrupted in two ways: ======output.html========= <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml"> <body> <h2 class="speechTitle"> </h2> <p>Foo</p> </body> </html> ======================= Problem 1: The namespace declaration is duplicated (resulting in invalid XML) Problem 2: The <p> tag was moved outside of the <h2> tag. Basic dom structure has been re-arranged. Neither of these problems happen when I use the XMLParser (e.g. 'xmlTree = etree.parse(open("test2.html"), parser=etree.XMLParser())'). However I don't know if my source html will be valid XHTML or, more likely, just HTML. Is there a way to get the HTMLParser to do the right thing? Thanks for any suggestions or advice.

3 2

[lxml-dev] lxml 2.2.3 - html.usedoctest bug?
by Martin Aspeli Nov. 14, 2009

Nov. 14, 2009

Hi, I put this into my doctest: >>> import lxml.html.usedoctest And now I get: Error in test /Users/optilude/Development/Plone/Code/Build/plone/4.0/src/plone.app.blocks/plone/app/blocks/rendering.txt Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.6/unittest.py", line 279, in run testMethod() File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 2327, in runTest test, out=write, clear_globs=False) File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 1497, in run return self.__run(test, compileflags, out) File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 1376, in __run if check(example.want, got, self.optionflags): File "/Users/optilude/.buildout/eggs/lxml-2.2.3-py2.6-macosx-10.6-i386.egg/lxml/doctestcompare.py", line 100, in check_output except etree.XMLSyntaxError: NameError: global name 'etree' is not defined This is pretty weird, since 'etree' is definitely imported in doctestcompare.py in my lxml egg. Any ideas? Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

1 0

[lxml-dev] lxml 2.2.4 released
by Stefan Behnel Nov. 11, 2009

Nov. 11, 2009

Hi all, I just pushed a new bug-fix release to PyPI. The *only* change is a fix for the static build, which was accidentally broken in 2.2.3. There are no code changes in this release. Have fun, Stefan

1 0

[lxml-dev] Entity References resolution
by Emanuele D'Arrigo Nov. 10, 2009

Nov. 10, 2009

Hi everybody, today I tried this: parser = etree.XMLParser(resolve_entities=False) tree = etree.parse(StringIO("<aRoot>&aReference;</aRoot>"), parser) I was expecting it to simply treat the entity reference as text but it raises an error. What is the parser's parameter resolve_entities for then? Manu

3 2

Re: [lxml-dev] Handling processing instructions
by Chuck Bearden Nov. 9, 2009

Nov. 9, 2009

On Wed, Oct 28, 2009 at 4:21 AM, Emanuele D'Arrigo <manu3d(a)gmail.com> wrote: > 2009/10/27 Chuck Bearden <cfbearden(a)gmail.com> >> >> On Tue, Oct 27, 2009 at 1:25 PM, Emanuele D'Arrigo <manu3d(a)gmail.com> >> wrote: >> > 1) short of iterating from the root element using repeatedly >> > getprevious(), >> > is there a way to obtain the processing instructions that are listed >> > before >> >> It is possible via XPath: >> >>> xmlTree.getroot().xpath('preceding-sibling::node()') > > Hi Chuck, thanks for your help! > > How is the xpath approach above different from: > > xmlTree.getRoot().getprevious() > > ? Evidently I mis-typed or mis-read the first time I tried the ElementTree way of getting a PI that precedes the root element. The obvious ElementTree way does indeed work. My mistake. >>> from lxml import etree >>> from StringIO import StringIO >>> xmlString = '''<?xml version="1.0"?> ... <?version 1.3?> ... <root> ... <branch> ... <leaf/> ... </branch> ... </root>''' >>> xmlTree = etree.parse(StringIO(xmlString)) >>> xmlTree.getroot().xpath('preceding-sibling::node()') [<?version 1.3?>] >>> xmlTree.getroot().getprevious() <?version 1.3?> >>> > And would I be correct in saying one would still need to iterate "upward" to > find the very first processing instruction? I guess I think of .getprevious() and the preceding-sibling XPath axis as horizontal rather than vertical, so I would say that you need to recur or iterate "backwards" to get all nodes preceding the root element. >>> from lxml import etree >>> from StringIO import StringIO >>> xmlString = '''<?xml version="1.0"?> ... <?first PI?> ... <?version 1.3?> ... <root> ... <branch> ... <leaf/> ... </branch> ... </root>''' >>> xmlTree = etree.parse(StringIO(xmlString)) >>> xmlTree.getroot().xpath('preceding-sibling::node()') [<?first PI?>, <?version 1.3?>] >>> xmlTree.getroot().getprevious() <?version 1.3?> >>> xmlTree.getroot().getprevious().getprevious() <?first PI?> >>> xmlTree.getroot().getprevious().getprevious().getprevious() >>> Chuck

3 4

[lxml-dev] ancestor-or-self
by Polat Tuzla Nov. 9, 2009

Nov. 9, 2009

Hi, I'm observing that xpath axis "ancestor-or-self" does not function properly. Below outputs demonstrate the case. In the first one whole tree is printed, and in the second one ancestor-or-self is used, but the result does not differ. I'll open a ticket for this as a bug, unless someone tells me that I'm missing a point. Thanks, Polat Tuzla In [309]: print etree.tostring(root, pretty_print=True) <a> <b> <c/> <x> <z/> </x> </b> </a> In [310]: print etree.tostring(root.xpath("/a/b/c/ancestor-or-self::*")[0], pretty_print=True) .....: <a> <b> <c/> <x> <z/> </x> </b> </a>

4 8

[lxml-dev] Bug? lxml.html produces bogus results if HTML contains control characters
by Joseph Barillari Nov. 8, 2009

Nov. 8, 2009

Hi, I'm a big fan of lxml.html, but I think I've just found a bug in it: Here's a test case. html = """ <html><body> <table> <tr> <td>one</td> <td>\x05two</td> <td>three</td> </tr> </table> </body></html> """ import lxml.html tree = lxml.html.fromstring(html) xpath = "/descendant::table" cells = tree.xpath(xpath)[0].getchildren()[0].getchildren() print [cell.text_content() for cell in cells] # prints ['one', ''] tree = lxml.html.fromstring(html.replace("\x05","")) cells = tree.xpath(xpath)[0].getchildren()[0].getchildren() print [cell.text_content() for cell in cells] # prints ['one', 'two', 'three'] The apparent bug is that lxml.html fails to parse the above HTML properly when it contains a control character (\x05 a.k.a ^E). Obviously, well-formed HTML will not contain such characters, but real-world HTML often does. The library does not report an error, but simply truncates the row at the control character, which made this behavior tricky for me track down. I'm running the following versions: lxml.etree: (2, 2, 2, 0) libxml used: (2, 7, 6) libxml compiled: (2, 7, 5) libxslt used: (1, 1, 24) libxslt compiled: (1, 1, 26) I must admit that I do not know if this is a bug in lxml or in libxml, but I am certainly willing to help investigate. Best, Joe

1 0

[lxml-dev] Fun with unicode errors
by Praktikant3 - SAG Nov. 7, 2009

Nov. 7, 2009

Hi, Maybe you have an idea what could be happening here, otherwise I will (try to) come back with a more complete example. For now I have this small code excerpt that behaves strangely: (isinstance(output_xml, lxml.etree._Element) is True) # The two ET.tostring() invocations below, (1) and (2), show the # following behaviour: # (1) "works" (UnicodeDecodeError about el.text after (2)) # (1) (2) "works" (UnicodeDecodeError about el.text after (2)) # (2) does not work, lxml.etree.SerialisationError: IO_ENCODER about ET.tostsring() (2) ET.tostring(output_xml) # (1) # Make pretty-printing work by removing unnecessary whitespace: for el in output_xml.iter(): ET.tostring(el) # (2) if len(el) and el.text and not el.text.strip(): el.text = None if el.tail and not el.tail.strip(): el.tail = None (1) and (2) are commented out to run them in the different combinations discussed. If you have a hint about what might be the issue there, I'd be very glad to hear it. I'm using this code on WinXP SP3, Python 2.6.4, lxml 2.2.2 (pre-compiled package). Expected behaviour would be to run without raising any exception in any of the runs. Something about output_xml changed, as this code snipped used to work. Thanks, Felix

3 5

[lxml-dev] No module named lxml
by Kevin Ar18 Nov. 7, 2009

Nov. 7, 2009

I'm not really sure how to setup lxml properly, so I need some help. Summary: import lxml gives the following error: ImportError: No module named lxml Background: I first tried out the install instructions here: http://codespeak.net/lxml/installation.html#ms-windows This involved 3 steps: 1) using easy_install lxml 2) Installing libxml 3) Installing libxslt Suffice to say that method never worked right. During all those steps, I probably did something wrong. Next, I tried the Windows binary egg method (after I deleted the previous lxml install). I downloaded: lxml-2.2.2-py2.6-win-amd64.egg and I copied the file to: Python2.6.1\Lib\site-packages\lxml-2.2.2-py2.6-win-amd64.egg Questions: * Is this the correct way to install an egg? (or is there more to it?) * Could, elementree or html5lib packages be causing a conflict? * So, what am I doing wrong? :) Is it something simple that I missed or should it be working the way I have it now? System info: Windows Vista 64bit Python 2.6.4 64bit Contents of my site-packages: babel-0.9.4-py2.6.egg cherrypy-3.1.1-py2.6.egg decorator-3.0.0-py2.6.egg django-1.0.2_final-py2.6.egg django_wikiapp-0.2.0-py2.6.egg 939 easy-install.pth elementtree-1.2.7_20070827_preview-py2.6.egg elixir-0.3.0-py2.6.egg elixir-0.6.1-py2.6.egg FormEncode-1.2.3dev_r0-py2.6.egg Genshi-0.5.1-py2.6.egg Genshi.egg-info html5lib-0.11.1-py2.6 - Copy.zip html5lib-0.11.1-py2.6.egg lxml-2.2.2-py2.6-win-amd64.egg mako-0.2.4-py2.6.egg nose-0.10.4-py2.6.egg paste-1.7.2-py2.6.egg pastedeploy-1.3.3-py2.6.egg pastescript-1.7.3-py2.6.egg posterity-0.6-py2.6.egg Pylons-0.9.7-py2.6.egg pysqlite-2.5.5-py2.6.egg-info pysqlite2 pytidylib-0.1.2-py2.6.egg-info README.txt resolver-0.2.1-py2.6.egg selector-0.8.11-py2.6.egg setuptools-0.6c9-py2.6.egg setuptools.pth sqlalchemy-0.3.11-py2.6.egg sqlalchemy-0.5.3-py2.6.egg tempita-0.2-py2.6.egg tidy tidylib uTidylib-0.2-py2.6.egg-info weberror-0.10.1-py2.6.egg webob-0.9.6.1-py2.6.egg webtest-1.1-py2.6.egg wsgiref-0.1.2-py2.6.egg xlrd xlrd-0.6.1-py2.6.egg-info zzzzzzhtml5lib-0.11.1-py2.6 - Copyzzzzzzzzzzz _________________________________________________________________ Bing brings you maps, menus, and reviews organized in one place. http://www.bing.com/search?q=restaurants&form=MFESRP&publ=WLHMTAG&crea=TEXT…

2 4

[lxml-dev] 2.2.2 binary egg for Mac OS X 10.6
by Martin Aspeli Nov. 2, 2009

Nov. 2, 2009

Hi, We have binary eggs for Python 2.4, 2.5 and 2.6 on Mac OS X 10.5. Unfortunately, we don't have them for 10.6 (Snow Leopard), and the 10.5 egg doesn't work on 10.6 (it crashes on startup). Is there any chance we could have 10.6 eggs? If there are reliable build instructions, I can help build them. Cheers, Martin

2 3

Re: [lxml-dev] lxml 2.2.3 released
by Nicholas Dudfield Nov. 2, 2009

Nov. 2, 2009

Stefan & Co, Great news on 2.2.3. Any ETA for windows 2.5 binaries? Cheers.

1 0

[lxml-dev] About encoding question !
by David Shieh Nov. 2, 2009

Nov. 2, 2009

Hey guys, I recently use lxml to do my HTML parsing, it's really great, and indeed the fastest one compare to other libraries. But since I begin to parse some other pages using gb2312 coding, I've a problem. The output is in here: http://david-paste.cn/paste/50/ Please help me with this, thanks you guys. Regards, David -- ---------------------------------------------- Attitude determines everything ! ----------------------------------------------

2 1