Mailman 3 November 2009 - lxml - The Python XML Toolkit

Re: [lxml-dev] etree Parser corrupting HTML
by Stefan Behnel 16 Nov '09

16 Nov '09

Clif Swiggett, 16.11.2009 17:29: > Thanks for the tips to tighten up my code. > Do you have any suggestions for how best to detect whether a source file > is html or xml? I'm harvesting arbitrary files off the web ... You can either pass it into the XML parser and fall back to the HTML parser if that fails (which it most likely will for the large majority of HTML files), or, if you expect more HTML pages than XML pages, you can read the page into a string and search for the XHTML namespace before passing it into the parser. If you want to parse arbitrary web pages, you will likely end up using some kind of parser cascade anyway to make sure you get the best conformance and speed in the normal cases and the best error recovery in the pathological cases. Remember that lxml's parser is very fast, so it's ok to parse things multiple times with different parser configurations. Stefan

1 0

Re: [lxml-dev] inconsistency in the text nodes?
by Hervé Cauwelier 16 Nov '09

16 Nov '09

On 16/11/2009 10:18, Stefan Behnel wrote: > > Hervé Cauwelier, 16.11.2009 10:07: >> I inherit from the Python unicode class, and wrap _Element*Result's >> parent in my own element class. > > That doesn't tell me why you would want to instantiate them yourself. It's because of the first test case where I test the construction of my own instances. I give an instance of text result without using XPath (which is tested in another test case). There's a factory function but it's not exposed to the Python level. Is it just because Cython requires variable initialization that these are declared in _ElementUnicodeResult but not in _ElementStringResult? I didn't noticed at first glance that _ElementStringResult is pure Python. I'm not familiar with Cython. Hervé

2 1

[lxml-dev] problem with etree.tostring and special characters inxml-text-attributes
by SCHLÖGL Thomas 16 Nov '09

16 Nov '09

Hi, I have a problem with the replacement of special characters (xml-entities) in xml-text-properties of the etree.tostring method: After etree.tostring the characters &,<,> are replaced with & < > but ' and " remain in the text-properties. e.g.: <Property Name="Value">final_delim=end, record_delim='\n', delim='|', quote=double</Property> But I need: <Property Name="Value">final_delim=end, record_delim='\n', delim='|', quote=double</Property> I wrote a method that iterates through all text properties after the tostring-method and replaces ' and " of this text-properties in the serialized string, but with large xml-files this consumes a lot of time. Does anybody have a better solution? Kind Regards, Thomas Ing. Thomas Schlögl 4266-UGG3C1 / CFO Datawarehouse Core Solutions Member of the Crossfunctions-Team UniCredit Global Information Services S.p.A. (UGIS S.p.A.) Zweigniederlassung Österreich Nordbergstrasse 13 A-1090 Vienna Room: TZB-6.18 Tel: +43 1 71730 50499 Mobile: - mailto:thomas.schloegl@wave-solutions.com <mailto:anton.wazin@wave-solutions.com>

2 1

[lxml-dev] FW: problem with etree.tostring and special characters inxml-text-attributes
by SCHLÖGL Thomas 16 Nov '09

16 Nov '09

Isn't there anybody who can give a comment to my question below (e.q. you're too stupid for lxml) ? Regards, Thomas From: SCHLÖGL Thomas Sent: Monday, November 09, 2009 4:20 PM To: 'lxml-dev(a)codespeak.net' Subject: problem with etree.tostring and special characters inxml-text-attributes Hi, I have a problem with the replacement of special characters (xml-entities) in xml-text-properties of the etree.tostring method: After etree.tostring the characters &,<,> are replaced with & < > but ' and " remain in the text-properties. e.g.: <Property Name="Value">final_delim=end, record_delim='\n', delim='|', quote=double</Property> But I need: <Property Name="Value">final_delim=end, record_delim='\n', delim='|', quote=double</Property> I wrote a method that iterates through all text properties after the tostring-method and replaces ' and " of this text-properties in the serialized string, but with large xml-files this consumes a lot of time. Does anybody have a better solution? Kind Regards, Thomas Ing. Thomas Schlögl 4266-UGG3C1 / CFO Datawarehouse Core Solutions Member of the Crossfunctions-Team UniCredit Global Information Services S.p.A. (UGIS S.p.A.) Zweigniederlassung Österreich Nordbergstrasse 13 A-1090 Vienna Room: TZB-6.18 Tel: +43 1 71730 50499 Mobile: - mailto:thomas.schloegl@wave-solutions.com <mailto:anton.wazin@wave-solutions.com>

2 1

[lxml-dev] inconsistency in the text nodes?
by Hervé Cauwelier 16 Nov '09

16 Nov '09

Hi, I've set up the following excerpt to reproduce it: >>> from lxml import etree >>> s = etree._ElementStringResult("herve") >>> u = etree._ElementUnicodeResult(u"Hervé") >>> print s.getparent() Traceback (most recent call last): File "<input>", line 1, in <module> File "extensions.pxi", line 603, in lxml.etree._ElementStringResult.getparent (src/lxml/lxml.etree.c:99519) AttributeError: '_ElementStringResult' object has no attribute '_parent' >>> print u.getparent() None I'd like both of them to have the same behaviour, my personal preference being returning None. I guess it would look like that: --- lxml/extensions.pxi.orig 2009-11-13 09:46:27.760913564 +0100 +++ lxml/extensions.pxi 2009-11-13 09:46:34.869420791 +0100 @@ -597,6 +597,8 @@ return self._parent class _ElementStringResult(str): + cdef _Element _parent + cdef readonly object is_tail + cdef readonly object is_text + cdef readonly object is_attribute + # we need to use a Python class here, str cannot be C-subclassed # in Pyrex/Cython def getparent(self): I'm playing with these directly because I need to wrap them. Regards, Hervé

2 2

[lxml-dev] etree Parser corrupting HTML
by Clif Swiggett 16 Nov '09

16 Nov '09

I've had two mysteries come up recently using etree.parse() (lxml version 2.2.2). Can anyone shed some light on how to work around these? Start with this input ======input.html========= <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <body> <h2 class="speechTitle"> <p>Foo</p> </h2> </body> </html> ====================== ... then parse using ... htmlTree = etree.parse(open("input.html"), parser=etree.HTMLParser()) open("out.xml", "wb").write(etree.tostring(htmlTree)) ... the resulting XML is corrupted in two ways: ======output.html========= <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml"> <body> <h2 class="speechTitle"> </h2> <p>Foo</p> </body> </html> ======================= Problem 1: The namespace declaration is duplicated (resulting in invalid XML) Problem 2: The <p> tag was moved outside of the <h2> tag. Basic dom structure has been re-arranged. Neither of these problems happen when I use the XMLParser (e.g. 'xmlTree = etree.parse(open("test2.html"), parser=etree.XMLParser())'). However I don't know if my source html will be valid XHTML or, more likely, just HTML. Is there a way to get the HTMLParser to do the right thing? Thanks for any suggestions or advice.

3 2

[lxml-dev] lxml 2.2.3 - html.usedoctest bug?
by Martin Aspeli 14 Nov '09

14 Nov '09

Hi, I put this into my doctest: >>> import lxml.html.usedoctest And now I get: Error in test /Users/optilude/Development/Plone/Code/Build/plone/4.0/src/plone.app.blocks/plone/app/blocks/rendering.txt Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.6/unittest.py", line 279, in run testMethod() File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 2327, in runTest test, out=write, clear_globs=False) File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 1497, in run return self.__run(test, compileflags, out) File "/Users/optilude/.buildout/eggs/zope.testing-3.7.7-py2.6.egg/zope/testing/doctest.py", line 1376, in __run if check(example.want, got, self.optionflags): File "/Users/optilude/.buildout/eggs/lxml-2.2.3-py2.6-macosx-10.6-i386.egg/lxml/doctestcompare.py", line 100, in check_output except etree.XMLSyntaxError: NameError: global name 'etree' is not defined This is pretty weird, since 'etree' is definitely imported in doctestcompare.py in my lxml egg. Any ideas? Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

1 0

[lxml-dev] lxml 2.2.4 released
by Stefan Behnel 11 Nov '09

11 Nov '09

Hi all, I just pushed a new bug-fix release to PyPI. The *only* change is a fix for the static build, which was accidentally broken in 2.2.3. There are no code changes in this release. Have fun, Stefan

1 0

[lxml-dev] Entity References resolution
by Emanuele D'Arrigo 10 Nov '09

10 Nov '09

Hi everybody, today I tried this: parser = etree.XMLParser(resolve_entities=False) tree = etree.parse(StringIO("<aRoot>&aReference;</aRoot>"), parser) I was expecting it to simply treat the entity reference as text but it raises an error. What is the parser's parameter resolve_entities for then? Manu

3 2

Re: [lxml-dev] Handling processing instructions
by Chuck Bearden 09 Nov '09

09 Nov '09

On Wed, Oct 28, 2009 at 4:21 AM, Emanuele D'Arrigo <manu3d(a)gmail.com> wrote: > 2009/10/27 Chuck Bearden <cfbearden(a)gmail.com> >> >> On Tue, Oct 27, 2009 at 1:25 PM, Emanuele D'Arrigo <manu3d(a)gmail.com> >> wrote: >> > 1) short of iterating from the root element using repeatedly >> > getprevious(), >> > is there a way to obtain the processing instructions that are listed >> > before >> >> It is possible via XPath: >> >>> xmlTree.getroot().xpath('preceding-sibling::node()') > > Hi Chuck, thanks for your help! > > How is the xpath approach above different from: > > xmlTree.getRoot().getprevious() > > ? Evidently I mis-typed or mis-read the first time I tried the ElementTree way of getting a PI that precedes the root element. The obvious ElementTree way does indeed work. My mistake. >>> from lxml import etree >>> from StringIO import StringIO >>> xmlString = '''<?xml version="1.0"?> ... <?version 1.3?> ... <root> ... <branch> ... <leaf/> ... </branch> ... </root>''' >>> xmlTree = etree.parse(StringIO(xmlString)) >>> xmlTree.getroot().xpath('preceding-sibling::node()') [<?version 1.3?>] >>> xmlTree.getroot().getprevious() <?version 1.3?> >>> > And would I be correct in saying one would still need to iterate "upward" to > find the very first processing instruction? I guess I think of .getprevious() and the preceding-sibling XPath axis as horizontal rather than vertical, so I would say that you need to recur or iterate "backwards" to get all nodes preceding the root element. >>> from lxml import etree >>> from StringIO import StringIO >>> xmlString = '''<?xml version="1.0"?> ... <?first PI?> ... <?version 1.3?> ... <root> ... <branch> ... <leaf/> ... </branch> ... </root>''' >>> xmlTree = etree.parse(StringIO(xmlString)) >>> xmlTree.getroot().xpath('preceding-sibling::node()') [<?first PI?>, <?version 1.3?>] >>> xmlTree.getroot().getprevious() <?version 1.3?> >>> xmlTree.getroot().getprevious().getprevious() <?first PI?> >>> xmlTree.getroot().getprevious().getprevious().getprevious() >>> Chuck

3 4