Mailman 3 August 2012 - lxml - The Python XML Toolkit

[lxml-dev] confusing xpath performance characteristics
by jholg＠gmx.de 24 Aug '12

24 Aug '12

Hi, I ran into some performance characteristics of lxml/libxml2 xpath that I find rather confusing: I try to find the @type attribute of a certain element in an XML Schema (which contains lots of complexType definitions with lots of elements in them; unfortunately I can't post the schema): >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.095885038375854492, 0.096823930740356445, 0.096174955368041992] So I think I'm being smart and give a little more path information - reckoning that this should *improve* performance: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.1770780086517334, 0.1775970458984375, 0.17748594284057617] Hm. Performance degrades slightly. I'm adding even more of the path to where my desired elements live in the schema: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [103.79744100570679, 103.83671712875366, 103.61817717552185] What??? >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('/ae/data/pydev/hjoukl/NDM/SVN_CO/TRUNK/ndm/reference/xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType/*/xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.044407129287719727, 0.044126987457275391, 0.044229030609130859] >>> Ok, this version's better than my naive approach, which seems logical to me. But why would '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' perform drastically slower than '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' ? libxml2 problem? Running the same xpaths in Oxygen I don't notice performance differences (can't profile this). Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 5

The L in lxml
by Simon Sapin 16 Aug '12

16 Aug '12

Hi, Does the L in lxml mean something, or is the name just a contraction of libxml2? I took this L for lselect, but you know what they say about naming things… -- Simon Sapin

2 2

Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8
by 7stud 13 Aug '12

13 Aug '12

Thanks! I got lxml installed by setting the environment variable like this: $ export LIBXML2_VERSION=2.8.0 Then doing: $ p3 setup.py build --static-deps And finally: $ p3 setup.py install Initially, I thought I could just follow the alternate Mac OS X instructions: 1) Download libxml2-2.8.0.tar.gz and libxslt-1.1.26.tar.gz and place them in a folder I created called libs inside lxml-2.3.5/. 2) Then issue the command: $ p3 setup.py build --static-deps \ --libxml2-version=2.8.0 \ --libxslt-version=1.1.26 \ (and hit return a couple of times) But the output from that command ended with some warnings: ld: warning: in /Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/lib/libexslt.a, file was built for unsupported file format which is not the architecture being linked (i386) ld: warning: in /Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/lib/libiconv.a, file was built for unsupported file format which is not the architecture being linked (i386) ld: warning: in /Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/lib/libxml2.a, file was built for unsupported file format which is not the architecture being linked (i386) ld: warning: in /Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/lib/libxslt.a, file was built for unsupported file format which is not the architecture being linked (i386) $ -----Original Message----- From: "Stefan Behnel" [stefan_ml(a)behnel.de] Date: 08/13/2012 02:42 PM To: "lxml mailing list" <lxml(a)lxml.de> Subject: Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8 7stud, 13.08.2012 20:18: > From: "Stefan Behnel" > Date: 08/13/2012 09:24 AM > > 7stud, 13.08.2012 10:02: >> $ tar xfvz lxml-2.3.5.tar.gz >> >> $ cd lxml-2.3.5 >> >> $ p3 setup.py build --static-deps >> Building lxml version 2.3.5. >> Latest version of libiconv is 1.14 >> Downloading libiconv into libs/libiconv-1.14.tar.gz >> Unpacking libiconv-1.14.tar.gz into build/tmp >> Latest version of libxml2 is 2.9.0-rc1 >> Downloading libxml2 into libs/libxml2-2.9.0-rc1.tar.gz >> Unpacking libxml2-2.9.0-rc1.tar.gz into build/tmp >> Latest version of libxslt is 1.1.26 >> Downloading libxslt into libs/libxslt-1.1.26.tar.gz >> Unpacking libxslt-1.1.26.tar.gz into build/tmp >> Starting build in build/tmp/libiconv-1.14 >> … >> … >> … > > Could you provide the error message please? It should have turned up right > here. > > >> make[2]: *** [xsltutils.lo] Error 1 >> make[2]: *** Waiting for unfinished jobs.... >> make[1]: *** [all-recursive] Error 1 >> make: *** [all] Error 2 > > ==== > > Is this what you are after? > > libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c xslt.c -o xslt.o > /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c -o xsltutils.lo xsltutils.c > libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c xsltutils.c -o xsltutils.o > /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c -o pattern.lo pattern.c > xsltutils.c: In function ‘xsltSaveResultTo’: > xsltutils.c:1541: warning: passing argument 1 of ‘xmlBufferWriteQuotedString’ from incompatible pointer type > xsltutils.c:1554: warning: passing argument 1 of ‘xmlBufferWriteQuotedString’ from incompatible pointer type > xsltutils.c: In function ‘xsltSaveResultToString’: > xsltutils.c:1758: error: dereferencing pointer to incomplete type > xsltutils.c:1759: error: dereferencing pointer to incomplete type > xsltutils.c:1761: error: dereferencing pointer to incomplete type > xsltutils.c:1762: error: dereferencing pointer to incomplete type > libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c pattern.c -o pattern.o > make[2]: *** [xsltutils.lo] Error 1 > make[2]: *** Waiting for unfinished jobs.... > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 Yes, that's exactly it. It's failing to build libxslt. Looking further up now, I see that it has downloaded libxml2 2.9.0-rc1. That version is known to be incompatible with the latest libxslt release. You can make it work by setting this environment variable: LIBXML2_VERSION=2.8.0 Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8
by 7stud 13 Aug '12

13 Aug '12

-----Original Message----- From: "Stefan Behnel" [stefan_ml(a)behnel.de] Date: 08/13/2012 09:24 AM To: "lxml mailing list" <lxml(a)lxml.de> Subject: Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8 7stud, 13.08.2012 10:02: > $ tar xfvz lxml-2.3.5.tar.gz > > $ cd lxml-2.3.5 > > $ p3 setup.py build --static-deps > Building lxml version 2.3.5. > Latest version of libiconv is 1.14 > Downloading libiconv into libs/libiconv-1.14.tar.gz > Unpacking libiconv-1.14.tar.gz into build/tmp > Latest version of libxml2 is 2.9.0-rc1 > Downloading libxml2 into libs/libxml2-2.9.0-rc1.tar.gz > Unpacking libxml2-2.9.0-rc1.tar.gz into build/tmp > Latest version of libxslt is 1.1.26 > Downloading libxslt into libs/libxslt-1.1.26.tar.gz > Unpacking libxslt-1.1.26.tar.gz into build/tmp > Starting build in build/tmp/libiconv-1.14 > … > … > … Could you provide the error message please? It should have turned up right here. > make[2]: *** [xsltutils.lo] Error 1 > make[2]: *** Waiting for unfinished jobs.... > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 ==== Is this what you are after? libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c xslt.c -o xslt.o /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c -o xsltutils.lo xsltutils.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c xsltutils.c -o xsltutils.o /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c -o pattern.lo pattern.c xsltutils.c: In function ‘xsltSaveResultTo’: xsltutils.c:1541: warning: passing argument 1 of ‘xmlBufferWriteQuotedString’ from incompatible pointer type xsltutils.c:1554: warning: passing argument 1 of ‘xmlBufferWriteQuotedString’ from incompatible pointer type xsltutils.c: In function ‘xsltSaveResultToString’: xsltutils.c:1758: error: dereferencing pointer to incomplete type xsltutils.c:1759: error: dereferencing pointer to incomplete type xsltutils.c:1761: error: dereferencing pointer to incomplete type xsltutils.c:1762: error: dereferencing pointer to incomplete type libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I.. -I../libxslt -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include/libxml2 -I/Users/me/Downloads/lxml-2.3.5/build/tmp/libxml2/include -g -O2 -Wall -c pattern.c -o pattern.o make[2]: *** [xsltutils.lo] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2

2 1

Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8
by 7stud 13 Aug '12

13 Aug '12

-----Original Message----- From: "Stefan Behnel" [stefan_ml(a)behnel.de] Date: 08/13/2012 09:24 AM To: "lxml mailing list" <lxml(a)lxml.de> Subject: Re: [lxml] Error installing lxml 2.3.5 on OS X 10.6.8 7stud, 13.08.2012 10:02: > $ tar xfvz lxml-2.3.5.tar.gz > > $ cd lxml-2.3.5 > > $ p3 setup.py build --static-deps > Building lxml version 2.3.5. > Latest version of libiconv is 1.14 > Downloading libiconv into libs/libiconv-1.14.tar.gz > Unpacking libiconv-1.14.tar.gz into build/tmp > Latest version of libxml2 is 2.9.0-rc1 > Downloading libxml2 into libs/libxml2-2.9.0-rc1.tar.gz > Unpacking libxml2-2.9.0-rc1.tar.gz into build/tmp > Latest version of libxslt is 1.1.26 > Downloading libxslt into libs/libxslt-1.1.26.tar.gz > Unpacking libxslt-1.1.26.tar.gz into build/tmp > Starting build in build/tmp/libiconv-1.14 > … > … > … Could you provide the error message please? It should have turned up right here. > make[2]: *** [xsltutils.lo] Error 1 > make[2]: *** Waiting for unfinished jobs.... > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 > Traceback (most recent call last): > File "setup.py", line 233, in <module> > **setup_extra_options() > File "setup.py", line 145, in setup_extra_options > STATIC_CFLAGS, STATIC_BINARIES) > File "/Users/me/Downloads/lxml-2.3.5/setupinfo.py", line 56, in ext_modules > multicore=OPTION_MULTICORE) > File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 335, in build_libxml2xslt > cmmi(libxslt_configure_cmd, libxslt_dir, multicore, **call_setup) > File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 266, in cmmi > cwd=build_dir, **call_setup) > File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 249, in call_subprocess > raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode)) > Exception: Command "make -j3" returned code 512 > $ ==== The complete error message is in the text I included after the lines: ... ... ... The lines you indicated that you want all start with: "checking whether .... _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

Error installing lxml 2.3.5 on OS X 10.6.8
by 7stud 13 Aug '12

13 Aug '12

$ tar xfvz lxml-2.3.5.tar.gz $ cd lxml-2.3.5 $ p3 setup.py build --static-deps Building lxml version 2.3.5. Latest version of libiconv is 1.14 Downloading libiconv into libs/libiconv-1.14.tar.gz Unpacking libiconv-1.14.tar.gz into build/tmp Latest version of libxml2 is 2.9.0-rc1 Downloading libxml2 into libs/libxml2-2.9.0-rc1.tar.gz Unpacking libxml2-2.9.0-rc1.tar.gz into build/tmp Latest version of libxslt is 1.1.26 Downloading libxslt into libs/libxslt-1.1.26.tar.gz Unpacking libxslt-1.1.26.tar.gz into build/tmp Starting build in build/tmp/libiconv-1.14 … … … make[2]: *** [xsltutils.lo] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 Traceback (most recent call last): File "setup.py", line 233, in <module> **setup_extra_options() File "setup.py", line 145, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "/Users/me/Downloads/lxml-2.3.5/setupinfo.py", line 56, in ext_modules multicore=OPTION_MULTICORE) File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 335, in build_libxml2xslt cmmi(libxslt_configure_cmd, libxslt_dir, multicore, **call_setup) File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 266, in cmmi cwd=build_dir, **call_setup) File "/Users/me/Downloads/lxml-2.3.5/buildlibxml.py", line 249, in call_subprocess raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode)) Exception: Command "make -j3" returned code 512 $ p3 is an alias for python 3.2.3, which I installed using python-3.2.3-macosx10.6.dmg. My Xcode version is 3.2.2. OS X 10.6.8

2 1

SelectorSyntaxError elements with name E in any namespace *|E
by Jack Bates 11 Aug '12

11 Aug '12

Hi, I love lxml, thanks for the hard work I just tried "sel = CSSSelector('*|Content')" with the latest lxml release (2.3.5) and got the exception at the bottom of this message I checked the CSS specification and it discusses selectors of this format [1][2] > ns|E > elements with name E in namespace ns > *|E > elements with name E in any namespace, including those without a namespace > |E > elements with name E without a namespace > E > if no default namespace has been declared for selectors, this is equivalent to *|E. Otherwise it is equivalent to ns|E where ns is the default namespace. I also checked the lxml bug tracker but didn't find this issue of elements in any namespace mentioned Is this something that could be supported by lxml? Should I open an issue in the bug tracker for it? Also, I found this thread on the mailing list which discusses support for elements in any namespace [3] and proposes a patch [1] http://www.w3.org/TR/css3-selectors/#typenmsp [2] http://www.w3.org/TR/css3-namespace/#css-qnames [3] http://thread.gmane.org/gmane.comp.python.lxml.devel/4937 > Traceback (most recent call last): > File "example.py", line 35, in <module> > sel = CSSSelector('*|Content') > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 51, in __init__ > path = css_to_xpath(css) > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 537, in css_to_xpath > css_expr = parse(css_expr) > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 662, in parse > return parse_selector_group(stream) > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 677, in parse_selector_group > result.append(parse_selector(stream)) > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 688, in parse_selector > result = parse_simple_selector(stream) > File "/home/nottheoilrig/lxml-2.3.5/src/lxml/cssselect.py", line 724, in parse_simple_selector > "Expected symbol, got '%s'" % next) > lxml.cssselect.SelectorSyntaxError: Expected symbol, got '*' at [Token(u'*', 0), Token(u'|', 1), Symbol(u'Content', 2)] -> None

3 21

Re : translating CSS selectors to Python code
by Simon Sapin 06 Aug '12

06 Aug '12

Sorry, I meant in 3.0. ----- Reply message ----- De : "Stefan Behnel" <stefan_ml(a)behnel.de> Pour : <lxml(a)lxml.de> Objet : [lxml] translating CSS selectors to Python code Date : lun., août 6, 2012 19:04 Simon Sapin, 06.08.2012 19:02: > Le 06/08/2012 18:54, Stefan Behnel a écrit : >> The code is in lxml.etree.pyx class >> _MultiTagMatcher, and apihelpers.pxi function _mapTagsToQnameMatchArray(), >> in case someone wants to give it a try. > > If the patch is done soonish, can I expect to have it in lxml? Sure - depending on the patch. ;) Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

Escaped Non-ASCII characters in attribute values
by Butler, John (NIH/NLM) [E] 04 Aug '12

04 Aug '12

The changelog for lxml 2.1beta3 states that "Non-ASCII characters in attribute values are no longer escaped on serialisation." I am using a product that is also making use of the C libraries libxml2 and libxslt. The XML being produced has all the Non-ASCII characters in attribute values escaped even though they were initially valid UTF-8 encoded characters. The vendor has informed me that there is nothing they can do about this as the escaping is being done in the libxslt C library. Can someone on the lxml development team describe to me how lxml was able to resolve this issue? Was this done by post processing strings produced by libxslt or is there some way to get libxslt not to escape Non-ASCII characters in attribute values? Thanks, John Butler

2 1

getiterator vs xpath question
by Tim Arnold 31 Jul '12

31 Jul '12

I am misunderstanding the difference between these two code blocks. I thought they would have the same result. I want to find every element in the tree that has an 'id' or a 'name' attribute so I can store that attribute. (I'm link-checking a static html site). starting out: >>> from lxml import etree >>> parser = etree.HTMLParser() >>> tree = etree.parse('ugdet17.htm', parser=parser) Then using getiterator(), >>> for elem in tree.getiterator(): ... if elem.tag == 'div': ... if elem.get('class') == 'section': ... print elem.attrib ... {'class': 'section', 'id': 'ugseldet'} And I *thought* the xpath would return the same thing. >>> for div in tree.xpath('//div[@class="section"]'): ... print div.attrib ... {'class': 'section', 'id': 'ugseldet'} {'class': 'section', 'id': 'ugstep'} So how do I go through the tree and get each id or name attribute value? thanks, --Tim Arnold

2 1