Mailman 3 July 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Apache2 crashes
by Brieuc SCHAFF 12 Feb '13

12 Feb '13

Hi, We are using lxml in several of our applications with Python 2.6 and from time to time, the application stops responding after a segmentation fault error, and this kind of backtrace: *Jul 1 15:35:01 server4 httpd: *** glibc detected *** /usr/sbin/apache2: malloc(): smallbin double linked list corrupted: 0x00007f583a03f500 **** *Jul 1 15:35:01 server4 httpd: ======= Backtrace: =========* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f58376c2806]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x7bb39)[0x7f58376c6b39]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(__libc_malloc+0x6e)[0x7f58376c77de]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlInitParserCtxt+0x317)[0x7f58343f2887]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlNewParserCtxt+0x2c)[0x7f58343f2acc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x54eda)[0x7f5818db6eda]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xddcf3)[0x7f5818e3fcf3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xab5ee)[0x7f5818e0d5ee]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb7d8e)[0x7f5818e19d8e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb827e)[0x7f5818e1a27e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f582face030]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5a98)[0x7f582face928]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f582facc96f]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f582fa39cef]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f582fa8e6dc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f582fac8193] * *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f582fe9bfdb]* *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f582fea0eed]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f5837c14eb3]* *Jul 1 15:35:01 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f58379d49ca]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f5837731cdd]* ----------------- *Jul 11 10:44:48 server4 httpd: *** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007f3968b8cb68 **** *Jul 11 10:44:48 server4 httpd: ======= Backtrace: =========* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f39789dd806]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlResetError+0x22)[0x7f39755038e2]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlRaiseError+0x237)[0x7f3975504e07]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlSimpleError+0x6b)[0x7f397550543b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputBufferGrow+0x16b)[0x7f397553271b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputGrow+0x4c)[0x7f397550739c]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x3bd52)[0x7f397550bd52]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParseDocument+0x4b8)[0x7f39755204c8]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x505c5)[0x7f39755205c5]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x8fab0)[0x7f3958ad8ab0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2daa0)[0x7f3958a76aa0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2ddc9)[0x7f3958a76dc9]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc26f0)[0x7f3958b0b6f0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc431e)[0x7f3958b0d31e]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f397063b030]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f397063996f]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f39705a6cef]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f39705fb6dc]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f3970635193] * *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f3970a08fdb]* *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f3970a0deed]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f3978f2feb3]* *Jul 11 10:44:48 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f3978cef9ca]* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f3978a4ccdd]* Stopping apache and restarting it fix the issue, but we have no idea why this is happening. We use Unbuntu 10.10's package of lxml, which is version 2.2.4. *>>> print("%-20s: %s" % ('Python', sys.version_info))* *Python : (2, 6, 5, 'final', 0)* *>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))* *lxml.etree : (2, 2, 4, 0)* *>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))* *libxml used : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))* *libxml compiled : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))* *libxslt used : (1, 1, 26)* *>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))* *libxslt compiled : (1, 1, 26)* * * Do you have any idea of what could cause these issues ? I wonder if upgrading to a newer version of the library could change anything. Thanks in advance for any answer you could provide. Regards, B. _____________________ Brieuc SCHAFF +33609898260 _____________________

2 3

Memory debugging
by Christian Heimes 19 Sep '12

19 Sep '12

Hello, I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code. In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case. It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too. Regards, Christian Example patch: diff -ru lxml-2.3.4.orig/src/lxml/lxml.etree.pyx lxml-2.3.4/src/lxml/lxml.etree.pyx --- lxml-2.3.4.orig/src/lxml/lxml.etree.pyx 2012-03-26 12:28:58.000000000 +0200 +++ lxml-2.3.4/src/lxml/lxml.etree.pyx 2012-05-03 18:33:00.702130662 +0200 @@ -2956,6 +2956,11 @@ except _TargetParserResult, result_container: return result_container.result +def memoryUsed(): + return tree.xmlMemUsed() + +def memoryBlocks(): + return tree.xmlMemBlocks() ################################################################################ # Include submodules diff -ru lxml-2.3.4.orig/src/lxml/tree.pxd lxml-2.3.4/src/lxml/tree.pxd --- lxml-2.3.4.orig/src/lxml/tree.pxd 2011-09-25 18:58:05.000000000 +0200 +++ lxml-2.3.4/src/lxml/tree.pxd 2012-05-03 17:44:50.323798041 +0200 @@ -322,6 +322,7 @@ cdef extern from "libxml/xmlmemory.h": cdef void* xmlMalloc(size_t size) nogil cdef int xmlMemBlocks() nogil + cdef int xmlMemUsed() nogil cdef extern from "etree_defs.h": cdef bint _isElement(xmlNode* node) nogil [1] http://xmlsoft.org/html/libxml-xmlmemory.html#DEBUG_MEMORY [2] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemBlocks [3] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemUsed [4] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemDisplay

2 2

[lxml-dev] confusing xpath performance characteristics
by jholg＠gmx.de 24 Aug '12

24 Aug '12

Hi, I ran into some performance characteristics of lxml/libxml2 xpath that I find rather confusing: I try to find the @type attribute of a certain element in an XML Schema (which contains lots of complexType definitions with lots of elements in them; unfortunately I can't post the schema): >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.095885038375854492, 0.096823930740356445, 0.096174955368041992] So I think I'm being smart and give a little more path information - reckoning that this should *improve* performance: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.1770780086517334, 0.1775970458984375, 0.17748594284057617] Hm. Performance degrades slightly. I'm adding even more of the path to where my desired elements live in the schema: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [103.79744100570679, 103.83671712875366, 103.61817717552185] What??? >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('/ae/data/pydev/hjoukl/NDM/SVN_CO/TRUNK/ndm/reference/xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType/*/xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.044407129287719727, 0.044126987457275391, 0.044229030609130859] >>> Ok, this version's better than my naive approach, which seems logical to me. But why would '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' perform drastically slower than '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' ? libxml2 problem? Running the same xpaths in Oxygen I don't notice performance differences (can't profile this). Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 5

getiterator vs xpath question
by Tim Arnold 01 Aug '12

01 Aug '12

I am misunderstanding the difference between these two code blocks. I thought they would have the same result. I want to find every element in the tree that has an 'id' or a 'name' attribute so I can store that attribute. (I'm link-checking a static html site). starting out: >>> from lxml import etree >>> parser = etree.HTMLParser() >>> tree = etree.parse('ugdet17.htm', parser=parser) Then using getiterator(), >>> for elem in tree.getiterator(): ... if elem.tag == 'div': ... if elem.get('class') == 'section': ... print elem.attrib ... {'class': 'section', 'id': 'ugseldet'} And I *thought* the xpath would return the same thing. >>> for div in tree.xpath('//div[@class="section"]'): ... print div.attrib ... {'class': 'section', 'id': 'ugseldet'} {'class': 'section', 'id': 'ugstep'} So how do I go through the tree and get each id or name attribute value? thanks, --Tim Arnold

2 1