Mailman 3 September 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 11, 2023

April 11, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 10, 2023

April 10, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) May 3, 2013

May 3, 2013

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Apache2 crashes
by Brieuc SCHAFF Feb. 12, 2013

Feb. 12, 2013

Hi, We are using lxml in several of our applications with Python 2.6 and from time to time, the application stops responding after a segmentation fault error, and this kind of backtrace: *Jul 1 15:35:01 server4 httpd: *** glibc detected *** /usr/sbin/apache2: malloc(): smallbin double linked list corrupted: 0x00007f583a03f500 **** *Jul 1 15:35:01 server4 httpd: ======= Backtrace: =========* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f58376c2806]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x7bb39)[0x7f58376c6b39]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(__libc_malloc+0x6e)[0x7f58376c77de]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlInitParserCtxt+0x317)[0x7f58343f2887]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlNewParserCtxt+0x2c)[0x7f58343f2acc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x54eda)[0x7f5818db6eda]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xddcf3)[0x7f5818e3fcf3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xab5ee)[0x7f5818e0d5ee]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb7d8e)[0x7f5818e19d8e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb827e)[0x7f5818e1a27e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f582face030]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5a98)[0x7f582face928]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f582facc96f]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f582fa39cef]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f582fa8e6dc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f582fac8193] * *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f582fe9bfdb]* *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f582fea0eed]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f5837c14eb3]* *Jul 1 15:35:01 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f58379d49ca]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f5837731cdd]* ----------------- *Jul 11 10:44:48 server4 httpd: *** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007f3968b8cb68 **** *Jul 11 10:44:48 server4 httpd: ======= Backtrace: =========* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f39789dd806]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlResetError+0x22)[0x7f39755038e2]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlRaiseError+0x237)[0x7f3975504e07]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlSimpleError+0x6b)[0x7f397550543b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputBufferGrow+0x16b)[0x7f397553271b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputGrow+0x4c)[0x7f397550739c]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x3bd52)[0x7f397550bd52]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParseDocument+0x4b8)[0x7f39755204c8]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x505c5)[0x7f39755205c5]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x8fab0)[0x7f3958ad8ab0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2daa0)[0x7f3958a76aa0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2ddc9)[0x7f3958a76dc9]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc26f0)[0x7f3958b0b6f0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc431e)[0x7f3958b0d31e]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f397063b030]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f397063996f]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f39705a6cef]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f39705fb6dc]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f3970635193] * *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f3970a08fdb]* *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f3970a0deed]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f3978f2feb3]* *Jul 11 10:44:48 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f3978cef9ca]* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f3978a4ccdd]* Stopping apache and restarting it fix the issue, but we have no idea why this is happening. We use Unbuntu 10.10's package of lxml, which is version 2.2.4. *>>> print("%-20s: %s" % ('Python', sys.version_info))* *Python : (2, 6, 5, 'final', 0)* *>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))* *lxml.etree : (2, 2, 4, 0)* *>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))* *libxml used : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))* *libxml compiled : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))* *libxslt used : (1, 1, 26)* *>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))* *libxslt compiled : (1, 1, 26)* * * Do you have any idea of what could cause these issues ? I wonder if upgrading to a newer version of the library could change anything. Thanks in advance for any answer you could provide. Regards, B. _____________________ Brieuc SCHAFF +33609898260 _____________________

2 3

ValueError on python3.3
by Andriy Kornatskyy Sept. 30, 2012

Sept. 30, 2012

Here is an error with python3.3 >>> from lxml.etree import HTMLParser Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 48, in init lxml.etree (src/lxml/lxml.etree.c:159821) ValueError: level must be >= 0 Is that something known? Please advise. Thanks. Andriy Kornatskyy

3 6

Re: [lxml] Installing with pip seems to always choose version 3.0.alpha2
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Tal Einat, 29.09.2012 16:25: > On Tue, Sep 25, 2012 at 6:16 PM, Stefan Behnel wrote: >> Anyway, I deleted the version from PyPI now, so it can't be downloaded >> anymore. > > Thanks for the fix! > > It might indeed be a pip issue, but note that (before the fix) I was > telling pip to install specifically lower versions, e.g.: > > pip install lxml==2.3 > pip install lxml==2.3.5 > pip install "lxml<3" > > In all of these cases, pip installed 3.0.alpha2. I simply couldn't get it > to install any other version. > > Should I file a bug report for pip in this regard? Please do. At least bring it up on their mailing list so that they can comment on it. Stefan

1 0

lxml 2.3.6 released
by Stefan Behnel Sept. 28, 2012

Sept. 28, 2012

Hi, I just released lxml 2.3.6. It's a pure bug fix release that closes one bug that was already fixed in 3.0 alpha 2. http://pypi.python.org/pypi/lxml/2.3.6 Bugs fixed * Passing long Unicode strings into the feed() parser interface failed to read the entire string. This release was built with Cython 0.15.1. Have fun, Stefan

1 0