Mailman 3 December 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 11, 2023

April 11, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 10, 2023

April 10, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) May 3, 2013

May 3, 2013

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Apache2 crashes
by Brieuc SCHAFF Feb. 12, 2013

Feb. 12, 2013

Hi, We are using lxml in several of our applications with Python 2.6 and from time to time, the application stops responding after a segmentation fault error, and this kind of backtrace: *Jul 1 15:35:01 server4 httpd: *** glibc detected *** /usr/sbin/apache2: malloc(): smallbin double linked list corrupted: 0x00007f583a03f500 **** *Jul 1 15:35:01 server4 httpd: ======= Backtrace: =========* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f58376c2806]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x7bb39)[0x7f58376c6b39]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(__libc_malloc+0x6e)[0x7f58376c77de]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlInitParserCtxt+0x317)[0x7f58343f2887]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlNewParserCtxt+0x2c)[0x7f58343f2acc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x54eda)[0x7f5818db6eda]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xddcf3)[0x7f5818e3fcf3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xab5ee)[0x7f5818e0d5ee]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb7d8e)[0x7f5818e19d8e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb827e)[0x7f5818e1a27e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f582face030]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5a98)[0x7f582face928]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f582facc96f]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f582fa39cef]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f582fa8e6dc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f582fac8193] * *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f582fe9bfdb]* *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f582fea0eed]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f5837c14eb3]* *Jul 1 15:35:01 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f58379d49ca]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f5837731cdd]* ----------------- *Jul 11 10:44:48 server4 httpd: *** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007f3968b8cb68 **** *Jul 11 10:44:48 server4 httpd: ======= Backtrace: =========* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f39789dd806]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlResetError+0x22)[0x7f39755038e2]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlRaiseError+0x237)[0x7f3975504e07]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlSimpleError+0x6b)[0x7f397550543b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputBufferGrow+0x16b)[0x7f397553271b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputGrow+0x4c)[0x7f397550739c]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x3bd52)[0x7f397550bd52]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParseDocument+0x4b8)[0x7f39755204c8]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x505c5)[0x7f39755205c5]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x8fab0)[0x7f3958ad8ab0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2daa0)[0x7f3958a76aa0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2ddc9)[0x7f3958a76dc9]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc26f0)[0x7f3958b0b6f0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc431e)[0x7f3958b0d31e]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f397063b030]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f397063996f]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f39705a6cef]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f39705fb6dc]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f3970635193] * *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f3970a08fdb]* *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f3970a0deed]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f3978f2feb3]* *Jul 11 10:44:48 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f3978cef9ca]* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f3978a4ccdd]* Stopping apache and restarting it fix the issue, but we have no idea why this is happening. We use Unbuntu 10.10's package of lxml, which is version 2.2.4. *>>> print("%-20s: %s" % ('Python', sys.version_info))* *Python : (2, 6, 5, 'final', 0)* *>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))* *lxml.etree : (2, 2, 4, 0)* *>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))* *libxml used : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))* *libxml compiled : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))* *libxslt used : (1, 1, 26)* *>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))* *libxslt compiled : (1, 1, 26)* * * Do you have any idea of what could cause these issues ? I wonder if upgrading to a newer version of the library could change anything. Thanks in advance for any answer you could provide. Regards, B. _____________________ Brieuc SCHAFF +33609898260 _____________________

2 3

Unicode decoding is not supported on this platform with python3.3 debug
by Mark Grandi Dec. 28, 2012

Dec. 28, 2012

Im trying to debug a python problem and I built lxml using the pydebug compiled version of python3.3, but the XMLParser.feed() method isn't working... Corvidae:Python-3.3.0 markgrandi$ ./python.exe Python 3.3.0 (default, Dec 9 2012, 14:01:13) [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> for line in x: ... print(x) [126070 refs] >>> for line in x: ... parser.feed(x.readline()) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> File "parser.pxi", line 1105, in lxml.etree._FeedParser.feed (src/lxml/lxml.etree.c:87183) lxml.etree.ParserError: Unicode parsing is not supported on this platform [126026 refs] any idea on what is causing this and how i can fix it? ~mark

2 5

lxml 3.1 beta1 released
by Stefan Behnel Dec. 21, 2012

Dec. 21, 2012

Hi everyone, here's a first beta release of the upcoming lxml 3.1. The main new feature is a simple API for incremental XML generation. Special thanks goes to Simon Sapin for testing and improving it. More feedback is appreciated. You can get the release from here: http://pypi.python.org/pypi/lxml/3.1beta1 http://lxml.de/files/lxml-3.1beta1.tgz The corresponding website is here: http://lxml.de/3.1/ This release was built using Cython 0.17.3. Have fun, Stefan 3.1beta1 (2012-12-21) ===================== Features added -------------- * New build-time option ``--with-unicode-strings`` for Python 2 that makes the API always return Unicode strings for names and text instead of byte strings for plain ASCII content. * New incremental XML file writing API ``etree.xmlfile()``. * E factory in lxml.objectify is callable to simplify the creation of tags with non-identifier names without having to resort to getattr(). Bugs fixed ---------- * When starting from a non-namespaced element in lxml.objectify, searching for a child without explicitly specifying a namespace incorrectly found namespaced elements with the requested local name, instead of restricting the search to non-namespaced children. * GH#85: Deprecation warnings were fixed for Python 3.x. * GH#33: lxml.html.fromstring() failed to accept bytes input in Py3. * LP#1080792: Static build of libxml2 2.9.0 failed due to missing file. Other changes ------------- * The externally useless class ``_ObjectifyElementMakerCaller`` was removed from the module API of lxml.objectify. * LP#1075622: lxml.builder is faster for adding text to elements with many children. Patch by Anders Hammarquist.

1 0

[objectify] namespace handling in child element attribute access
by Holger Joukl Dec. 19, 2012

Dec. 19, 2012

Hi, just noticed that namespace handling in child element access does not work correctly (any more) if the parent element is in no-namespace: >>> from lxml import etree, objectify >>> print "etree.__version__:", etree.__version__ etree.__version__: 2.3.5 >>> print etree.LIBXML_VERSION (2, 7, 8) >>> objectify.enable_recursive_str() >>> root = objectify.fromstring(""" ... <root> ... <foo:x xmlns:foo="/foo/bar">1</foo:x> ... <x>2</x> ... </root>""") >>> print root root = None [ObjectifiedElement] {/foo/bar}x = 1 [IntElement] x = 2 [IntElement] >>> print "root.x:", root.x root.x: 1 >>> assert root.x == 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError >>> As root lives in no-namespace root.x should return the second, no-namespace x child element. This worked correctly up to ~2.0 (based on libxml 2.6.27) and stopped working for ~2.1.5 (libxml 2.6.32), judging from the versions I have tried. Funny that we never noticed it up to now. Not too much insane namespace mixing going on here, obviously ;-) >>> from lxml import etree, objectify >>> print "etree.__version__:", etree.__version__ etree.__version__: 2.0.alpha4 >>> print etree.LIBXML_VERSION (2, 6, 27) >>> objectify.enableRecursiveStr() >>> root = objectify.fromstring(""" ... <root> ... <foo:x xmlns:foo="/foo/bar">1</foo:x> ... <x>2</x> ... </root>""") >>> print root root = None [ObjectifiedElement] {/foo/bar}x = 1 [IntElement] x = 2 [IntElement] >>> print "root.x:", root.x root.x: 2 >>> assert root.x == 2 >>> Btw this works correctly if the parent element is in a namespace, which is the case the documentation shows (and doctests). Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart

2 2