lxml - The Python XML Toolkit

Download

lxml@python.org

May 2013

17 participants
27 discussions

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Namespaced attributes losing their namespace
by Tyler Spivey 01 May '13

01 May '13

In the below test script, I'm not getting the result I expect. Is this a bug in lxml? If not, what am I doing wrong? I'm using lxml 3.1.1 on win32, downloaded from PyPi. What I get: <root xmlns="url1"> <test attrib="value"/> </root> I expect attrib to not lose its namespace, as shown when I re-parse the output: >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'attrib': 'value'} If I swap the append and the attribute setting, things work: <test xmlns:ns0="url1" ns0:attrib="value"/> >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'{url1}attrib': 'value'} If this is intended behaviour, I'll have to see how to fit that into my design, since my functions return elements ready for appending. from lxml import etree nsmap = {None: 'url1'} root = etree.Element('{url1}root', nsmap=nsmap) test = etree.Element('{url1}test') test.attrib['{url1}attrib'] = 'value' root.append(test) print etree.tostring(root, encoding='utf-8', pretty_print=True)

2 6