Mailman 3 May 2013 - lxml - The Python XML Toolkit

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

results[0] is type str which, like type list, causes append() to cause an exception further up the stack. The contents of results[0] is the string analogous to the XPath text() function of the query results. In other words, it's a flattened representation of the result tree. On 05/06/2013 06:36 PM, Simon Sapin wrote: > Le 07/05/2013 00:19, Eric Levy a écrit : >> I want to append the first item in the list. I tried passing string[0] >> to apply_templates(), and then passing results[0] to append(), but >> regardless of whether I am passing the entire list or only the first >> element, the stack trace looks the same, with an exception coming from >> lxml.etree.c. > > And the message is still "TypeError: invalid argument type <type > 'list'>"? What does results[0] and type(results[0]) look like? (Try > using pdb or adding print statements.) >

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

I want to append the first item in the list. I tried passing string[0] to apply_templates(), and then passing results[0] to append(), but regardless of whether I am passing the entire list or only the first element, the stack trace looks the same, with an exception coming from lxml.etree.c. On 05/06/2013 04:55 PM, Eric P. Levy wrote: > As stated, I want to append the first item, but I'm not seeing how your suggestion differs from what I have. > > Simon Sapin <simon.sapin(a)exyr.org> wrote: > > Le 06/05/2013 22:07, Eric Levy a écrit : >> Hopefully this will help. Note the callback occurs about halfway up the >> stack. >> >> Traceback (most recent call last): >> File "index.py", line 107, in <module> >> rslt_tree = trans(in_tree, mobile="'" + mobile_str + "'") >> File "xslt.pxi", line 582, in lxml.etree.XSLT.__call__ >> (src/lxml/lxml.etree.c:139795) >> File "lxml.etree.pyx", line 294, in >> lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9359) >> File "xsltext.pxi", line 195, in lxml.etree._callExtensionElement >> (src/lxml/lxml.etree.c:144738) >> File "index.py", line 100, in execute >> output_parent.append(results) >> File "readonlytree.pxi", line 475, in >> lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69888) >> File "readonlytree.pxi", line 481, in >> lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69807) >> File "readonlytree.pxi", line 534, in lxml.etree._roNodeOf >> (src/lxml/lxml.etree.c:70448) >> TypeError: invalid argument type <type 'list'> > > It seems this is the same issue as before: apply_templates() returns a > list of results; element objects behave kind of like a list of their > children and the .append() method expects a child element object rather > than a list. > > Try one of these, depending on what you mean: > > output_parent.append(results[0]) > output_parent.extend(results) > > (append takes one item to add, the list grows by one; extend takes an > iterable of items, the list grows by the length of the iterable.) >

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

Hopefully this will help. Note the callback occurs about halfway up the stack. Traceback (most recent call last): File "index.py", line 107, in <module> rslt_tree = trans(in_tree, mobile="'" + mobile_str + "'") File "xslt.pxi", line 582, in lxml.etree.XSLT.__call__ (src/lxml/lxml.etree.c:139795) File "lxml.etree.pyx", line 294, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9359) File "xsltext.pxi", line 195, in lxml.etree._callExtensionElement (src/lxml/lxml.etree.c:144738) File "index.py", line 100, in execute output_parent.append(results) File "readonlytree.pxi", line 475, in lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69888) File "readonlytree.pxi", line 481, in lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69807) File "readonlytree.pxi", line 534, in lxml.etree._roNodeOf (src/lxml/lxml.etree.c:70448) TypeError: invalid argument type <type 'list'> On 05/06/2013 04:01 PM, Simon Sapin wrote: > Le 06/05/2013 21:51, Eric Levy a écrit : >> I may have put the issue in a confusing way, because I used the term >> "string", but I am actually trying to capture a an entire results tree >> (or the first item if there is a list of results), and if possible, >> recursively apply XSLT templates to those results. Consequently, I >> cannot use the XPath text() function because this flattens the results, >> rather than keeping them as a tree. I should also note that in the line >> reading "results = self.apply_templates(context, string)", I tried >> changing "string" to "string[0]", but the effect was the same. >> >> I hope this makes clearer what I am trying to do. > > Ok, so forget the part about strings and text. Can you copy the full > traceback you get with string[0]? > > Cheers,

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

I may have put the issue in a confusing way, because I used the term "string", but I am actually trying to capture a an entire results tree (or the first item if there is a list of results), and if possible, recursively apply XSLT templates to those results. Consequently, I cannot use the XPath text() function because this flattens the results, rather than keeping them as a tree. I should also note that in the line reading "results = self.apply_templates(context, string)", I tried changing "string" to "string[0]", but the effect was the same. I hope this makes clearer what I am trying to do. On 05/06/2013 03:37 PM, Simon Sapin wrote: > Le 06/05/2013 18:41, Eric Levy a écrit : >> Currently, the apply_templates call is failing, with the following error: >>> TypeError: invalid argument type <type 'list'> > > Hi, > > That seems pretty self-explanatory. XPath returns a list of results, and > according to the doc[1] example apply_templates() expects a single > element object. > > [1] http://lxml.de/extensions.html > > If your XPath expression is constructed such that there is exactly one > result, use `foo[0]` to extract it from the list. If you use `text()` in > XPath or something such that results are strings rather than element > objects, you might need to construct a dummy object to pass to > apply_templates(): > > el = lxml.etree.Element('dummy') > el.text = xpath_results[0] >

2 1

bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

Hello, I am experience an issue writing data python data that had been generated by an XPath query. It appears that lxml uses some pythonic magic to transform between Python objects and XML data. It also appears, unfortunately that the same data that was generated by the library cannot be written back to XML. Specifically, I am seeking to develop a simple web page that is based on lxml applying XSLT templates. Human readable text, as well as high-level formatting information, is kept in XML documents. XSLT templates transform the documents to HTML pages. Because internationalization is necessary, text is kept separate documents. The particular document used depends on the locale, which is determined at run time by the Python application. To satisfy these requirements, I have attempted to use the XSLT extensions feature of lxml. Before processing begins, I parse the XML document containing human-readable text. I then perform the transformation on the main XML document, passing an extension callback that is meant to read one of the attribute values of the extension XSLT element, and use the value to generate an XPath query. The goal for the extension element to evaluate to the result tree of the XPath query on the auxiliary document. Additionally, markup in the results tree from the query should be processed according to the template rules. To illustrate, consider this snippet: in_tree = etree.parse("document.xml") tpl_tree = etree.parse("template.xslt") strings_tree = etree.parse("aux_doc.xml") class StringsElement(XSLTExtension): def execute(self, context, self_node, input_node, output_parent): id_ = self_node.attrib["id"] string = strings_tree.xpath(generate_xpath_query(id_)) results = self.apply_templates(context, string) output_parent.append(results) xslt_exts = { ( "mynamespace", "string" ) : StringElement() } template = etree.XSLT(tpl_tree, extensions = xslt_exts) rslt_tree = trans(in_tree) rslt_tree.write(sys.stdout) Currently, the apply_templates call is failing, with the following error: TypeError: invalid argument type <type 'list'> I have been struggling with this issue and appreciate your help. Sincerely, Eric Levy

2 2

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) May 3, 2013

May 3, 2013

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Namespaced attributes losing their namespace
by Tyler Spivey May 1, 2013

May 1, 2013

In the below test script, I'm not getting the result I expect. Is this a bug in lxml? If not, what am I doing wrong? I'm using lxml 3.1.1 on win32, downloaded from PyPi. What I get: <root xmlns="url1"> <test attrib="value"/> </root> I expect attrib to not lose its namespace, as shown when I re-parse the output: >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'attrib': 'value'} If I swap the append and the attribute setting, things work: <test xmlns:ns0="url1" ns0:attrib="value"/> >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'{url1}attrib': 'value'} If this is intended behaviour, I'll have to see how to fit that into my design, since my functions return elements ready for appending. from lxml import etree nsmap = {None: 'url1'} root = etree.Element('{url1}root', nsmap=nsmap) test = etree.Element('{url1}test') test.attrib['{url1}attrib'] = 'value' root.append(test) print etree.tostring(root, encoding='utf-8', pretty_print=True)

2 6