Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] Remarks on implementing iterparse()
by Stefan Behnel 22 Jun '06

22 Jun '06

Hi all, since I won't have the time to implement iterparse() any time soon, here's a proposal on how it should be implemented, in case someone wants to take a shot at it. "iterparse" will be (or will return) an iterable object, let's call it IterParse for clarity. A class is basically the only way of implementing iterators in Pyrex. For the internal SAX part, IterParse will likely work a lot like lxml.sax.ElementTreeContentHandler. We'd need a custom wrapper to the default libxml2 SAX handler to intercept the parse events (this means implementing C helper functions for the SAX events) /after/ they were processed by libxml2. See xmlSAXVersion (SAX2) on how to retrieve the SAX2 default parser structure. IterParse should pass chunks into the parser and buffer the events it receives. When its __next__() method is called, it returns one event or passes new chunks until there is an event to return. This is needed as IterParse has to convert between libxml2 push (SAX) and Python pull (iter). As for the input to the libxml2 parser, there are two possible ways: one is to pass data chunks in through xmlParseChunk and the other is to use xmlCreateIOParserCtxt and implement xmlInputReadCallback (xmlio.h) to have libxml2 request data by itself. Python events (start, end, start-ns, end-ns) are created as follows: * "*-ns" events must be extracted from the libxml2 xmlSAX2StartElementNs call (passed in arguments "prefix"/"URI" and the char* array "namespaces"). They must be stored on a stack to build the respective "end-ns" events. * "start" is somewhat tricky, as it would be a bad idea to allow modifications of the XML structure during that iterator cycle. Maybe it's enough to document that, but there may be ways to crash lxml with certain tree operations. Note also that care has to be taken to prevent Python from garbage collecting the element before the "end" event. The best way to do that is to store a Python reference to that element on a stack. * "end" is simple then: pop the element from the stack and return it. That's all I can come up with so far. So, if anyone is interested in taking a look at it, I'd be glad to hear about it. :) Stefan

3 4

[lxml-dev] element of an xpath evaluation
by Andreas Pakulat 03 Jun '06

03 Jun '06

Hi, very recently I needed a program that evaluates a xpath and displays the "result" graphically in a tree of the xml file that was used. I found XPath Explorer a java application, however it does quite some more stuff than I need and there certain things that just don't work as I want. I figured I could very easily provide something similar using lxml and PyQt4. However if I want to highlight the tree node that the xpath matches I have a "problem" when the xpath matches attributes or text nodes. So the question is: Is there a way using lxml to find out to which element a certain non-element result of an xpath evaluation belongs? Andreas -- You now have Asian Flu.

2 7

[lxml-dev] Defining name spaces in lxml/xslt
by Petr van Blokland 02 Jun '06

02 Jun '06

Hi, new to lxml I may be overlooking some hints in the doc files. I am trying to define a name space (myspace) in Python, but I don't know by which method my name space object is getting called from lxml/xslt. class AbcElement(ElementBase): def ###METHODNAME###(self): return 'Output of abc' ns = Namespace('http://xml.petr.com/xpyth3/ns/myspace') ns['abc'] = AbcElement xslt = '''<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:abc="http://xml.petr.com/xpyth3/xpath/ns/myspace" xsl:extension-element-prefixes='myspace' > <xsl:template match="/"> <foo> < myspace:abc/> </foo> </xsl:template> </xsl:stylesheet>''' Where ###METHODNAME### I don't know. I am using lxml 1.0 beta Suggestions? Kind regards, Petr van Blokland ---------------------------------------------- Petr van Blokland buro(a)petr.com | www.petr.com | +31 15 219 10 40 ----------------------------------------------

2 5

[lxml-dev] improved valgrind suppressions for Python?
by Martijn Faassen 01 Jun '06

01 Jun '06

Hi there, Since upgrading to a new version of valgrind quite a while ago, valgrind gets spammy, even with the Python suppressions in place. I suspect this is because valgrind is getting more picky. I'm getting stuff like this: ==10199== Conditional jump or move depends on uninitialised value(s) ==10199== at 0x1B8F4FD1: (within /lib/ld-2.3.5.so) ==10199== by 0x1B8EA4AA: (within /lib/ld-2.3.5.so) ==10199== by 0x4111F18F: (within /lib/tls/libc-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x4111FB85: _dl_open (in /lib/tls/libc-2.3.5.so) ==10199== by 0x4118FD32: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x41190486: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x4118FDB0: dlopen (in /lib/tls/libdl-2.3.5.so) ==10199== by 0x80DCE40: _PyImport_GetDynLoadFunc (in /usr/bin/python2.4) ==10199== by 0x80D2622: _PyImport_LoadDynamicModule (in /usr/bin/python2.4) ==10199== by 0x80D223F: (within /usr/bin/python2.4) and generally ld related output, presumably something to do with dynamic linking. This makes it harder to pick out potentially real memory issues. I'm wondering whether others get this same behavior with valgrind and have it fixed, or whether everybody is stuck with it. I looked for newer suppressions in the Python repository at some point, but couldn't find it then. My valgrind version is 3.0.1 Regards, Martijn

2 7

[lxml-dev] Faster iteration and deallocation
by Stefan Behnel 31 May '06

31 May '06

Hi all, I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway and found a lot of code duplication between functions that iterate over the tree. I replaced that by a set of BEGIN/END C macros, which has three effects: * Tree iteration now uses the same code throughout lxml * It's much shorter and thus (hopefully) easier to understand * It's about 20-30% faster than the previous recursive implementations I took another somewhat experimental step and removed the _NodeBase superclass from _Attrib to make it a plain Python object that sits on an _Element. This allowed for another major clean up and a huge simplification of the proxy code, since there is only one proxy type left. I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain. According to the test cases, everything works nicely as before and I will give valgrind a run to see if there's anything suspicious left. It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it. Any feedback is appreciated. Stefan

4 8

[lxml-dev] filtering namespaces in findall
by Stefan Behnel 31 May '06

31 May '06

Hi, since I was rewriting the tree iterator anyway, I stumbled over the fact that ElementTree's getiterator() can filter for any tag ("*"), but not for any tag within a namespace ("{namespace}*"). Since this is just a cheap and straight forward little extension to the node filter (and a pretty nice feature to have), I added that to lxml and hope for inclusion in a future ElementTree version also (or maybe a hint why Fredrik assumes this to be inappropriate). Note that it's not currently supported in findall, though, as we use the same implementation as ElementTree there. Stefan

1 1

[lxml-dev] Bug in memory management (I guess ;)
by Johnny deBris 31 May '06

31 May '06

Hi! The past couple of months I kept running into some bug. For some reason the namespace information of certain nodes got lost after moving the nodes from one document to another. I kept trying to isolate the problem, but without success, until this week I found that the reason the information gets lost seems to have to do with the document getting garbage collected even though some nodes are still referred to. With this in mind I managed to write the following demonstration snippet: #-------------------------------------------------------------------------- from lxml import etree s1 = '<a xmlns:a="A:"><b a:b="b" /></a>' tree1 = etree.fromstring(s1) btag = tree1.xpath('//b')[0] del tree1 s2 = '<x />' tree2 = etree.fromstring(s2) tree2.append(btag) # this produces crap... print etree.tostring(tree2) #-------------------------------------------------------------------------- The problem seems to be the 'del' statement: this seems to free the document instance or something, even though there's still a reference to one of its nodes. Serializing the node later on will result in garbage. When the 'del' line is commented out, the XML is produced properly. I hope this is enough information for you guys to track the bug: I'm no C programmer myself, so I don't dare diving into the Pyrex, and also I may have made some errors in my explanation... If you need more information, let me know. Cheers, Guido

2 5

[lxml-dev] Bug with attribute mangling when adding child elements
by Noah Slater 30 May '06

30 May '06

Hello, I noticed today that certain child attribute values are mangled when you try to insert/append child elements onto parent elements. I have written a test script for this as usual. I'm probably missing something... again. :) Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

3 6

[lxml-dev] Difference between ResultTree.__str__ and ResultTree.write
by Noah Slater 30 May '06

30 May '06

Hi, Could we get the difference between ResultTree.__str__ and ResultTree.write properly documented somewhere ([1]?) on the website or via docstrings etc as there only seems to be one reference to it that I can find [2] By the way, the ability to change "output" parameters with write is an excellent addition to this package - such a time saver for me! :) Thanks, Noah [1] http://codespeak.net/lxml/api.html [2] http://codespeak.net/pipermail/lxml-dev/2006-May.txt -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 5

[lxml-dev] credits update: Stefan is the maintainer
by Martijn Faassen 30 May '06

30 May '06

Hi there, I've just updated the CREDITS.txt of lxml to reflect what had become the situation for a long time already: Stefan Behnel is currently the main developer of lxml and its maintainer, and my own role as creator and initial maintainer has diminished somewhat. I'm not disappearing and will remain involved in lxml development for the foreseeable future, but Stefan calls the shots, as he took the responsibility - this is as it should be in open source. I recognized what was going on in december and in personal communication then gave Stefan the keys to the lxml car. He's been doing a great job with it since, and as the 1.0 release is pending we thought we should announce this officially. I personally think Stefan's involvement is an absolutely awesome development. The most obvious reason why this is great is of course Stefan's qualities as a developer and project leader. Less obvious but also very exciting to me is that it makes lxml into a true open source project in the community sense. lxml's future is no longer dependent on a single person, namely myself. Stefan has shown great skill in both sides of managing a successful open source project: the code side and the people side. lxml has now become one of the most powerful XML libraries in the Python world, in the area of feature support as well as in the area of performance. We came a long way in the year and almost-a-half since my initial breakthrough in getting the lxml ball rolling. On the more personal front I also met a few people I'm happy to know now, got to speak about lxml at conferences, and of course got a great XML library out of it too. Thanks to all contributors, and of course especially to Stefan, in making this possible! Regards, Martijn

3 2