Mailman 3 August 2006 - lxml - The Python XML Toolkit

[lxml-dev] Developer version of the web pages online
by Stefan Behnel Aug. 9, 2006

Aug. 9, 2006

Hi all, I thought it would be a good idea to advocate the current developer version of lxml a bit more. So I uploaded the web pages from the trunk to http://codespeak.net/lxml/dev/ Their differences are obviously generated by a script using lxml.etree. :) Stefan

1 0

[lxml-dev] ElementTree and lxml advertised by yahoo
by Olivier Grisel Aug. 9, 2006

Aug. 9, 2006

The lxml part is just a reference at the bottom of the page, but anyway that's still a good start :) http://developer.yahoo.com/python/python-xml.html#element -- Olivier

1 0

[lxml-dev] lxml 1.0.3 and 1.1beta on cheeseshop
by Stefan Behnel Aug. 9, 2006

Aug. 9, 2006

Hi all, I finally managed to get 1.1beta out, right after releasing 1.0.3. 1.0.3 is a bug fix release. Since it fixes a crash in XSLT result handling, updating is recommended. 1.1beta is the last pre-release before the shiny new 1.1 series will take the lead. Despite the surprisingly short change log, it contains tons of changes under the hood and some major improvements in flexibility. It is the first lxml version to compile and run under Python 2.5 beta (3), comes with a C-API that makes it extensible by other Python C modules, and features an additional data-binding API on top of etree (objectify). For further information on the features of lxml 1.1, please refer to the HTML documentation in the source distribution or read the text files online: http://codespeak.net/svn/lxml/trunk/doc Note that lxml 1.1 requires a patched Pyrex version if you want to compile from non-release or modified sources. It is available here: http://codespeak.net/svn/lxml/pyrex This version of Pyrex supports Python 2.5 and public C-API generation, so it may be of interest to more than only lxml developers. As always, I'm happy about any egg contributions or bug reports that help in making lxml 1.1 the greatest Python XML tool ever. Have fun, Stefan Changelogs: (note that 1.1beta also contains the changes from 1.0.3) 1.1beta (2006-08-08) Features added * Unlock the GIL for deep copying documents and for XPath() * Support for Python 2.5 beta * New compact keyword argument for parsing read-only documents * Support for parser options in iterparse() * The namespace axis is supported in XPath and returns (prefix, URI) tuples * The XPath expression "/" now returns an empty list instead of raising an exception * XML-Object API on top of lxml (lxml.objectify) * Customizable Element class lookup: o Support for externally provided lookup functions o lxml.elements.classlookup module implements different lookup mechanisms * Support for processing instructions (ET-like, not compatible) * Public C-level API for independent extension modules Bugs fixed * XPathSyntaxError now inherits from XPathError * Threading race conditions in RelaxNG and XMLSchema * Crash when mixing elements from XSLT results into other trees, concurrent XSLT is only allowed when the stylesheet was parsed in the main thread * The EXSLT regexp:match function now works as defined (except for some differences in the regular expression syntax) * Setting element.text to '' returned None on request, not the empty string * iterparse() could crash on long XML files * Creating documents no longer copies the parser for later URL resolving. For performance reasons, only a reference is kept. Resolver updates on the parser will now be reflected by documents that were parsed before the change. Although this should rarely become visible, it is a behavioral change from 1.0. 1.0.3 (2006-08-08) Features added * Element.replace(old, new) method to replace a subelement by another one Bugs fixed * Crash when mixing elements from XSLT results into other trees * Copying/deepcopying did not work for ElementTree objects * Setting an attribute to a non-string value did not raise an exception * Element.remove() deleted the tail text from the removed Element

1 1

[lxml-dev] Google Analytics tagger script based on lxml
by Olivier Grisel Aug. 9, 2006

Aug. 9, 2006

Hi list, Thanks to the neat HTMLParser feature in lxml I was able to quickly write a simple script to add Google Analytics tags at the end of static HTML files (generated from a REST source for instance). Feel free to use it should you find it any useful: http://champiland.homelinux.net/evogrid/code/evogrid.og.main/ga_tagger.py NB: google analytics is a free as in beer web traffic analyser: http://www.google.com/analytics/ Best, -- Olivier

1 0

Re: [lxml-dev] Segfault in lxml during element copy
by Stefan Behnel Aug. 5, 2006

Aug. 5, 2006

Hi John, John Krukoff wrote: > Okay, I've managed to create a crashing test case that's down to a > reasonable number of lines of code. I don't think I can remove anything > else and still have it crash. Great, thanks for stripping this down. > There's even some very odd changes that stop it from crashing, such as > shortening "fieldset" to "f". Fortunately, narrowing this down allowed > me to create a workaround for the real program, so fixing this is no > longer so urgent for me. > > I've also attached the results of a valgrind run using the recommended > command line parameters on the test program. I didn't bother gzipping > it, because it's pretty small. > > Please let me know if this fails to crash for you. I have to run it > using "python test.py" instead of "./test.py" to see the glibc error. It 'nicely' crashes for me and I think I can tell where it comes from. We use a global dictionary in the parser that stores tag names, attribute values, etc. It mainly serves the purpose of reducing the number of expensive malloc calls and avoiding duplicated storage of constant strings. Normally, it works just fine, unless there are operations that create additional dictionaries, like XSLT. :( So what happens in your case, is: when you move the content of the XSLT result document over to the document you parsed, it will contain strings from two different dictionaries (I just verified that). When the documents are freed, libxml2 checks if the strings it frees are in the document dictionary, sees that it is not the case (as it came from a different dictionary) and then frees it. This leaves stale pointers in the second dictionary. It's too bad we can't control the dictionary created by libxslt for transformations, as it is automatically created and used when we request a transformation context. So we can't just replace the dictionary afterwards. I'm not quite sure what to do here. There are ways to fix this, but they can be expensive, so I'll just have to figure out which one to go. One solution could be to extend the deep traversal that follows moving a subtree to a different document. We could let it check if the dicts are the same, and if they are not, copy the strings stored in the source dictionary to the destination dictionary. As I said, this can be expensive but is a rare case as (so far) it only applies to partial XSLT results being moved around. On the other hand, this would also allow moving subtrees between threads (which use independent dictionaries as well), so maybe it's worth it... As this problem (currently) only appears in XSLT, a second way to handle it would be to replace the dictionary of the transformation context after initialisation, but /before/ running the transform. That way, there should be less content already stored in it that would have to be moved. While the second one sounds like the least expensive, maybe there are even better ways I did not think of. I'll take a look at it. Again, thanks for reporting this and for providing a test case, Stefan > ------------------------------------------------------------------------ > > import lxml.etree as etree > > definitionXml = etree.XML( '''<transform> > <widget displayType="fieldset"/> > </transform>''' ) > > definitionXml[ : ] = etree.XSLT( etree.XML( '''<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> > <xsl:output method="html" indent="no"/> > > <xsl:template match="/"> > <html> > <xsl:apply-templates/> > </html> > </xsl:template> > > <xsl:template match="widget"> > <xsl:element name="{@displayType}"/> > </xsl:template> > > </xsl:stylesheet>''' ) )( definitionXml[ 0 ] ).getroot( )[ : ] > > # Segfault occurs on this line. > del definitionXml > > print "Didn't crash!" > ------------------------------------------------------------------------ > ==29947== Invalid free() / delete / delete[] > ==29947== at 0x401C0C3: free (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so) > ==29947== by 0x45A161A: xmlFreeNodeList (in /usr/lib/libxml2.so.2.6.26) > ==29947== Address 0x48AD24C is 20 bytes inside a block of size 1,024 free'd > ==29947== at 0x401C0C3: free (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so) > ==29947== by 0x46B3EE5: xmlDictFree (in /usr/lib/libxml2.so.2.6.26)

3 7

[lxml-dev] objectify, ObjectPath and Benchmarks
by Stefan Behnel Aug. 4, 2006

Aug. 4, 2006

Hi all, I have already mentioned that lxml 1.1 will feature an alternative API, lxml.elements.objectify, which is similar to Amara and gnosis.objectify, but written in Pyrex. The implementation is now nearing completion, so that 1.1beta will hopefully find its way towards cheeseshop early next week. It allows you to access XML in a data-binding like style, so that you can do this: >>> root=XML('<root><a><b><c><d>HALLO</d><d>WORLD</d></c></b></a></root>') >>> print root.a.b.c.d, '--', root.a.b.c.d[1] HALLO -- WORLD A complete description is here: http://codespeak.net/svn/lxml/branch/capi/doc/objectify.txt objectify also features an additional path language (ObjectPath) based on the normal object attribute access scheme. It is implemented independent of the actual objectify API so that it can be used without switching the Element implementation over to 'objectify'. The language accepts expressions like the two used above, just written as strings or lists: * "root.a.{someNamespace}b.c.d" * "root.a.b.c.d[1]" * ".a.b.c" * ['root', '{otherNamespace}a'] * ['root', 'a', 'b', 'c', '{andAnotherNamespace}d[1]'] Here are a few timeit benchmarks: Setup: from lxml.elements.objectify import register, ObjectPath register() from lxml.etree import XML root = XML('<root><a><b><c><d/><d/><d/></c></b></a></root>') Normal Python object access tests for comparison: * root.a.b.c.d 10000 loops, best of 3: 16.3 usec per loop * root.a.b.c.d[0] 10000 loops, best of 3: 16.7 usec per loop * root.a.b.c.d[2] 100000 loops, best of 3: 18.4 usec per loop ObjectPath tests *without* parsing, i.e. timings of the call "path(root)" after an additional Setup as follows: * path = ObjectPath('root.a.b.c.d') 100000 loops, best of 3: 2.76 usec per loop * path = ObjectPath('root.a.b.c.d[0]') 100000 loops, best of 3: 2.77 usec per loop * path = ObjectPath('root.a.b.c.d[2]') 100000 loops, best of 3: 2.85 usec per loop Including parsing: * "path=ObjectPath('root.a.b.c.d'); path(root)" 10000 loops, best of 3: 27 usec per loop * "path=ObjectPath('root.a.b.c.d[2]'); path(root)" 10000 loops, best of 3: 29.7 usec per loop The same based on lists: * "path=ObjectPath(['root', 'a', 'b', 'c', 'd']); path(root)" 10000 loops, best of 3: 16.7 usec per loop * "path=ObjectPath(['root', 'a', 'b', 'c', 'd[2]']); path(root)" 10000 loops, best of 3: 18 usec per loop As you can see, the parser is not the fastest, especially for strings. It actually uses REs internally, as ObjectPath expressions are non trivial to parse (namespaces, indexes, ...). However, once the expression is parsed, element access is impressively fast, as it runs entirely in C. In the limited area of its applicability, it is even faster than full fledged XPath: * Setup: path=XPath('/root/a/b/c/d') Timing: "path(root)" 10000 loops, best of 3: 10.4 usec per loop * Timing: "path=XPath('/root/a/b/c/d'); path(root)" 10000 loops, best of 3: 44.8 usec per loop So I hope people find it useful. Stefan

2 5

[lxml-dev] lxml goals
by Татаринов Андрей Aug. 4, 2006

Aug. 4, 2006

Hi, This is all very interesting, but the only thing I can't understand what does it have in common with lxml? In fact, for a quiet some time I do not understand the goal of lxml project. At first it was "ElementTree on top of libxml2", after it becames more and more bloated with ET-Incompatible API, now, program that uses lxml cannot be easy ported back to ElementTree. May be it's time to split into ET-implementations and lxml-specific? Or to say "lxml is no more just an ElementTree implementation, but a separate project with it's own ideoms"? 03.08.06, 20:02, Stefan Behnel <behnel_ml(a)gkec.informatik.tu-darmstadt.de>: > Hi all, > I have already mentioned that lxml 1.1 will feature an alternative API, > lxml.elements.objectify, which is similar to Amara and gnosis.objectify, but > written in Pyrex. The implementation is now nearing completion, so that > 1.1beta will hopefully find its way towards cheeseshop early next week. [...] > Stefan

3 2

[lxml-dev] XML Schema: Getting more information on validation failures?
by Adhamh Findlay Aug. 2, 2006

Aug. 2, 2006

Hello, I'm new to lxml and I'm trying to get more information on why some validation is failing. Here is the code I am currently using: try: xmlschema.assertValid(xml_doc) except etree.DocumentInvalid: traceback.print_exc() print log print error.domain_name print error.type_name sys.exit() Here's the output I get: Traceback (most recent call last): File "./xml.py", line 42, in ? xmlschema.assertValid(xml_doc) File "etree.pyx", line 1624, in etree._Validator.assertValid DocumentInvalid: Document does not comply with schema Traceback (most recent call last): File "./xml.py", line 46, in ? print error.domain_name AttributeError: 'NoneType' object has no attribute 'domain_name' Is there any way to get more information than this? Thanks, Adhamh

2 1

[lxml-dev] Some performance results with threads
by Stefan Behnel Aug. 2, 2006

Aug. 2, 2006

Hi, I did a little testing on a dual processor linux machine with the current trunk. I had a very simple setup with a number of threads (8-16) that each created a separate parser, parsed a 3MB string or file and then ran a small XSLT on it. Parsing and XSLT are operations that free the GIL for the majority of their internal work. The outcome was that the system was always between 20% and 40% idle. So, there is a certain speedup in multi-processor environments, but don't expect too much, especially when adding more processors. It shows that it makes sense to use threads on, say, a web server that has to serve other content in parallel (like static content), so that it can make use of a third of the processing time itself. But it will not get you 100% more throughput by doubling the number of processors. It looks like you should really expect less than a 50% speedup, depending on how much time your application actually spends in parsing, serialising, validating and XSLT. If your application does a lot of XML handling in Python code (like tree iteration etc.), the ratio can get close to 0, but if you have complex XSLTs or large schemas/documents to validate, the speedup can potentially be much higher. (I never thought I'd ever tell someone to rewrite code in XSLT to make it /faster/ ...) Stefan

2 2

[lxml-dev] ElementTree comment behavior
by Martijn Faassen Aug. 2, 2006

Aug. 2, 2006

Hi there, The whole XPath root node issue led me to investigate lxml's behavior with comment nodes, thinking that we might not do the right thing with mutation (as Comments subclass Element). However, it seems to behave rationally enough: >>> import lxml >>> from lxml import etree >>> c = etree.Comment('foo') >>> c.append(etree.Element('bar')) >>> len(c.getchildren()) 0 (I wonder what happens in the C tree though here.. cursory inspection of the tree.c code of libxml2 doesn't reveal special code to handle this case) Unfortunately, ElementTree behaves differently in this case! >>> from elementtree import ElementTree as etree2 >>> c = etree2.Comment('foo') >>> c.append(etree.Element('bar')) >>> len(c.getchildren()) 1 Evidently it allows child Elements to be added to comments. What to do in this case? Regards, Martijn

2 3