Mailman 3 June 2006 - lxml - The Python XML Toolkit

[lxml-dev] 'tree.iterwrite()' ?
by Stefan Behnel 15 Jun '06

15 Jun '06

Hi all, we had some requests on the list about custom serialisers. The main problem is that you can't just do this with el.getiterator(), since you do not get the back-tracking events that are required to close the tags. I didn't really think about it, but since I stumbled over iterparse() again recently, I happened to get the idea that an interesting way to support serialisers would be an equivalent iterwrite() method on ElementTree objects. The semantics would be exactly like iterparse() with the only exception that it iterates over the existing tree. Another difference would be that it should return all events (start, end, start-ns, end-ns) by default, since all of them are commonly required for serialisation. So, basically, a custom serialiser without namespace output would look like this: for event, element in tree.iterwrite(events=("start", "end")): tagname = element.tag.split("}")[-1] if event == 'start': print "<%s%s%s>%s" % (tagname, formatAttr(element.attrib), element.text or '') else: print "</%s>%s" % (tagname, element.tail or '') When namespace support is added, this becomes somewhat more complicated, though. It would mean that you'd have to collect new namespaces on "start-ns" and write them into the next element. That's not too much work, but somewhat of a hassle, since basically every serialiser has to do that. I still think such an interface is worth thinking about. Obviously, the alternative would be an object with hook methods, but normally that's much more code and internal state keeping than an iterator gives you. There's also an alternative iterwrite that yields triples including the new namespaces as the third item, something like this (now defaulting to yield "start" and "end" as only events): for event, element, new_ns_map in tree.iterwrite(): tagname = element.tag.split("}")[-1] if event == 'start': print "<%s%s%s>%s" % (tagname, formatAttr(element.attrib), formatNs(new_ns_map), element.text or '') else: print "</%s>%s" % (tagname, element.tail or '') I think that's a reasonable modification for targeting serialisers. Note that the above code does not currently keep track of prefixes. That makes it more complicated again, as it still requires some stack keeping. We could perhaps get around that by doing some magic in the new_ns_map dictionary(-like object), although too much magic isn't a good thing. A mapping of current namespaces could be helpful anyway. Also, I'm not sure about the right name for that method, as serialisation is not its only application. You could also imagine copying trees with it in a customised way, including dropping elements after examination, flattening subtrees etc. So, maybe there's a more general name like "iterstartend()" or something. I didn't come up with one yet. iterwrite() still sounds somewhat tempting. I'm not sure if this is a good idea. Namespaces make this more complex then you'd think. A subclassable serialiser that keeps track of namespace mappings or simply recursive serialisation might still be better options. I just liked the idea of iteration in this context... Any comments on this? Stefan

2 2

[lxml-dev] saxify only produces events for *NS handlers
by Andreas Pakulat 12 Jun '06

12 Jun '06

Hi, could you please add a comment to the sax.txt that saxify _always_ produces SAX events for the *NS handlers? I just stumbled over this parsing an html file and using saxify to extract some content and had to look up the code in sax.py. I know the examples in sax.txt use *NS, however it's not clear that there won't be any calls ever to the non-NS functions. Andreas -- You never hesitate to tackle the most difficult problems.

3 5

[lxml-dev] Error messages in XSLT
by Petr van Blokland 12 Jun '06

12 Jun '06

Stefan, is it possible to show more of the error message than only: etree.XSLTApplyError: runtime error (element 'value-of') e.g. which <xsl:value-of> in the template is wrong or even what is wrong on the XPath? Kind regards, Petr ---------------------------------------------- Petr van Blokland buro(a)petr.com | www.petr.com | +31 15 219 10 40 ----------------------------------------------

2 2

[lxml-dev] Crash with lxml on Mac OS X
by Piet van Oostrum 11 Jun '06

11 Jun '06

I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: from lxml import etree root = etree.Element("html") head = etree.SubElement(root, "head") title = etree.SubElement(head, "title") title.text = "Page Title" body = etree.SubElement(root, "body") body.set("bgcolor", "#ffffff") body.text = "Hello, World!" tree = etree.ElementTree(root) tree.write("page.xhtml") tree = etree.parse("page.xhtml") print tree.findtext("head/title") The error messages are: TEMP> python testlxml.py python(3059) malloc: *** Deallocation of a pointer not malloced: 0x2; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0xa86c1dbc; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0x97e90; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0x80; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Page Title Modules/gcmodule.c:215: failed assertion `gc->gc.gc_refs == GC_REACHABLE' Abort trap The file is created correctly, however. When I insert some print messages, some or all of the errors disappear. When I split the program in two, one writing and another reading the file they run without problems. I have installed through darwinports: libxml2 @2.6.23_0 libxslt @1.1.15_0 I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors. -- Piet van Oostrum <piet(a)cs.uu.nl> URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet(a)vanoostrum.org

3 10

[lxml-dev] Note to egg contributors: libxml2 2.6.26 breaks backwards compatibility
by Stefan Behnel 11 Jun '06

11 Jun '06

Hi all, libxml2 2.6.26 introduces a new function xmlXPathContextSetCache in xpath.h. This means that eggs compiled against this version will not run with older installations, as the dynamic linking process will bail out with a missing symbol. I therefore have to ask our helpful egg contributors not to upgrade to that version or to make sure the eggs are still compiled against an older version (like 2.6.24). Sadly enough, libxml2 does not use versioned headers, so we can't easily get around this problem ourselves. Stefan

2 1

[lxml-dev] Threading branch merged into trunk
by Stefan Behnel 10 Jun '06

10 Jun '06

Hello everyone, I managed to fix a couple of bugs that remained the threading branch and merged it into the trunk. As it looks now, thread support will become an official feature in 1.1. Parsers now use locks to prevent concurrent access, which means that the access to a single parser is completely serialised. If you want concurrency, you can either use the default parser (which is now cloned for each thread on first request) or simply copy() the parser of your choice to use it in a thread. Note that calls to XML() and HTML() always use the same parser respectively, so these are non-threaded. According to the libxml2 docs, validation is supposed to be thread-safe, so you should be able to share RelaxNG and XMLSchema objects between threads. I did not test this, though. Note also that the error_log property is not currently thread aware, so error reporting may not work as expected. I will also take a look at XSLT, which is said to be thread-safe in the docs. Hoever, the current implementation does not allow this, as it holds further state within the XSLT objects. I'll have to fix that somehow. There are currently some things to remember: sharing generated trees between threads should be ok, but you should avoid copying elements between trees that were generated in different threads. This is error prone, as part of the data remains in the originating thread (the parser dictionary), so if the elements are removed from the document and the originating thread is terminated, this data may be freed without warning! It should be safe to do the parse-generate-serialise cycle inside a thread and it should be safe to set up trees, validators, etc. in the main program and use it in threads. However, things like creating a new thread for each parse process, terminating it when parsing is done and then starting to use the generated tree in other threads are generally A BAD IDEA. We could fix that by not using parser dictionaries in threads, but that would reduce the parser performance. I'll just write up a doc/threading.txt (and also a FAQ entry) to clearly mark this kind of stuff as a no-no. If things work out well, there will be an alpha release, uhm, in the not so far future. :) Stefan

1 0

[lxml-dev] XML_PARSE_COMPACT
by Martijn Faassen 09 Jun '06

09 Jun '06

Hi there, I just ran into the XML_PARSE_COMPACT option for libxml2. There are some caveats to using it apparently so we should carefully check whether lxml can work with it, but it creates a more compact tree which can lead to performance gains, apparently. This seems to be describing how it works: http://mail.gnome.org/archives/xml/2005-August/msg00138.html Regards, Martijn

2 12

[lxml-dev] lxml 1.0.1 - includes important bug fixes
by Stefan Behnel 09 Jun '06

09 Jun '06

Hi everyone, after a major bug squeezing marathon during the passed week, I can finally announce the release of lxml 1.0.1. http://cheeseshop.python.org/pypi/lxml This is a bug-fix-only release for the 1.0 series. It fixes a number of important bugs, including possible crashes and parsing from file(-like) objects. Note that parsing unicode strings from these objects is no longer supported. Updating is recommended once the relevant eggs are on cheeseshop. Thanks to all bug-finders and test case contributors! Stefan Features added * Repeated calls to Element.attrib now efficiently return the same instance Bugs fixed * Document deallocation could crash in certain garbage collection scenarios * Extension function calls in XSLT variable declarations could break the stylesheet and crash on repeated calls * Deep copying Elements could loose namespaces declared in parents * Deep copying Elements did not copy tail * Parsing file(-like) objects failed to load external entities * Parsing 8-bit strings from file(-like) objects raised an exception * xsl:include failed when the stylesheet was parsed from a file-like object * lxml.sax.ElementTreeProducer did not call startDocument() / endDocument() * MSVC compiler complained about long strings (supports only 2048 bytes)

3 3

[lxml-dev] HTML serialization
by Ian Bicking 09 Jun '06

09 Jun '06

Howdy. Just started using lxml (http://blog.ianbicking.org/neutral-templating.html), mostly for its HTML parser. One thing that felt missing to me was (X)HTML serialization. Specifically, XML serialization tends to create HTML which browsers don't like (like <script src="..." />). But it's relatively easy to create XHTML that browsers like just fine, even if it is served up as text/html. I think some people have opinions on whether it is valid to try to produce output that can be served as HTML or XHTML... I personally don't care at all, I just know it works well for me. But of course, serializing to "real" HTML (e.g., <br> instead of <br />) is also an option. Anyway, HTML serialization would be really nice to have. I have a simple routine for doing this, mostly translated from something I wrote for ElementTree, which in turn is just a copy of ElementTree's serialization with rules for a few particular HTML elements: http://svn.pythonpaste.org/Paste/WSGIOverlay/trunk/wsgioverlay/htmlserializ…. I don't know how well this works with namespaces, and it was originally written for ElementTidy, where everything becomes namespaced XHTML; etree.HTML seems to parse to nodes without a namespace. But anyway... I don't really like my implementation, but it would be nice if there was some implementation in lxml. Ian

3 3

[lxml-dev] XPath default namespace
by Mircea Amarascu 08 Jun '06

08 Jun '06

Hello, I'm having this XML fragment: <a xmlns="ns1"> <b id="first">Text</b> </a> If I aplly the following XPath expression on the parsed tree: tree.xpath('/a/b[@id="first"]') this will not return anything. Of course I should give a second argument, a dictionary that maps the namespace prefix to the namespace URI: tree.xpath('/a/b[@id="first"]', {'?': 'ns1'}) but I don't know what prefix to use for the default namespace (None doesn't work, nor the empty string). What are the best sources for documentation for lxml's XPath and XMLSchema features? The answer to the XPath question is probably common knowledge for the rest of you, however I've posted the question here as I couldn't find detailed info regarding this topics. Thank you for your time.

4 8