Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] Remarks on implementing iterparse()
by Stefan Behnel 22 Jun '06

22 Jun '06

Hi all, since I won't have the time to implement iterparse() any time soon, here's a proposal on how it should be implemented, in case someone wants to take a shot at it. "iterparse" will be (or will return) an iterable object, let's call it IterParse for clarity. A class is basically the only way of implementing iterators in Pyrex. For the internal SAX part, IterParse will likely work a lot like lxml.sax.ElementTreeContentHandler. We'd need a custom wrapper to the default libxml2 SAX handler to intercept the parse events (this means implementing C helper functions for the SAX events) /after/ they were processed by libxml2. See xmlSAXVersion (SAX2) on how to retrieve the SAX2 default parser structure. IterParse should pass chunks into the parser and buffer the events it receives. When its __next__() method is called, it returns one event or passes new chunks until there is an event to return. This is needed as IterParse has to convert between libxml2 push (SAX) and Python pull (iter). As for the input to the libxml2 parser, there are two possible ways: one is to pass data chunks in through xmlParseChunk and the other is to use xmlCreateIOParserCtxt and implement xmlInputReadCallback (xmlio.h) to have libxml2 request data by itself. Python events (start, end, start-ns, end-ns) are created as follows: * "*-ns" events must be extracted from the libxml2 xmlSAX2StartElementNs call (passed in arguments "prefix"/"URI" and the char* array "namespaces"). They must be stored on a stack to build the respective "end-ns" events. * "start" is somewhat tricky, as it would be a bad idea to allow modifications of the XML structure during that iterator cycle. Maybe it's enough to document that, but there may be ways to crash lxml with certain tree operations. Note also that care has to be taken to prevent Python from garbage collecting the element before the "end" event. The best way to do that is to store a Python reference to that element on a stack. * "end" is simple then: pop the element from the stack and return it. That's all I can come up with so far. So, if anyone is interested in taking a look at it, I'd be glad to hear about it. :) Stefan

3 4

[lxml-dev] element of an xpath evaluation
by Andreas Pakulat 03 Jun '06

03 Jun '06

Hi, very recently I needed a program that evaluates a xpath and displays the "result" graphically in a tree of the xml file that was used. I found XPath Explorer a java application, however it does quite some more stuff than I need and there certain things that just don't work as I want. I figured I could very easily provide something similar using lxml and PyQt4. However if I want to highlight the tree node that the xpath matches I have a "problem" when the xpath matches attributes or text nodes. So the question is: Is there a way using lxml to find out to which element a certain non-element result of an xpath evaluation belongs? Andreas -- You now have Asian Flu.

2 7

[lxml-dev] Defining name spaces in lxml/xslt
by Petr van Blokland 02 Jun '06

02 Jun '06

Hi, new to lxml I may be overlooking some hints in the doc files. I am trying to define a name space (myspace) in Python, but I don't know by which method my name space object is getting called from lxml/xslt. class AbcElement(ElementBase): def ###METHODNAME###(self): return 'Output of abc' ns = Namespace('http://xml.petr.com/xpyth3/ns/myspace') ns['abc'] = AbcElement xslt = '''<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:abc="http://xml.petr.com/xpyth3/xpath/ns/myspace" xsl:extension-element-prefixes='myspace' > <xsl:template match="/"> <foo> < myspace:abc/> </foo> </xsl:template> </xsl:stylesheet>''' Where ###METHODNAME### I don't know. I am using lxml 1.0 beta Suggestions? Kind regards, Petr van Blokland ---------------------------------------------- Petr van Blokland buro(a)petr.com | www.petr.com | +31 15 219 10 40 ----------------------------------------------

2 5

[lxml-dev] improved valgrind suppressions for Python?
by Martijn Faassen 01 Jun '06

01 Jun '06

Hi there, Since upgrading to a new version of valgrind quite a while ago, valgrind gets spammy, even with the Python suppressions in place. I suspect this is because valgrind is getting more picky. I'm getting stuff like this: ==10199== Conditional jump or move depends on uninitialised value(s) ==10199== at 0x1B8F4FD1: (within /lib/ld-2.3.5.so) ==10199== by 0x1B8EA4AA: (within /lib/ld-2.3.5.so) ==10199== by 0x4111F18F: (within /lib/tls/libc-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x4111FB85: _dl_open (in /lib/tls/libc-2.3.5.so) ==10199== by 0x4118FD32: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x41190486: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x4118FDB0: dlopen (in /lib/tls/libdl-2.3.5.so) ==10199== by 0x80DCE40: _PyImport_GetDynLoadFunc (in /usr/bin/python2.4) ==10199== by 0x80D2622: _PyImport_LoadDynamicModule (in /usr/bin/python2.4) ==10199== by 0x80D223F: (within /usr/bin/python2.4) and generally ld related output, presumably something to do with dynamic linking. This makes it harder to pick out potentially real memory issues. I'm wondering whether others get this same behavior with valgrind and have it fixed, or whether everybody is stuck with it. I looked for newer suppressions in the Python repository at some point, but couldn't find it then. My valgrind version is 3.0.1 Regards, Martijn

2 7

[lxml-dev] Faster iteration and deallocation
by Stefan Behnel 31 May '06

31 May '06

Hi all, I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway and found a lot of code duplication between functions that iterate over the tree. I replaced that by a set of BEGIN/END C macros, which has three effects: * Tree iteration now uses the same code throughout lxml * It's much shorter and thus (hopefully) easier to understand * It's about 20-30% faster than the previous recursive implementations I took another somewhat experimental step and removed the _NodeBase superclass from _Attrib to make it a plain Python object that sits on an _Element. This allowed for another major clean up and a huge simplification of the proxy code, since there is only one proxy type left. I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain. According to the test cases, everything works nicely as before and I will give valgrind a run to see if there's anything suspicious left. It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it. Any feedback is appreciated. Stefan

4 8

[lxml-dev] filtering namespaces in findall
by Stefan Behnel 31 May '06

31 May '06

Hi, since I was rewriting the tree iterator anyway, I stumbled over the fact that ElementTree's getiterator() can filter for any tag ("*"), but not for any tag within a namespace ("{namespace}*"). Since this is just a cheap and straight forward little extension to the node filter (and a pretty nice feature to have), I added that to lxml and hope for inclusion in a future ElementTree version also (or maybe a hint why Fredrik assumes this to be inappropriate). Note that it's not currently supported in findall, though, as we use the same implementation as ElementTree there. Stefan

1 1

[lxml-dev] Bug in memory management (I guess ;)
by Johnny deBris 31 May '06

31 May '06

Hi! The past couple of months I kept running into some bug. For some reason the namespace information of certain nodes got lost after moving the nodes from one document to another. I kept trying to isolate the problem, but without success, until this week I found that the reason the information gets lost seems to have to do with the document getting garbage collected even though some nodes are still referred to. With this in mind I managed to write the following demonstration snippet: #-------------------------------------------------------------------------- from lxml import etree s1 = '<a xmlns:a="A:"></a>' tree1 = etree.fromstring(s1) btag = tree1.xpath('//b')[0] del tree1 s2 = '<x />' tree2 = etree.fromstring(s2) tree2.append(btag) # this produces crap... print etree.tostring(tree2) #-------------------------------------------------------------------------- The problem seems to be the 'del' statement: this seems to free the document instance or something, even though there's still a reference to one of its nodes. Serializing the node later on will result in garbage. When the 'del' line is commented out, the XML is produced properly. I hope this is enough information for you guys to track the bug: I'm no C programmer myself, so I don't dare diving into the Pyrex, and also I may have made some errors in my explanation... If you need more information, let me know. Cheers, Guido

2 5

[lxml-dev] Bug with attribute mangling when adding child elements
by Noah Slater 30 May '06

30 May '06

Hello, I noticed today that certain child attribute values are mangled when you try to insert/append child elements onto parent elements. I have written a test script for this as usual. I'm probably missing something... again. :) Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

3 6

[lxml-dev] Difference between ResultTree.__str__ and ResultTree.write
by Noah Slater 30 May '06

30 May '06

Hi, Could we get the difference between ResultTree.__str__ and ResultTree.write properly documented somewhere ([1]?) on the website or via docstrings etc as there only seems to be one reference to it that I can find [2] By the way, the ability to change "output" parameters with write is an excellent addition to this package - such a time saver for me! :) Thanks, Noah [1] http://codespeak.net/lxml/api.html [2] http://codespeak.net/pipermail/lxml-dev/2006-May.txt -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 5

[lxml-dev] credits update: Stefan is the maintainer
by Martijn Faassen 30 May '06

30 May '06

Hi there, I've just updated the CREDITS.txt of lxml to reflect what had become the situation for a long time already: Stefan Behnel is currently the main developer of lxml and its maintainer, and my own role as creator and initial maintainer has diminished somewhat. I'm not disappearing and will remain involved in lxml development for the foreseeable future, but Stefan calls the shots, as he took the responsibility - this is as it should be in open source. I recognized what was going on in december and in personal communication then gave Stefan the keys to the lxml car. He's been doing a great job with it since, and as the 1.0 release is pending we thought we should announce this officially. I personally think Stefan's involvement is an absolutely awesome development. The most obvious reason why this is great is of course Stefan's qualities as a developer and project leader. Less obvious but also very exciting to me is that it makes lxml into a true open source project in the community sense. lxml's future is no longer dependent on a single person, namely myself. Stefan has shown great skill in both sides of managing a successful open source project: the code side and the people side. lxml has now become one of the most powerful XML libraries in the Python world, in the area of feature support as well as in the area of performance. We came a long way in the year and almost-a-half since my initial breakthrough in getting the lxml ball rolling. On the more personal front I also met a few people I'm happy to know now, got to speak about lxml at conferences, and of course got a great XML library out of it too. Thanks to all contributors, and of course especially to Stefan, in making this possible! Regards, Martijn

3 2

[lxml-dev] building the trunk and test failures
by Martijn Faassen 29 May '06

29 May '06

Hi there, I just tried building the trunk: Python 2.4, Pyrex 0.9.4.1, libxml2 2.6.21 The first thing I noticed is that compilation of the C source code (with 'Make') seems to take extremely long in comparison to the past. What changed to cause that? The next thing is that I get some test failures (included below). The api.txt and the extensions.txt test failures both seem to be triggered by the same problem. What could be going on here? Different behavior between different versions of libxml2? Regards, Martijn python test.py -p -v 459/540 ( 85.0%): Doctest: extensions.txt ====================================================================== ERROR: test_docinfo_public (lxml.tests.test_etree.ETreeOnlyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/lxml/tests/test_etree.py", line 456, in test_docinfo_public tree = etree.parse(StringIO(xml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ====================================================================== FAIL: Doctest: api.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/doctest.py", line 2187, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for api.txt File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 0 ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 82, in api.txt Failed example: et = etree.parse(StringIO(xhtml)) Exception raised: Traceback (most recent call last): File "/home/faassen/working/lxml-trunk/src/doctest.py", line 1256, in __run compileflags, 1) in test.globs File "<doctest api.txt[19]>", line 1, in ? et = etree.parse(StringIO(xhtml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 84, in api.txt Failed example: print docinfo.public_id Expected: -//W3C//DTD XHTML 1.0 Transitional//EN Got: -//W3C//DTD HTML 4.0 Transitional//EN ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 86, in api.txt Failed example: print docinfo.system_url Expected: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd Got: http://www.w3.org/TR/REC-html40/loose.dtd ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 88, in api.txt Failed example: docinfo.doctype == doctype_string Expected: True Got: False ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 91, in api.txt Failed example: print docinfo.xml_version Expected: 1.0 Got: None ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 93, in api.txt Failed example: print docinfo.encoding Expected: ascii Got: None ---------------------------------------------------------------------- Ran 459 tests in 1.695s FAILED (failures=1, errors=1)

2 5

[lxml-dev] absolute XPath expressions on Elements
by Stefan Behnel 29 May '06

29 May '06

Hi, I implemented a getpath() method for the ElementTree class that returns an XPath expression for a node. While working out test cases for it, however, I realized that the semantics of evaluating absolute XPath expressions (/...) on elements were not clear at all in the current implementation. ET does not allow absolute expressions in Element.findall() and raises a SyntaxError instead. I think we should do the same for Element.xpath() to prevent mistakes like this: >>> a = etree.Element("a") >>> b = etree.SubElement(a, "b") >>> d0 = etree.SubElement(b, "d") >>> c = etree.SubElement(a, "c") >>> d1 = etree.SubElement(c, "d") >>> d2 = etree.SubElement(c, "d") >>> c.xpath("//d") The reasoning is that Elements do not have a root and therefore no absolute starting point for XPath. Only ElementTrees provide the required semantics, so it's perfectly valid to do this instead: >>> ElementTree(c).xpath("//d") Imagine the case where you have many ElementTrees wrapping various elements in a tree. Which one should be the starting point? Remember that documents and their absolute root node are not exposed through the API. The use case that brought me there is this: >>> tree = etree.ElementTree(c) >>> print tree.getpath(d2) /c/d[2] >>> tree.xpath(tree.getpath(d2)) == [d2] # fails! Intuitively, this should work. However, the current implementation fails here, as it starts searching at 'a' rather than 'c' and thus finds nothing. To fix this, we have to switch the root node during XPath evaluation. Doing this for ElementTree.xpath() is ok, but doing this for Element.xpath() also is impossible, as it breaks relative expressions like "..". So I decided to simply special case XPath expressions starting with '/' and raise exceptions for them. I know that this is not sufficient, as absolute paths can be hidden in things like "*[/a]" or "a|/a". But it's hopefully enough to make users aware and to prevent common mistakes. I also added a note in the documentation that the result of absolute expressions is undefined for Elements, I think that's the right way of saying it. I post this here as there will likely be code that breaks because of this change. I already found two test cases in the test suite that used this. It's just too easy to get wrong, so lxml is better off by raising exceptions where it can than just ignoring this problem. Stefan

3 10

[lxml-dev] ElementTree pretty printing (serialisation)
by Noah Slater 29 May '06

29 May '06

Hello again, My constant spamming of this list has finally paid of and I have something to show for all my questions. I have attached the source of a fairly advanced pretty printing serialiser for ElementTree (and ResultTree) objects. The script (after `chmod 755`) can be called from the command line like so: ./prettyprint.py [DOCUMENT] The script can also be imported and used like so: import sys from lxml import etree import prettyprint document = etree.parse(...) serialiser = prettyprint.ElementTreeSerialiser() serialiser.write(document, sys.stdout) I am new to python, and even newer to lxml and the ElementTree API. As a consequence I may be missing some obvious optimisations. This is only my first stab at this problem and I would love any feed back you care to offer. Is pretty printing something that is on the lxml time line - and if not, would my method demonstrated here interest you from an implementation point of view? Thanks so much for your time. Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 10

[lxml-dev] Best Way To Strip A Namespace
by Noah Slater 29 May '06

29 May '06

Hello, Say I have the following document: <box xmlns="http://example.com/box" xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> After I have parsed it into an element tree, how do I change it in a fool proof manner to loose the root namespace so it becomes: <box xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> This is important for processing my DocBook sources as the XSLT stylesheets cannot handle the namepace on the root element. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 3

[lxml-dev] Threading
by Stefan Behnel 29 May '06

29 May '06

Hi all, I looked a bit at the threading issues and found that there is not much to gain from using threads anyway. Calling code will suffer from the GIL, so it would be up to lxml to release the GIL on long-running operations to benefit from threading. However, releasing the GIL is non-trivial, as most of these operations can call back into the Python API and the interpreter: parsing calls resolvers, as does XSLT. XPath can call extension functions, even the error messages end up in Python objects and lists, etc. So, there are only very few places that could easily be wrapped by ALLOW_THREADS: * everything that traverses trees (which is so fast that the gain may be eaten by the locking overhead) and if error messages can somehow be made working without requiring the GIL: * serialisation to memory or 'real' files * validation (which current does not support custom resolvers) An alternative would be to always create separate Python threads for things like error handling and resolving, but that would require always releasing the GIL whenever these things *might* get called. So, it would be relatively easy to release the GIL in the ElementDepthFirstIterator to have a thread speedup for findall() etc., but everything else will be real work. And since the major overhead is in parsers and serialisers, I don't think it's worth changing this bit just to say "you can gain from threading". Considering this, I changed the FAQ entry on threading to simply state that threading is not supported, with a short explanation that the gain would be marginal without major changes in lxml. Stefan

1 0

[lxml-dev] INPUT: Iniitial FAQ list
by Paul Everitt 29 May '06

29 May '06

Howdy all. This is a thread to collect ideas for an initial FAQ list. Right now we have one on the table: """ How compatible are the findall() and xpath() methods ? findall() don't seem to handle more complicated XPath expressions. Why there is a difference between what they can handle ? """ Other ideas? --Paul

2 2

Re: [lxml-dev] segfault from thread on Windows
by Stefan Behnel 28 May '06

28 May '06

Hi Scott, Scott Haeger wrote: > Here is a small code snippet that demostrates my problem. As you mentioned, I can't reproduce any problems with this under Linux. Couldn't test it under Windows. > The code > works just fine as it is written. However, python crashes if you > comment out the parseFile( ) call from the main thread. I assume that what happens here is that libxml2 misses its thread-local setup. > This seems to > be contrary to what you described about using the default parser from > different threads. > > def parseFile(): > s = "<test><a>entrya</a>entryb</test>" > sio = StringIO.StringIO (s) > tree = etree.parse(sio) > tree.write(sys.stdout) > sys.stdout.flush() > > parseFile() > t = thread.start_new_thread(parseFile, ()) I understand what you did now. The current way do it this would be something like: def parseFileFromThread(): etree.initThread() # needs the current trunk to work thread_local_parser = etree.XMLParser() s = "<test><a>entrya</a>entryb</test>" sio = StringIO.StringIO(s) tree = etree.parse(sio, thread_local_parser) tree.write(sys.stdout) parseFile() t1 = thread.start_new_thread(parseFileFromThread, ()) t2 = thread.start_new_thread(parseFileFromThread, ()) 1.0.beta did not have an initThread() function yet (it had an initThreadLogging function, which did part of what the new initThread does). However, I noticed that it is needed to set up some thread-local libxml2 options. This means that you have to call this method from a newly created thread. It doesn't do any harm to call it from the main thread, though, so you can call the above parseFileFromThread also from the main thread. So, hopefully, your problem is solved in the current trunk. I'd be glad if you could test that with the above modifications to your code. I guess I'll have to write up some documentation on this, but it's good to have some feedback first. Stefan

1 0

[lxml-dev] Segmentation Fault
by Noah Slater 28 May '06

28 May '06

Hello guys, I have found a segfault bug with lxml - attached is a file which reproduces it. Hope this helps. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 1

[lxml-dev] Extending lxml.etree._ElementTree
by Noah Slater 28 May '06

28 May '06

Hello again, I would like to extend the _ElementTree class to include some handy methods - write_xml/write_xhtml/write_html are examples. In addition, this should enable me to return extended ElementTrees instead of ResultTrees after an XSLT transform. What would be the best way to go about this. While I can extend the base class I am struggling to understand how I can replace etree.parse() to return these objects. While I am on the subject, what is the reason for ResultTree objects as a pose to ElementTree. Why make the distinction to the user? Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 2

[lxml-dev] segfault on Windows
by Scott Haeger 28 May '06

28 May '06

I am trying to port an application to Windows. It works with no problems on Linux. On Windows, lxml parses a large file with no problems on the main thread. However, I am getting a segfault when it parses the same file from a spawned thread. Has anyone seen the same thing. Python 2.4.2 lxml 1.0 beta from binary built on May 18 Let me know if there is anything else I can provide. Thanks, Scott Haeger

2 1

[lxml-dev] findall()/xpath() differences ?
by Steve Howe 26 May '06

26 May '06

Hello all, How compatible are the findall() and xpath() methods ? findall() don't seem to handle more complicated XPath expressions. Why there is a difference between what they can handle ? I would expect findall() to be the same as xpath(), but searching from the context node and always returning a list of items, making it compatible with ET's. An example: Python 2.4.3 (#2, Apr 17 2006, 14:29:19) [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> root = etree.XML('''<obj class="list"> ... <obj class="str" value="xxx"/> ... </obj>''') >>> print root.xpath('obj[not(@name)]') [<Element obj at 81daa2c>] >>> print root.findall('obj[not(@name)]') Traceback (most recent call last): File "<stdin>", line 1, in ? File "etree.pyx", line 937, in etree._Element.findall File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 193, in findall return _compile(path).findall(element) File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 171, in _compile p = Path(path) File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 87, in __init__ raise SyntaxError( SyntaxError: expected path separator ([) -- Best regards, Steve mailto:howe@carcass.dhs.org

6 13

[lxml-dev] XMLFormatter revisited
by Stefan Behnel 22 May '06

22 May '06

Hi all, coming back to what I proposed a while ago, we currently have this: class XMLParser: def __init__(self, **options): self.options = options class HTMLParser: ... doc = ET.parse(source, parser=XMLParser(configuration)) At the time, I brought up an equivalent for output formatting: class XMLFormatter: ... That would give us a nice, symmetric API for input and output options. And it allows you to use sublasses to provide different formats: class XMLPrettyPrinter(XMLFormatter): def __init__(self): self._pretty_print = True class XHTMLFormatter(XMLFormatter): def __init__(self): self._xhtml = True xml = ET.tostring(element, formatter=XMLPrettyPrinter()) ET.parse("myfile.xhtml").write("out.xhtml", formatter=XHTMLFormatter()) After Noah's latest approach in that direction, I think we should adopt this API for 1.0. A version 1.0 is supposed to provide a stable and somewhat future-proof API, and adding keyword arguments to various API functions all the time is not quite what I call future-proof. One reason is that you can't test for the availability of keyword arguments, so adding features at that level is difficult to handle for code that wants to support them as an option. So, I propose to replace the current pretty_print keyword (which only appeared in the beta version anyway) with a new XMLPrettyPrinter class and to provide new features preferably at a subclass level of XMLFormatter (e.g. XHTMLFormatter, etc.). An alternative name would be XMLSerializer, maybe that's more general. We could also leave the pretty_print keyword in as a shortcut, but that would obviously make things a bit more complicated internally. Maybe pretty printing will just become a general keyword of the XMLFormatter class, I guess that would make sense. I will have to check how to make these classes nicely usable internally, but that's a minor problem. We already use a lot of xmlBuffer code, so that should give us a common ground for this API. If there are no objections, I'll start getting my hands on this next week, so if anyone has an opinion on this, please speak up soon. Stefan

1 1

Re: [lxml-dev] Can't change text in comments
by Stefan Behnel 22 May '06

22 May '06

Hi Noah, Noah Slater wrote: > I have noticed that setting the 'text' of a Comment does not seem to > change anything when the Comment is serialised. Oh well, /that/ was an old bug you found there. The code in that area dates from SVN revision 8082 - we're at 27509 now. Changing the text of a comment was never implemented, although ET supports it. The reason (I believe) why Martijn did not implement it at the time (unless he just forgot) is that changing comment texts in libxml2 is a bit tricky. There are no libxml2 functions for doing that and it requires checking the document dictionary to see if the original comment string isn't still in use anywhere else. Great place to add some more segfaults. I implemented this and actually found a couple of other bugs that I fixed (including the test cases that relied on these bugs). What bothers me is that lxml is consistent with ElementTree in that it adds whitespace around comment texts. I have no idea why ElementTree does that in the first place. AFAICT, this happily breaks things like SSI. Maybe Fredrik can enlighten us here? Stefan

4 4

[lxml-dev] lxml crash
by Steve Howe 22 May '06

22 May '06

Hello all, I know that the code is wring, but this crash is happening on my FreeBSD 6.1 and Win XP systems, with lxml 0.9.2: [carcass@/home/howe] python Python 2.4.3 (#2, Apr 17 2006, 14:29:19) [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> a = etree.Element('a') >>> a.append(None) Exception exceptions.TypeError: 'Argument must not be None.' in 'etree._raiseIfNone' ignored zsh: segmentation fault python It should be more friendly by just displaying the message and not crashing, right ? By the way, is there any way to get the lxml version from code ? Some module '__version__' attribute ? Shouldn't it be there ? Something like: lxml.__version__ -- Best regards, Steve mailto:howe@carcass.dhs.org

5 10

Re: [lxml-dev] XMl Processing Instructions
by Stefan Behnel 20 May '06

20 May '06

Hi Noah, Noah Slater wrote: > If I understand your email correctly the behaviour I describe is not > intentional and will be fixed shortly? It has been fixed in the developer version, but there is not yet a release that has the fix. > Just to clarify - I can parse any file, but when I serialise I loose > any processing instructions. This includes the <?xml ... ?> > declaration. I understood that. > This also happens with the ResultTree (?) when I transform using XSLT. > > As an example, I use the DocBook XSLT stylesheets to transform DocBook > XML. This can often set various things up with the processing > instructions - character encoding being the most important. > > When I perform these transformations using ElementTree I loose this > information. > > As a work around at the moment I am using lxml.etree to do the > transformations using UTF-8 as the encoding. I am then using libxml2 > and libxslt to transform the serialized document bytestream a second > time with the only operation being converting from UTF-8 to another > (variable) character encoding. > > This feels quite hackish Well, it /is/ hackish. :) > - and to be honest the whole point of me > moving to lxml was because I find the libxml2 and libxslt bindings > hateful. That's one of the main reasons that drive us in writing lxml - right after users saying "thank you, it's great and helps us do stuff!" :) > To summarise, in an ideal world I would like to be able to transform a > document using XSLT specifying an encoding at transformation time and > have the ResultTree serialise with all processing instructions intact. > Additionally I would like to be able to access these programmatically > - which I don't think is possible at the moment. That's also a feature of the developer version that will eventually become lxml 1.0. See the 'docinfo' feature described here: http://codespeak.net/svn/lxml/trunk/doc/api.txt See here for a complete list of changes since 0.9.2: http://codespeak.net/svn/lxml/trunk/CHANGES.txt > I hope all this makes sense. It does. Thanks for the report. Please use the current SVN trunk version for now, lxml 1.0 is expected to be released this month. See http://codespeak.net/lxml/installation.html and the bottom of the download section in http://codespeak.net/lxml/ on how to check it out of the subversion repository and compile it. Stefan

2 2

[lxml-dev] XMl Processing Instructions
by Noah Slater 20 May '06

20 May '06

Hello, Why I use the write method of the ElementTree class why does it strip out the XML processing insturctions? I would like my documents to start with the processing instruction so I can specify encodings other than UTF-8. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 5

[lxml-dev] Encoding Issues
by Noah Slater 20 May '06

20 May '06

Hello, I notice that I can pass an encoding parameter to an elementtree's write method. Two things: 1) When lxml supports xml processing instructions, will these be updated accordingly? I notice that the DocBook XSL will update meta-equiv HTML elements according to the value I pass in here - which is good, except for the following... 2) Why does this let me make up encodings such as "Noahs-Cool-Encoding" without raising an exception. If I was to take a guess I would imagine this falls back on UTF-8 but these is a bug IMO. In case you are wondering I am in the process of writing an XMl based HTTP publishing framework and lxml will be sitting at the very core of how I handle document conversion/manipulation/transformation. The problem with encoding lies within my content negotiation module which will (in a pythonic manner IMO) try to transform the document with each encoding specified in the Accept-Charset header of the client request. If the transform raises an exception we move on to the next one. My previous way of working would raise an exception if I tried an encoding it didn't recognise. While I am aware of how to look up encoding names using the python standard library - I am not sure if this correlates 100% with lxml and additionally I don't feel this extra step should be necessary. Thanks so much. Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 2

[lxml-dev] lxml/ET/cET performance comparison in doc/performance.txt
by Stefan Behnel 19 May '06

19 May '06

Hi all, I've re-run the benchmark suite (which works nicely now) and written up some documentation about the performance of lxml in comparison to ET/cET. It's in http://codespeak.net/svn/lxml/trunk/doc/performance.txt If anyone is interested in extending the benchmark suite over other parts of the API or in adding to the comparison, I'd be glad to see some input. :) Have fun, Stefan

1 0

[lxml-dev] Bugs in 1.0.beta
by Olivier Grisel 18 May '06

18 May '06

Hi list, On my box (Ubuntu Linux Dapper Drake on i686 / python2.4) all tests pass but running the bench scripts yields some problems: >>> lxml.etree.LIBXML_COMPILED_VERSION (2, 6, 24) >>> lxml.etree.LIBXSLT_COMPILED_VERSION (1, 1, 15) xpath_extensions_old and xslt_extensions_old trigger the following exceptions: lxe: xpath_extensions_old (-- T1 ) Traceback (most recent call last): File "bench.py", line 652, in ? result = run_bench(bench, *benchmark_setup) File "bench.py", line 606, in run_bench method_call(*args) File "bench.py", line 453, in bench_xpath_extensions_old xpath = self.etree.XPath("child(.)", extensions=extensions) File "xpath.pxi", line 166, in etree.XPath.__init__ File "xpath.pxi", line 53, in etree.XPathEvaluatorBase.__init__ File "xpath.pxi", line 17, in etree._XPathContext.__init__ File "extensions.pxi", line 45, in etree._BaseContext.__init__ TypeError: unindexable object lxe: xslt_extensions_old (-- T1 ) Traceback (most recent call last): File "bench.py", line 652, in ? result = run_bench(bench, *benchmark_setup) File "bench.py", line 606, in run_bench method_call(*args) File "bench.py", line 479, in bench_xslt_extensions_old transform = self.etree.XSLT(tree, extensions) File "xslt.pxi", line 265, in etree.XSLT.__init__ File "xslt.pxi", line 189, in etree._XSLTContext.__init__ File "extensions.pxi", line 45, in etree._BaseContext.__init__ ValueError: unpack sequence of wrong size And append_element and replace_children triggers segmentation faults. The first one crashed after child <Element {bcd}zzzzz at -48285674>. I don't have time to dig further right now. I'll have more time tonight. Best, -- Olivier

2 2

[lxml-dev] lxml 1.0.beta is on cheeseshop
by Stefan Behnel 18 May '06

18 May '06

Hi all, lxml 1.0.beta is available from cheeseshop. This is expected to be the only beta release before the glorious 1.0 will arrive on June 1st (this year). http://cheeseshop.python.org/pypi/lxml The changelog is long and the time-to-market for 1.0 is short, so, as usual, we are happy to receive your bug reports. :) Have fun, Stefan

1 0

[lxml-dev] Lxml 1.0 will be released on Thursday, June 1st
by Stefan Behnel 18 May '06

18 May '06

Hallo everyone, just to keep us from pushing back release dates, Martijn and I have fixed the release date of lxml 1.0 to June 1st. The (long) list of changes for 1.0 is here: http://codespeak.net/svn/lxml/trunk/CHANGES.txt This means for you: If you want a bug-free release ready to install, then come and help us by testing the trunk and reporting any bugs you can find. http://codespeak.net/svn/lxml/trunk Build instructions are here: http://codespeak.net/lxml/build.html It also really helps us if you read the docs and report any difficulties you encounter. The in-development documentation is here: http://codespeak.net/svn/lxml/trunk/doc/ especially http://codespeak.net/svn/lxml/trunk/doc/api.txt http://codespeak.net/svn/lxml/trunk/doc/resolvers.txt and a little bit of http://codespeak.net/svn/lxml/trunk/doc/extensions.txt Any help is appreciated to make lxml 1.0 the best XML tool for Python - ever. :) Have fun, Stefan

4 14

Re: [lxml-dev] What are the equivalents of nowrite, nomkdir options
by Stefan Behnel 18 May '06

18 May '06

Hi Noah, Noah Slater wrote: >> Noah, I'd be glad if you could test it and report back if it works as >> expected. I do not know if the directory creation business works now >> (i.e. if >> directories /are/ created). > > Hi, sorry I can't test until I have a .deb package to install - so I > am guessing I will have to wait until your changes arrive in Debian > unstable. That won't be before 1.0 then, I assume. It's actually not that hard to build Debian packages once they are part of Debian. Get the source .deb via apt and unpack it. Then you can replace the included lxml sources with the current SVN trunk (you may have to make a tgz from it), make sure the Debian package description has the version number you want and then rebuild it (IIRC, the command for that is "dpkg-rebuild" or something in that line). I don't have a Debian system to test, so I can't tell you what exactly you have to do. But there's an even simpler way. You can check out lxml from SVN and build it in-place in the source directory (see doc/build.txt on how to do that). To use it in your program, you can call Python like this: PYTHONPATH=/path/to/lxml-dir/src python myprogram.py To make sure it's the right version, use PYTHONPATH=/path/to/lxml-dir/src python -c \ 'import lxml.etree ; print lxml.etree.__version__' (the '/' makes it one line when you copy and paste it) Stefan

3 3

[lxml-dev] What are the equivalents of nowrite, nomkdir options
by Noah Slater 17 May '06

17 May '06

Hello all, Firstly thanks for such a great library. For a python module I am worryingly excited reading through the documentation. I have had a play around with etree.* and have almost replace 50% of the code in my project thus far to use this - but one concern remains: I cannot figure out from either the docs, snippets of source files I have read or google how I am meant to prevent XSLT from writing to the file system. Using xsltproc from the command line I can use the "--nowrite" and "--nomkdir" switches and would ideally like to be able to replicate this functionality. Thanks so much, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

4 7

[lxml-dev] new "doc/build.txt" on building lxml from sources
by Stefan Behnel 17 May '06

17 May '06

Hi all, I added a "doc/build.txt" apart from the simplified "INSTALL.txt" to help interested users build lxml from sources (and to prevent normal distrib users from caring about it). It also contains David's procedure on building lxml statically on Windows. http://codespeak.net/svn/lxml/trunk/doc/build.txt It's both in the trunk and the 0.9 branch, so, Martijn, please update the web site from the branch. Regards, Stefan

2 1

[lxml-dev] New CSS style for the lxml website @ codespeak
by Olivier Grisel 16 May '06

16 May '06

Hi list, As I found the current style a bit "too big" I refactored the css to get this: http://champiland.homelinux.net/lxml-newstyle/ If you like it I can check that in the svn. I enclose a patch to list the changes. To Stefan: your work on mail.txt has been done only in branch/lxml-0.9.x and not in trunk. Is it intentional? + it would be nice if your changes could went online. Who is in charge of the publishing process? Regards, -- Olivier Index: style.css =================================================================== --- style.css (revision 27205) +++ style.css (working copy) @@ -1,33 +1,62 @@ body { + /* CSS Hack for IE that does not respect the "margin: auto" rule at the + * document level */ + text-align: center; + padding: 1em; +} + +div.document { + width: 45em; + font: 13px Arial, Verdana, Helvetica, sans-serif; + margin: 1em auto 1em auto; + background-color: white; + color: #222; + text-align: left; +} + +h1.title { background: url(http://codespeak.net/img/codespeak1b.png) no-repeat; - font: 120% Arial, Verdana, Helvetica, sans-serif; - border: 0; - margin: 0.5em 0em 0.5em 0.5em; - padding: 0 0 0 145px; + padding: 20px 0 0 180px; + height: 60px; + font-size: 200%; } -a { - text-decoration: underline; - background-color: transparent; +h1, h2, h3 { + color: #333; + font-weight: bold; } -p { - /*margin: 0.5em 0em 1em 0em;*/ - text-align: left; - line-height: 1.5em; - margin: 0.5em 0em 0em 0em; +h1 { + font-size: 120%; } -p a { - text-decoration: underline; +h2 { + font-size: 110%; } +h3 { + font-size: 105%; +} -p a:active { - color: Red; +a, a:visited { background-color: transparent; + font-weight: bold; + color: Black; + text-decoration: none; } +a:active { + color: Red; + text-decoration: underline; +} + +p { + /*margin: 0.5em 0em 1em 0em;*/ + text-align: justify; + line-height: 1.5em; + margin: 0.5em 0em 0em 0em; +} + hr { clear: both; height: 1px; @@ -35,10 +64,8 @@ background-color: transparent; } - -ul { +ul { line-height: 1.5em; - /*list-style-image: url("bullet.gif"); */ margin-left: 1em; } @@ -47,28 +74,21 @@ margin-left: 0em; } -ul a, ol a { - text-decoration: underline; -} - blockquote { font-family: Times, "Times New Roman", serif; font-style: italic; - font-size: 120%; } code { - font-size: 120%; color: Black; - /*background-color: #dee7ec;*/ background-color: #cccccc; + font-family: Courier, monospace; } pre { - font-size: 120%; - padding: 1em; + padding: 0.5em; border: 1px solid #8cacbb; color: Black; - background-color: #dee7ec; background-color: #cccccc; + font-family: Courier, monospace; } Index: publish.py =================================================================== --- publish.py (revision 27205) +++ publish.py (working copy) @@ -1,9 +1,12 @@ -import os, sys +import os, shutil, sys def publish(dirname, lxml_path, release): if not os.path.exists(dirname): os.mkdir(dirname) - stylesheet_url = 'http://codespeak.net/lxml/style.css' + stylesheet_url = 'style.css' + + shutil.copy(stylesheet_url, dirname) + for name in ['main.txt', 'intro.txt', 'api.txt', 'compatibility.txt', 'extensions.txt', 'namespace_extensions.txt', 'sax.txt']: path = os.path.join(lxml_path, 'doc', name) @@ -22,10 +25,10 @@ os.path.join(dirname, 'index.html')) def rest2html(source_path, dest_path, stylesheet_url): - - command = ('rest2html --stylesheet=%s %s > %s' % + + command = ('rest2html --stylesheet=%s --link-stylesheet %s > %s' % (stylesheet_url, source_path, dest_path)) os.system(command) - + if __name__ == '__main__': publish(sys.argv[1], sys.argv[2], sys.argv[3])

3 9

[lxml-dev] Building lxml on windows: a guide.
by David Sankel 12 May '06

12 May '06

Note: This is my first post on this list. There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down). Unfortunately, there wasn't enough information to make the process of building on windows easy. I wrote this guide to solve both these problems. I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance. I also have a windows installer created for lxml 0.9.1 that is staticly built with iconv-1.9.1, libxml2 2.6.23, libxslt 1.1.15, and zlib 1.2.3. If there is some ftp site or something I can upload it to, in order to give wide access to it, please let me know what to do. David Sankel Building lxml on windows =================== First you'll need to download the latest version of all the required files. Download them all to the same directory. * libxml: Availible from http://codespeak.net/lxml/ * iconv, libxml2, libxslt, and zlib are all availible from xmlsoft.org. The place to go on the ftp site is ftp://xmlsoft.org/libxml2/win32. Your directory should now have something like the following files in it: iconv-1.9.1.win32.zip libxml2-2.6.23.win32.zip libxslt-1.1.15.win32.zip lxml-0.9.1.tgz zlib-1.2.3.win32.zip Now extract each of those files in the _same_ directory. Now you should have something like this: iconv-1.9.1.win32/ iconv-1.9.1.win32.zip libxml2-2.6.23.win32/ libxml2-2.6.23.win32.zip libxslt-1.1.15.win32/ libxslt-1.1.15.win32.zip lxml-0.9.1/ lxml-0.9.1.tgz zlib-1.2.3.win32/ zlib-1.2.3.win32.zip Go to the lxml-0.9.1 directory and edit the Makefile. There should be a section that looks like this:: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = ['-w'] + flags('xslt-config --cflags'), extra_link_args = flags('xslt-config --libs') )], Change it to this (Warning: make sure you are using version numbers that correspond to your downloads) (Note: the _a portion of the libraries means that we are statically linking. If you want to use dlls(why?), link to the dll version of the libraries):: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = [ "-w", "-I..\\libxml2-2.6.23.win32\\include", "-I..\\libxslt-1.1.15.win32\\include", "-I..\\zlib-1.2.3.win32\\include", "-I..\\iconv-1.9.1.win32\\include" ], extra_link_args = [ "..\\libxml2-2.6.23.win32\\lib\\libxml2_a.lib", "..\\libxslt-1.1.15.win32\\lib\\libxslt_a.lib", "..\\zlib-1.2.3.win32\\lib\\zlib.lib", "..\\iconv-1.9.1.win32\\lib\\iconv_a.lib" ] )], Now you should be able to use setup.py and everything should work well. "python setup.py bdist_wininst" will create a windows installer in the pkg directory.

4 12

[lxml-dev] Uche's OT-benchmark on lxml and (c)ElementTree
by Stefan Behnel 12 May '06

12 May '06

Hi all, I just ran a slight variant (doesn't print, builds list) of Uche's OT benchmark on * ElementTree 1.2.6 * cElementTree 1.0.5 * lxml (trunk/SVN 27065, timings for 0.9.2 are similar) The code I used is attached, it runs against the 3.3M ot.xml file from http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip Here's the (somewhat bogus) output on my machine (bench, result, time): bench_ET 120 1.59 bench_cET 120 0.31 bench_lxml_findall 120 0.32 bench_lxml_xpath 120 0.33 bench_lxml_xpath_all 120 0.26 I should note that lxml's findall() uses ElementTree's ElementPath implementation. Here are the real numbers from timeit: # python -m timeit -s "from otbench import *" "bench_ET()" 10 loops, best of 3: 1.36 sec per loop # python -m timeit -s "from otbench import *" "bench_cET()" 10 loops, best of 3: 260 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_findall()" 10 loops, best of 3: 295 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_xpath()" 10 loops, best of 3: 261 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_xpath_all()" 10 loops, best of 3: 214 msec per loop I think it's pretty interesting how close the timings of cET.findall() and lxml.xpath() are. Regards, Stefan

3 12

[lxml-dev] Python unicode string support in lxml
by Stefan Behnel 12 May '06

12 May '06

Hi all, I had a discussion with Fredrik lately and it convinced me (though not Fredrik) that it would be a good idea to improve the support for Python unicode strings in lxml.etree. I think that unicode strings are the most comfortable way of doing XML I/O from/to strings in Python, so I added support for simply calling unicode() on XML nodes and ElementTrees. It behaves just like tostring(), but always returns Python unicode strings: >>> from lxml import etree >>> uxml = u'<test> \uf8d1 + \uf8d2 </test>' >>> root = etree.XML(uxml) >>> unicode(root) u'<test> \uf8d1 + \uf8d2 </test>' >>> el = etree.Element("test") >>> unicode(el) u'<test/>' >>> subel = etree.SubElement(el, "subtest") >>> unicode(el) u'<test><subtest/></test>' >>> unicode( etree.ElementTree(el) ) u'<test><subtest/></test>' Note that ElementTree does not support this at all. It will raise a parser exception in the XML() call in the second line and return the same generic strings for unicode() as it does for str(). There is a longer doctest in http://codespeak.net/svn/lxml/trunk/doc/api.txt that explains this in more detail. As usual: any comments appreciated. Stefan

6 59

[lxml-dev] tounicode(), again
by Stefan Behnel 11 May '06

11 May '06

Hi all, we had a lengthy discussion yesterday and I guess we found a few use cases where tounicode() makes sense and a few counter-arguments why it might not be a good idea to expose that API at a similarly visible place as tostring(). I'm still convinced that it's a good idea to have that API, but as one of the arguments was that "people who don't understand unicode" (PeWDUUs) would be more likely to write broken code, I added this paragraph to api.txt, in the section that describes the unicode support of lxml. """ Note that the unicode strings returned by ``tounicode()`` never have an XML declaration and therefore do not specify an encoding. This makes it possible to pass them back into the lxml parsers. However, you may have to add a declaration yourself if you want to serialize such a unicode string to a byte stream later. In contrast, the ``tostring()`` function automatically adds a declaration as needed that reflects the encoding of the returned byte string. """ I hope that makes it clear enough for PeWDUUs what the advantage of using tostring() over tounicode() is and that you have to take care what you do with unicode strings. So, I propose leaving the API (and implementation) just as it is now. Regards, Stefan

2 1

[lxml-dev] lxml 0.9.2 released
by Stefan Behnel 10 May '06

10 May '06

Hello everyone, I just uploaded another bug fix release of the 0.9 branch to the Cheese shop. http://cheeseshop.python.org/pypi/lxml http://cheeseshop.python.org/packages/source/l/lxml/lxml-0.9.2.tar.gz This will (hopefully) be the last 0.9.x release before 1.0. It fixes a number of more or less annoying bugs and adds a number of smaller features. All the major, new, fancy features will go into the 1.0 release. Regards, Stefan Features added * Speedup for Element.makeelement(): the new element now reuses the original libxml2 document instead of creating a new empty one * Speedup for reversed() iteration over element children (Py2.4+ only) * ElementTree compatible QName class * RelaxNG and XMLSchema now accept any Element, not only ElementTrees Bugs fixed * str(xslt_result) was broken for XSLT output other than UTF-8 * Memory leak if write_c14n fails to write the file after conversion * Crash in XMLSchema and RelaxNG when passing non-schema documents * Memory leak in RelaxNG() when RelaxNGParseError is raised

1 0

[lxml-dev] The difference between str(xslt_result) and xslt_result.write()
by Matt Patterson 10 May '06

10 May '06

Hi, Is there a difference between the output from lxml.etree._XSLTResultTree.__str__() and lxml.etree._XSLTResultTree.write()? I'm trying to chase some output whitespace issues around, and I'm wondering if there's a difference in how serialisation is handled. Also, str(_XSLTResultTree) is failing for me with errors like "'ascii' codec can't encode character u'\xa9' in position 1608: ordinal not in range(128)" because I have unicode characters in my documents... lxml.etree._XSLTResultTree doesn't define a __unicode__ method so I can't use a unicode coercion to UTF-8 or something like that... All in all though, I'm really enjoying lxml: I spent a long time working with libxml & lbxslt's standard python interfaces and they much more of a pain than lxml! Thanks, Matt -- Matt Patterson | Design & Code <matt at reprocessed org> | http://www.reprocessed.org/

2 2

[lxml-dev] I/O benchmarks
by Stefan Behnel 08 May '06

08 May '06

Hi everyone, have you ever wondered why you should use lxml instead of cElementTree? Here is why. :) Stefan Preparing test suites and trees ... Running benchmark on lxe, ET, cET Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.1180 0.1158 0.1153 0.1178 0.1157 0.1173 T2: 0.1186 0.1202 0.1207 0.1233 0.1236 0.1250 T3: 0.0323 0.0252 0.0250 0.0489 0.0495 0.0492 T4: 0.0005 0.0005 0.0005 0.0010 0.0010 0.0010 ET : -- S- U- -A SA UA T1: 0.2305 0.2887 0.2193 0.2662 0.2872 0.2259 T2: 0.3038 0.3442 0.2823 0.3140 0.2364 0.4051 T3: 0.0534 0.0572 0.0523 0.0583 0.0553 0.0829 T4: 0.0010 0.0008 0.0007 0.0009 0.0008 0.0008 cET: -- S- U- -A SA UA T1: 0.0369 0.0353 0.0371 0.0341 0.0354 0.0345 T2: 0.0370 0.0364 0.0361 0.0371 0.0369 0.0358 T3: 0.0090 0.0091 0.0090 0.0125 0.0174 0.0235 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 lxe: tostring_utf16 (S- T1 ) 25.0344 24.7066 24.6755 msec/pass, best: 24.6755 ET : tostring_utf16 (S- T1 ) 715.0391 668.6494 668.2270 msec/pass, best: 668.2270 cET: tostring_utf16 (S- T1 ) 631.4080 634.0269 629.3236 msec/pass, best: 629.3236 lxe: tostring_utf16 (U- T1 ) 25.3342 25.4860 25.1003 msec/pass, best: 25.1003 ET : tostring_utf16 (U- T1 ) 666.5964 670.9383 664.8268 msec/pass, best: 664.8268 cET: tostring_utf16 (U- T1 ) 628.6623 628.9270 636.1863 msec/pass, best: 628.6623 lxe: tostring_utf16 (S- T2 ) 38.5328 27.5526 28.3839 msec/pass, best: 27.5526 ET : tostring_utf16 (S- T2 ) 696.8423 697.2251 698.0101 msec/pass, best: 696.8423 cET: tostring_utf16 (S- T2 ) 655.5692 652.2847 653.9454 msec/pass, best: 652.2847 lxe: tostring_utf16 (U- T2 ) 26.2363 26.8365 26.6303 msec/pass, best: 26.2363 ET : tostring_utf16 (U- T2 ) 698.3589 697.7249 698.7069 msec/pass, best: 697.7249 cET: tostring_utf16 (U- T2 ) 652.6794 652.6194 652.4247 msec/pass, best: 652.4247 lxe: tostring_utf16 (S- T3 ) 2.9159 2.9798 3.0483 msec/pass, best: 2.9159 ET : tostring_utf16 (S- T3 ) 90.1106 90.2723 90.6772 msec/pass, best: 90.1106 cET: tostring_utf16 (S- T3 ) 75.9803 75.7933 75.6844 msec/pass, best: 75.6844 lxe: tostring_utf16 (U- T3 ) 2.9326 2.9501 3.8947 msec/pass, best: 2.9326 ET : tostring_utf16 (U- T3 ) 90.8508 90.2919 90.4195 msec/pass, best: 90.2919 cET: tostring_utf16 (U- T3 ) 75.8780 75.6118 75.6281 msec/pass, best: 75.6118 lxe: tostring_utf16 (S- T4 ) 0.1184 0.1246 0.1174 msec/pass, best: 0.1174 ET : tostring_utf16 (S- T4 ) 4.0772 4.1340 4.0832 msec/pass, best: 4.0772 cET: tostring_utf16 (S- T4 ) 8.6555 3.7591 3.8131 msec/pass, best: 3.7591 lxe: tostring_utf16 (U- T4 ) 0.1150 0.1142 0.1470 msec/pass, best: 0.1142 ET : tostring_utf16 (U- T4 ) 4.1377 4.1255 4.0694 msec/pass, best: 4.0694 cET: tostring_utf16 (U- T4 ) 8.5306 3.7888 3.8399 msec/pass, best: 3.7888 lxe: tostring_utf8 (S- T1 ) 20.1342 20.8913 22.0570 msec/pass, best: 20.1342 ET : tostring_utf8 (S- T1 ) 658.5292 659.8311 659.2222 msec/pass, best: 658.5292 cET: tostring_utf8 (S- T1 ) 613.7725 616.7276 615.3487 msec/pass, best: 613.7725 lxe: tostring_utf8 (U- T1 ) 22.3571 21.6062 22.3468 msec/pass, best: 21.6062 ET : tostring_utf8 (U- T1 ) 658.4980 658.8312 659.7990 msec/pass, best: 658.4980 cET: tostring_utf8 (U- T1 ) 619.8824 620.7493 618.3270 msec/pass, best: 618.3270 lxe: tostring_utf8 (S- T2 ) 22.8893 23.1579 22.3014 msec/pass, best: 22.3014 ET : tostring_utf8 (S- T2 ) 695.0778 686.7107 686.9514 msec/pass, best: 686.7107 cET: tostring_utf8 (S- T2 ) 644.8464 645.2512 645.4735 msec/pass, best: 644.8464 lxe: tostring_utf8 (U- T2 ) 21.8574 21.5462 21.9939 msec/pass, best: 21.5462 ET : tostring_utf8 (U- T2 ) 689.5708 689.1687 685.5371 msec/pass, best: 685.5371 cET: tostring_utf8 (U- T2 ) 643.3239 644.8895 641.5675 msec/pass, best: 641.5675 lxe: tostring_utf8 (S- T3 ) 2.3862 2.2847 2.3411 msec/pass, best: 2.2847 ET : tostring_utf8 (S- T3 ) 91.7382 89.4661 90.1840 msec/pass, best: 89.4661 cET: tostring_utf8 (S- T3 ) 76.4675 74.7183 74.7682 msec/pass, best: 74.7183 lxe: tostring_utf8 (U- T3 ) 2.3286 2.4161 2.3287 msec/pass, best: 2.3286 ET : tostring_utf8 (U- T3 ) 89.9778 92.0483 91.0223 msec/pass, best: 89.9778 cET: tostring_utf8 (U- T3 ) 74.8817 74.8538 74.9363 msec/pass, best: 74.8538 lxe: tostring_utf8 (S- T4 ) 0.1028 0.1006 0.1058 msec/pass, best: 0.1006 ET : tostring_utf8 (S- T4 ) 8.9541 4.1052 4.0422 msec/pass, best: 4.0422 cET: tostring_utf8 (S- T4 ) 8.5588 3.7166 3.7450 msec/pass, best: 3.7166 lxe: tostring_utf8 (U- T4 ) 0.1178 0.1000 0.1040 msec/pass, best: 0.1000 ET : tostring_utf8 (U- T4 ) 4.0836 4.0845 4.0257 msec/pass, best: 4.0257 cET: tostring_utf8 (U- T4 ) 8.5069 3.7946 3.7505 msec/pass, best: 3.7505 lxe: tostring_utf8_unicode_XML (S- T1 ) 217.3858 194.6370 196.9607 msec/pass, best: 194.6370 ET : tostring_utf8_unicode_XML (S- T1 ) 1044.2196 983.0496 1006.1930 msec/pass, best: 983.0496 cET: tostring_utf8_unicode_XML (S- T1 ) 684.9565 679.0308 674.1449 msec/pass, best: 674.1449 lxe: tostring_utf8_unicode_XML (U- T1 ) 203.8817 203.9280 200.4173 msec/pass, best: 200.4173 ET : tostring_utf8_unicode_XML (U- T1 ) 981.1272 977.5449 978.9895 msec/pass, best: 977.5449 cET: tostring_utf8_unicode_XML (U- T1 ) 653.3920 655.3704 651.0385 msec/pass, best: 651.0385 lxe: tostring_utf8_unicode_XML (S- T2 ) 208.8770 211.5672 210.4380 msec/pass, best: 208.8770 ET : tostring_utf8_unicode_XML (S- T2 ) 1021.3773 1020.3279 1041.1114 msec/pass, best: 1020.3279 cET: tostring_utf8_unicode_XML (S- T2 ) 688.2031 681.9181 680.2590 msec/pass, best: 680.2590 lxe: tostring_utf8_unicode_XML (U- T2 ) 209.5904 210.3374 209.5674 msec/pass, best: 209.5674 ET : tostring_utf8_unicode_XML (U- T2 ) 1027.9991 1022.0539 1022.0318 msec/pass, best: 1022.0318 cET: tostring_utf8_unicode_XML (U- T2 ) 681.4484 678.8596 680.9469 msec/pass, best: 678.8596 lxe: tostring_utf8_unicode_XML (S- T3 ) 12.5315 12.0217 11.9403 msec/pass, best: 11.9403 ET : tostring_utf8_unicode_XML (S- T3 ) 202.9581 203.0447 204.2288 msec/pass, best: 202.9581 cET: tostring_utf8_unicode_XML (S- T3 ) 82.3561 82.4286 82.1372 msec/pass, best: 82.1372 lxe: tostring_utf8_unicode_XML (U- T3 ) 11.6045 11.8548 11.7764 msec/pass, best: 11.6045 ET : tostring_utf8_unicode_XML (U- T3 ) 201.2770 201.0283 202.3813 msec/pass, best: 201.0283 cET: tostring_utf8_unicode_XML (U- T3 ) 82.3942 82.6146 82.3579 msec/pass, best: 82.3579 lxe: tostring_utf8_unicode_XML (S- T4 ) 5.3620 0.4832 0.5012 msec/pass, best: 0.4832 ET : tostring_utf8_unicode_XML (S- T4 ) 6.4547 6.3800 6.3898 msec/pass, best: 6.3800 cET: tostring_utf8_unicode_XML (S- T4 ) 8.8447 3.9611 4.0390 msec/pass, best: 3.9611 lxe: tostring_utf8_unicode_XML (U- T4 ) 5.2781 0.4689 0.4798 msec/pass, best: 0.4689 ET : tostring_utf8_unicode_XML (U- T4 ) 6.5955 6.4135 6.4196 msec/pass, best: 6.4135 cET: tostring_utf8_unicode_XML (U- T4 ) 8.7307 4.0647 4.0202 msec/pass, best: 4.0202 lxe: write_utf8_parse_stringIO (S- T1 ) 198.0141 188.3298 190.1872 msec/pass, best: 188.3298 ET : write_utf8_parse_stringIO (S- T1 ) 1143.8117 1176.7961 1152.1253 msec/pass, best: 1143.8117 cET: write_utf8_parse_stringIO (S- T1 ) 814.6583 847.3877 810.7611 msec/pass, best: 810.7611 lxe: write_utf8_parse_stringIO (U- T1 ) 194.5397 195.0845 194.7964 msec/pass, best: 194.5397 ET : write_utf8_parse_stringIO (U- T1 ) 1153.3167 1142.9091 1146.2344 msec/pass, best: 1142.9091 cET: write_utf8_parse_stringIO (U- T1 ) 810.8662 808.3027 811.9326 msec/pass, best: 808.3027 lxe: write_utf8_parse_stringIO (S- T2 ) 205.9781 202.8390 202.2034 msec/pass, best: 202.2034 ET : write_utf8_parse_stringIO (S- T2 ) 1195.9166 1195.7337 1193.5966 msec/pass, best: 1193.5966 cET: write_utf8_parse_stringIO (S- T2 ) 846.5763 851.8866 848.6367 msec/pass, best: 846.5763 lxe: write_utf8_parse_stringIO (U- T2 ) 203.5371 202.8143 204.4644 msec/pass, best: 202.8143 ET : write_utf8_parse_stringIO (U- T2 ) 1218.4967 1245.6663 1255.3381 msec/pass, best: 1218.4967 cET: write_utf8_parse_stringIO (U- T2 ) 858.2268 849.1432 858.5724 msec/pass, best: 849.1432 lxe: write_utf8_parse_stringIO (S- T3 ) 17.0564 11.5748 12.5815 msec/pass, best: 11.5748 ET : write_utf8_parse_stringIO (S- T3 ) 235.0894 234.2171 233.4405 msec/pass, best: 233.4405 cET: write_utf8_parse_stringIO (S- T3 ) 113.0575 112.7246 112.9978 msec/pass, best: 112.7246 lxe: write_utf8_parse_stringIO (U- T3 ) 11.1004 11.0245 11.3229 msec/pass, best: 11.0245 ET : write_utf8_parse_stringIO (U- T3 ) 238.6545 243.0529 251.3970 msec/pass, best: 238.6545 cET: write_utf8_parse_stringIO (U- T3 ) 122.8126 115.6941 120.9536 msec/pass, best: 115.6941 lxe: write_utf8_parse_stringIO (S- T4 ) 0.4456 0.4642 0.4494 msec/pass, best: 0.4456 ET : write_utf8_parse_stringIO (S- T4 ) 8.0726 8.1274 7.9427 msec/pass, best: 7.9427 cET: write_utf8_parse_stringIO (S- T4 ) 6.0532 5.0335 5.0780 msec/pass, best: 5.0335 lxe: write_utf8_parse_stringIO (U- T4 ) 0.4778 0.4506 0.4657 msec/pass, best: 0.4506 ET : write_utf8_parse_stringIO (U- T4 ) 7.4578 7.3894 8.0762 msec/pass, best: 7.3894 cET: write_utf8_parse_stringIO (U- T4 ) 6.2156 5.5292 5.0383 msec/pass, best: 5.0383

1 0

Re: [lxml-dev] HTMLParser behavior - ill formed HTML
by Stefan Behnel 04 May '06

04 May '06

Hi Dean, Dean Pavlekovic wrote: > On 5/2/06, Stefan Behnel <behnel_ml(a)gkec.informatik.tu-darmstadt.de> wrote: > >> I changed the trunk to always accept ill-formed results if the recover >> option is set and no lxml-internal errors were raised. Please try if that >> helps. I couldn't come up with a sufficiently short example of broken >> HTML where this problem occurs, so I couldn't test it. The examples that >> were tested so far can be found in src/lxml/tests/test_htmlparser.py. > > Unfortunately it's not working and I'm sorry but I didn't have the time to > look at it deeper. It appears that the value of options is changed > somewhere in libxml context during the htmlCtxtReadFile. I printed out the > value of parser's self._parse_options before (97, that is > XML_PARSE_RECOVER_FLAG set) and the value of ctxt.options after the > htmlCtxtReadFile call which was 96 meaning the flag was reset by libxml. Interesting. Could you tell me what version of libxml2 you are using (see below)? My guess is that it's older than 2.6.21. Libxml2 copies the options by hand, so if the RECOVER option is unknown, it will not turn up in the context options. That makes me wonder why recovery worked for Paul on 2.6.16 in the first place... Anyway. I changed the trunk to pass the option explicitly to _handleParseResult so that we no longer rely on libxml2. Please test if this works for you now. I'd be glad if you could come up with a short piece of broken HTML code that triggers the "not well formed" case in recovery mode. That would allow us to set up a test case to check that it actually works (and keeps working). To check which versions you are using, I added module attributes "LXML_VERSION", "LIBXML_VERSION" and "LIBXSLT_VERSION" that carry tuples containing the respective versions used by lxml, e.g. (2, 6, 23) for "2.6.23". "LIBXML_COMPILED_VERSION" and "LIBXSLT_COMPILED_VERSION" show against which versions lxml was compiled. BTW, these attributes are mainly meant for debugging purposes. Since they will first (officially) appear in lxml 1.0, code that wants to use them anyway will have to check if they are available (hasattr or try-except) before accessing them. All of them appeared at the same time, so if one is there, the others will be there, too. > Btw. is there a reason that the XML_... enums are assigned an explicit > value in xmlparser.pxd and the same is not done for HTML_... enums in > htmlparser.pxd? I'm not familiar with Pyrex and wheter it parses c/c++ .h > files to match values... Pyrex doesn't care about the values of enums and C uses the .h files directly, so, since they are just copy&pasted into the .pxd files, they sometimes have numbers and sometimes not. Stefan

2 2

[lxml-dev] 'docinfo' property on ElementTree
by Stefan Behnel 02 May '06

02 May '06

Hi all, I updated the trunk to provide ElementTree objects with access to the document information provided by the parser: DOCTYPE, XML version and original encoding. Paul Everitt had some use cases related to the HTML parser, but I think it's generally a good idea to make this kind of information available. The new API works as follows: >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN" >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url) >>> xml_header = '<?xml version="1.0" encoding="ascii"?>' >>> xhtml = xml_header + doctype_string + '<html><body></body></html>' >>> et = lxml.etree.parse(StringIO(xhtml)) >>> docinfo = et.docinfo >>> print docinfo.public_id -//W3C//DTD XHTML 1.0 Transitional//EN >>> print docinfo.system_url http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd >>> docinfo.doctype == doctype_string True >>> print docinfo.xml_version 1.0 >>> print docinfo.encoding ascii This is backed by a DocInfo object that you can also instantiate on an ElementTree (or Element) by hand. The docinfo property just does it for you. Any of the attributes above may be None if the information is not available. Stefan

1 0

[lxml-dev] HTMLParser behavior - ill formed HTML
by Dean Pavlekovic 02 May '06

02 May '06

Hi all, My name is Dean and I've recently joined the lxml-dev to watch how things are going most notably since the HTML parser has been added to the trunk. We (my colleague and I) have been using lxml in the past for processing of rather large DocBook type XML docs aplit across multiple files and in mixed namespaces without any problems and the experiences with lxml have been nothing but good, so let me first congratulate the developers on a great work lxml is. Lately I've looked into using the lxml trunk revision for analyzing some HTML (not well-formed). I've stumbled upon the feature for which I'm not sure whether it's intended to be. When I feed an ill-formed HTML to a parser via > doc=etree.parse('afile.html', parser=etree.HTMLParser()) #recover=True by default an exception is raised and the function yields no result. When I use libxml2 directly: > doc=libxml2.htmlParseFile('afile.html', None) libxml2 prints out some errors/warnings but I _do_ get a reference to a document, which can normally be used. I've also tried to use the procedure as in parser.pxi i.e. htmlCreateMemoryParserCtxt() and htmlCtxtReadDoc() with the same results. So I tracked this behavior down to calling _handleParseResult(pctxt, result, NULL) at the end of parse* methods in HTMLParser in parser.pxi, note the 'if.ctxt.wellFormed' part:  where the document is destroyed if libxml2's context wellFormed flag is not set. I checked this by calling libxml2 htmlCtxtReadDoc() directly on that document and indeed the wellFormed flag turned out to be 0. Now, shouldn't the HTMLParser also return the document reference in this case if recover=True flag is specified since libxml apparently does not have problems with that. I've checked this by modifying the _handleParseResult with 'accept_ill_formed' argument. If that flag is set, ctxt.wellFormed would be ignored. Also modified _handleParseResult calls in HTMLParser's parse... methods to specify accept flag if 'recover' was set in the constructor. This turned to work just well and I was able to navigate the document with xpath, even the errors (using '&' in href attributes) were corrected. So the question would be is the present behavior correct due to something I'm possibly missing? I think it should be dependent on whether RECOVER flags have been specified or not. Best regards, Dean

2 1