Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] Remarks on implementing iterparse()
by Stefan Behnel 22 Jun '06

22 Jun '06

Hi all, since I won't have the time to implement iterparse() any time soon, here's a proposal on how it should be implemented, in case someone wants to take a shot at it. "iterparse" will be (or will return) an iterable object, let's call it IterParse for clarity. A class is basically the only way of implementing iterators in Pyrex. For the internal SAX part, IterParse will likely work a lot like lxml.sax.ElementTreeContentHandler. We'd need a custom wrapper to the default libxml2 SAX handler to intercept the parse events (this means implementing C helper functions for the SAX events) /after/ they were processed by libxml2. See xmlSAXVersion (SAX2) on how to retrieve the SAX2 default parser structure. IterParse should pass chunks into the parser and buffer the events it receives. When its __next__() method is called, it returns one event or passes new chunks until there is an event to return. This is needed as IterParse has to convert between libxml2 push (SAX) and Python pull (iter). As for the input to the libxml2 parser, there are two possible ways: one is to pass data chunks in through xmlParseChunk and the other is to use xmlCreateIOParserCtxt and implement xmlInputReadCallback (xmlio.h) to have libxml2 request data by itself. Python events (start, end, start-ns, end-ns) are created as follows: * "*-ns" events must be extracted from the libxml2 xmlSAX2StartElementNs call (passed in arguments "prefix"/"URI" and the char* array "namespaces"). They must be stored on a stack to build the respective "end-ns" events. * "start" is somewhat tricky, as it would be a bad idea to allow modifications of the XML structure during that iterator cycle. Maybe it's enough to document that, but there may be ways to crash lxml with certain tree operations. Note also that care has to be taken to prevent Python from garbage collecting the element before the "end" event. The best way to do that is to store a Python reference to that element on a stack. * "end" is simple then: pop the element from the stack and return it. That's all I can come up with so far. So, if anyone is interested in taking a look at it, I'd be glad to hear about it. :) Stefan

3 4

[lxml-dev] element of an xpath evaluation
by Andreas Pakulat 03 Jun '06

03 Jun '06

Hi, very recently I needed a program that evaluates a xpath and displays the "result" graphically in a tree of the xml file that was used. I found XPath Explorer a java application, however it does quite some more stuff than I need and there certain things that just don't work as I want. I figured I could very easily provide something similar using lxml and PyQt4. However if I want to highlight the tree node that the xpath matches I have a "problem" when the xpath matches attributes or text nodes. So the question is: Is there a way using lxml to find out to which element a certain non-element result of an xpath evaluation belongs? Andreas -- You now have Asian Flu.

2 7

[lxml-dev] Defining name spaces in lxml/xslt
by Petr van Blokland 02 Jun '06

02 Jun '06

Hi, new to lxml I may be overlooking some hints in the doc files. I am trying to define a name space (myspace) in Python, but I don't know by which method my name space object is getting called from lxml/xslt. class AbcElement(ElementBase): def ###METHODNAME###(self): return 'Output of abc' ns = Namespace('http://xml.petr.com/xpyth3/ns/myspace') ns['abc'] = AbcElement xslt = '''<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:abc="http://xml.petr.com/xpyth3/xpath/ns/myspace" xsl:extension-element-prefixes='myspace' > <xsl:template match="/"> <foo> < myspace:abc/> </foo> </xsl:template> </xsl:stylesheet>''' Where ###METHODNAME### I don't know. I am using lxml 1.0 beta Suggestions? Kind regards, Petr van Blokland ---------------------------------------------- Petr van Blokland buro(a)petr.com | www.petr.com | +31 15 219 10 40 ----------------------------------------------

2 5

[lxml-dev] improved valgrind suppressions for Python?
by Martijn Faassen 01 Jun '06

01 Jun '06

Hi there, Since upgrading to a new version of valgrind quite a while ago, valgrind gets spammy, even with the Python suppressions in place. I suspect this is because valgrind is getting more picky. I'm getting stuff like this: ==10199== Conditional jump or move depends on uninitialised value(s) ==10199== at 0x1B8F4FD1: (within /lib/ld-2.3.5.so) ==10199== by 0x1B8EA4AA: (within /lib/ld-2.3.5.so) ==10199== by 0x4111F18F: (within /lib/tls/libc-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x4111FB85: _dl_open (in /lib/tls/libc-2.3.5.so) ==10199== by 0x4118FD32: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x1B8EF026: (within /lib/ld-2.3.5.so) ==10199== by 0x41190486: (within /lib/tls/libdl-2.3.5.so) ==10199== by 0x4118FDB0: dlopen (in /lib/tls/libdl-2.3.5.so) ==10199== by 0x80DCE40: _PyImport_GetDynLoadFunc (in /usr/bin/python2.4) ==10199== by 0x80D2622: _PyImport_LoadDynamicModule (in /usr/bin/python2.4) ==10199== by 0x80D223F: (within /usr/bin/python2.4) and generally ld related output, presumably something to do with dynamic linking. This makes it harder to pick out potentially real memory issues. I'm wondering whether others get this same behavior with valgrind and have it fixed, or whether everybody is stuck with it. I looked for newer suppressions in the Python repository at some point, but couldn't find it then. My valgrind version is 3.0.1 Regards, Martijn

2 7

[lxml-dev] Faster iteration and deallocation
by Stefan Behnel 31 May '06

31 May '06

Hi all, I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway and found a lot of code duplication between functions that iterate over the tree. I replaced that by a set of BEGIN/END C macros, which has three effects: * Tree iteration now uses the same code throughout lxml * It's much shorter and thus (hopefully) easier to understand * It's about 20-30% faster than the previous recursive implementations I took another somewhat experimental step and removed the _NodeBase superclass from _Attrib to make it a plain Python object that sits on an _Element. This allowed for another major clean up and a huge simplification of the proxy code, since there is only one proxy type left. I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain. According to the test cases, everything works nicely as before and I will give valgrind a run to see if there's anything suspicious left. It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it. Any feedback is appreciated. Stefan

4 8

[lxml-dev] filtering namespaces in findall
by Stefan Behnel 31 May '06

31 May '06

Hi, since I was rewriting the tree iterator anyway, I stumbled over the fact that ElementTree's getiterator() can filter for any tag ("*"), but not for any tag within a namespace ("{namespace}*"). Since this is just a cheap and straight forward little extension to the node filter (and a pretty nice feature to have), I added that to lxml and hope for inclusion in a future ElementTree version also (or maybe a hint why Fredrik assumes this to be inappropriate). Note that it's not currently supported in findall, though, as we use the same implementation as ElementTree there. Stefan

1 1

[lxml-dev] Bug in memory management (I guess ;)
by Johnny deBris 31 May '06

31 May '06

Hi! The past couple of months I kept running into some bug. For some reason the namespace information of certain nodes got lost after moving the nodes from one document to another. I kept trying to isolate the problem, but without success, until this week I found that the reason the information gets lost seems to have to do with the document getting garbage collected even though some nodes are still referred to. With this in mind I managed to write the following demonstration snippet: #-------------------------------------------------------------------------- from lxml import etree s1 = '<a xmlns:a="A:"></a>' tree1 = etree.fromstring(s1) btag = tree1.xpath('//b')[0] del tree1 s2 = '<x />' tree2 = etree.fromstring(s2) tree2.append(btag) # this produces crap... print etree.tostring(tree2) #-------------------------------------------------------------------------- The problem seems to be the 'del' statement: this seems to free the document instance or something, even though there's still a reference to one of its nodes. Serializing the node later on will result in garbage. When the 'del' line is commented out, the XML is produced properly. I hope this is enough information for you guys to track the bug: I'm no C programmer myself, so I don't dare diving into the Pyrex, and also I may have made some errors in my explanation... If you need more information, let me know. Cheers, Guido

2 5

[lxml-dev] Bug with attribute mangling when adding child elements
by Noah Slater 30 May '06

30 May '06

Hello, I noticed today that certain child attribute values are mangled when you try to insert/append child elements onto parent elements. I have written a test script for this as usual. I'm probably missing something... again. :) Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

3 6

[lxml-dev] Difference between ResultTree.__str__ and ResultTree.write
by Noah Slater 30 May '06

30 May '06

Hi, Could we get the difference between ResultTree.__str__ and ResultTree.write properly documented somewhere ([1]?) on the website or via docstrings etc as there only seems to be one reference to it that I can find [2] By the way, the ability to change "output" parameters with write is an excellent addition to this package - such a time saver for me! :) Thanks, Noah [1] http://codespeak.net/lxml/api.html [2] http://codespeak.net/pipermail/lxml-dev/2006-May.txt -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 5

[lxml-dev] credits update: Stefan is the maintainer
by Martijn Faassen 30 May '06

30 May '06

Hi there, I've just updated the CREDITS.txt of lxml to reflect what had become the situation for a long time already: Stefan Behnel is currently the main developer of lxml and its maintainer, and my own role as creator and initial maintainer has diminished somewhat. I'm not disappearing and will remain involved in lxml development for the foreseeable future, but Stefan calls the shots, as he took the responsibility - this is as it should be in open source. I recognized what was going on in december and in personal communication then gave Stefan the keys to the lxml car. He's been doing a great job with it since, and as the 1.0 release is pending we thought we should announce this officially. I personally think Stefan's involvement is an absolutely awesome development. The most obvious reason why this is great is of course Stefan's qualities as a developer and project leader. Less obvious but also very exciting to me is that it makes lxml into a true open source project in the community sense. lxml's future is no longer dependent on a single person, namely myself. Stefan has shown great skill in both sides of managing a successful open source project: the code side and the people side. lxml has now become one of the most powerful XML libraries in the Python world, in the area of feature support as well as in the area of performance. We came a long way in the year and almost-a-half since my initial breakthrough in getting the lxml ball rolling. On the more personal front I also met a few people I'm happy to know now, got to speak about lxml at conferences, and of course got a great XML library out of it too. Thanks to all contributors, and of course especially to Stefan, in making this possible! Regards, Martijn

3 2

[lxml-dev] building the trunk and test failures
by Martijn Faassen 29 May '06

29 May '06

Hi there, I just tried building the trunk: Python 2.4, Pyrex 0.9.4.1, libxml2 2.6.21 The first thing I noticed is that compilation of the C source code (with 'Make') seems to take extremely long in comparison to the past. What changed to cause that? The next thing is that I get some test failures (included below). The api.txt and the extensions.txt test failures both seem to be triggered by the same problem. What could be going on here? Different behavior between different versions of libxml2? Regards, Martijn python test.py -p -v 459/540 ( 85.0%): Doctest: extensions.txt ====================================================================== ERROR: test_docinfo_public (lxml.tests.test_etree.ETreeOnlyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/lxml/tests/test_etree.py", line 456, in test_docinfo_public tree = etree.parse(StringIO(xml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ====================================================================== FAIL: Doctest: api.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/doctest.py", line 2187, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for api.txt File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 0 ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 82, in api.txt Failed example: et = etree.parse(StringIO(xhtml)) Exception raised: Traceback (most recent call last): File "/home/faassen/working/lxml-trunk/src/doctest.py", line 1256, in __run compileflags, 1) in test.globs File "<doctest api.txt[19]>", line 1, in ? et = etree.parse(StringIO(xhtml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 84, in api.txt Failed example: print docinfo.public_id Expected: -//W3C//DTD XHTML 1.0 Transitional//EN Got: -//W3C//DTD HTML 4.0 Transitional//EN ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 86, in api.txt Failed example: print docinfo.system_url Expected: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd Got: http://www.w3.org/TR/REC-html40/loose.dtd ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 88, in api.txt Failed example: docinfo.doctype == doctype_string Expected: True Got: False ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 91, in api.txt Failed example: print docinfo.xml_version Expected: 1.0 Got: None ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 93, in api.txt Failed example: print docinfo.encoding Expected: ascii Got: None ---------------------------------------------------------------------- Ran 459 tests in 1.695s FAILED (failures=1, errors=1)

2 5

[lxml-dev] absolute XPath expressions on Elements
by Stefan Behnel 29 May '06

29 May '06

Hi, I implemented a getpath() method for the ElementTree class that returns an XPath expression for a node. While working out test cases for it, however, I realized that the semantics of evaluating absolute XPath expressions (/...) on elements were not clear at all in the current implementation. ET does not allow absolute expressions in Element.findall() and raises a SyntaxError instead. I think we should do the same for Element.xpath() to prevent mistakes like this: >>> a = etree.Element("a") >>> b = etree.SubElement(a, "b") >>> d0 = etree.SubElement(b, "d") >>> c = etree.SubElement(a, "c") >>> d1 = etree.SubElement(c, "d") >>> d2 = etree.SubElement(c, "d") >>> c.xpath("//d") The reasoning is that Elements do not have a root and therefore no absolute starting point for XPath. Only ElementTrees provide the required semantics, so it's perfectly valid to do this instead: >>> ElementTree(c).xpath("//d") Imagine the case where you have many ElementTrees wrapping various elements in a tree. Which one should be the starting point? Remember that documents and their absolute root node are not exposed through the API. The use case that brought me there is this: >>> tree = etree.ElementTree(c) >>> print tree.getpath(d2) /c/d[2] >>> tree.xpath(tree.getpath(d2)) == [d2] # fails! Intuitively, this should work. However, the current implementation fails here, as it starts searching at 'a' rather than 'c' and thus finds nothing. To fix this, we have to switch the root node during XPath evaluation. Doing this for ElementTree.xpath() is ok, but doing this for Element.xpath() also is impossible, as it breaks relative expressions like "..". So I decided to simply special case XPath expressions starting with '/' and raise exceptions for them. I know that this is not sufficient, as absolute paths can be hidden in things like "*[/a]" or "a|/a". But it's hopefully enough to make users aware and to prevent common mistakes. I also added a note in the documentation that the result of absolute expressions is undefined for Elements, I think that's the right way of saying it. I post this here as there will likely be code that breaks because of this change. I already found two test cases in the test suite that used this. It's just too easy to get wrong, so lxml is better off by raising exceptions where it can than just ignoring this problem. Stefan

3 10

[lxml-dev] ElementTree pretty printing (serialisation)
by Noah Slater 29 May '06

29 May '06

Hello again, My constant spamming of this list has finally paid of and I have something to show for all my questions. I have attached the source of a fairly advanced pretty printing serialiser for ElementTree (and ResultTree) objects. The script (after `chmod 755`) can be called from the command line like so: ./prettyprint.py [DOCUMENT] The script can also be imported and used like so: import sys from lxml import etree import prettyprint document = etree.parse(...) serialiser = prettyprint.ElementTreeSerialiser() serialiser.write(document, sys.stdout) I am new to python, and even newer to lxml and the ElementTree API. As a consequence I may be missing some obvious optimisations. This is only my first stab at this problem and I would love any feed back you care to offer. Is pretty printing something that is on the lxml time line - and if not, would my method demonstrated here interest you from an implementation point of view? Thanks so much for your time. Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 10

[lxml-dev] Best Way To Strip A Namespace
by Noah Slater 29 May '06

29 May '06

Hello, Say I have the following document: <box xmlns="http://example.com/box" xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> After I have parsed it into an element tree, how do I change it in a fool proof manner to loose the root namespace so it becomes: <box xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> This is important for processing my DocBook sources as the XSLT stylesheets cannot handle the namepace on the root element. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 3

[lxml-dev] Threading
by Stefan Behnel 29 May '06

29 May '06

Hi all, I looked a bit at the threading issues and found that there is not much to gain from using threads anyway. Calling code will suffer from the GIL, so it would be up to lxml to release the GIL on long-running operations to benefit from threading. However, releasing the GIL is non-trivial, as most of these operations can call back into the Python API and the interpreter: parsing calls resolvers, as does XSLT. XPath can call extension functions, even the error messages end up in Python objects and lists, etc. So, there are only very few places that could easily be wrapped by ALLOW_THREADS: * everything that traverses trees (which is so fast that the gain may be eaten by the locking overhead) and if error messages can somehow be made working without requiring the GIL: * serialisation to memory or 'real' files * validation (which current does not support custom resolvers) An alternative would be to always create separate Python threads for things like error handling and resolving, but that would require always releasing the GIL whenever these things *might* get called. So, it would be relatively easy to release the GIL in the ElementDepthFirstIterator to have a thread speedup for findall() etc., but everything else will be real work. And since the major overhead is in parsers and serialisers, I don't think it's worth changing this bit just to say "you can gain from threading". Considering this, I changed the FAQ entry on threading to simply state that threading is not supported, with a short explanation that the gain would be marginal without major changes in lxml. Stefan

1 0

[lxml-dev] INPUT: Iniitial FAQ list
by Paul Everitt 29 May '06

29 May '06

Howdy all. This is a thread to collect ideas for an initial FAQ list. Right now we have one on the table: """ How compatible are the findall() and xpath() methods ? findall() don't seem to handle more complicated XPath expressions. Why there is a difference between what they can handle ? """ Other ideas? --Paul

2 2

Re: [lxml-dev] segfault from thread on Windows
by Stefan Behnel 28 May '06

28 May '06

Hi Scott, Scott Haeger wrote: > Here is a small code snippet that demostrates my problem. As you mentioned, I can't reproduce any problems with this under Linux. Couldn't test it under Windows. > The code > works just fine as it is written. However, python crashes if you > comment out the parseFile( ) call from the main thread. I assume that what happens here is that libxml2 misses its thread-local setup. > This seems to > be contrary to what you described about using the default parser from > different threads. > > def parseFile(): > s = "<test><a>entrya</a>entryb</test>" > sio = StringIO.StringIO (s) > tree = etree.parse(sio) > tree.write(sys.stdout) > sys.stdout.flush() > > parseFile() > t = thread.start_new_thread(parseFile, ()) I understand what you did now. The current way do it this would be something like: def parseFileFromThread(): etree.initThread() # needs the current trunk to work thread_local_parser = etree.XMLParser() s = "<test><a>entrya</a>entryb</test>" sio = StringIO.StringIO(s) tree = etree.parse(sio, thread_local_parser) tree.write(sys.stdout) parseFile() t1 = thread.start_new_thread(parseFileFromThread, ()) t2 = thread.start_new_thread(parseFileFromThread, ()) 1.0.beta did not have an initThread() function yet (it had an initThreadLogging function, which did part of what the new initThread does). However, I noticed that it is needed to set up some thread-local libxml2 options. This means that you have to call this method from a newly created thread. It doesn't do any harm to call it from the main thread, though, so you can call the above parseFileFromThread also from the main thread. So, hopefully, your problem is solved in the current trunk. I'd be glad if you could test that with the above modifications to your code. I guess I'll have to write up some documentation on this, but it's good to have some feedback first. Stefan

1 0

[lxml-dev] Segmentation Fault
by Noah Slater 28 May '06

28 May '06

Hello guys, I have found a segfault bug with lxml - attached is a file which reproduces it. Hope this helps. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 1

[lxml-dev] Extending lxml.etree._ElementTree
by Noah Slater 28 May '06

28 May '06

Hello again, I would like to extend the _ElementTree class to include some handy methods - write_xml/write_xhtml/write_html are examples. In addition, this should enable me to return extended ElementTrees instead of ResultTrees after an XSLT transform. What would be the best way to go about this. While I can extend the base class I am struggling to understand how I can replace etree.parse() to return these objects. While I am on the subject, what is the reason for ResultTree objects as a pose to ElementTree. Why make the distinction to the user? Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 2

[lxml-dev] segfault on Windows
by Scott Haeger 28 May '06

28 May '06

I am trying to port an application to Windows. It works with no problems on Linux. On Windows, lxml parses a large file with no problems on the main thread. However, I am getting a segfault when it parses the same file from a spawned thread. Has anyone seen the same thing. Python 2.4.2 lxml 1.0 beta from binary built on May 18 Let me know if there is anything else I can provide. Thanks, Scott Haeger

2 1

[lxml-dev] findall()/xpath() differences ?
by Steve Howe 26 May '06

26 May '06

Hello all, How compatible are the findall() and xpath() methods ? findall() don't seem to handle more complicated XPath expressions. Why there is a difference between what they can handle ? I would expect findall() to be the same as xpath(), but searching from the context node and always returning a list of items, making it compatible with ET's. An example: Python 2.4.3 (#2, Apr 17 2006, 14:29:19) [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> root = etree.XML('''<obj class="list"> ... <obj class="str" value="xxx"/> ... </obj>''') >>> print root.xpath('obj[not(@name)]') [<Element obj at 81daa2c>] >>> print root.findall('obj[not(@name)]') Traceback (most recent call last): File "<stdin>", line 1, in ? File "etree.pyx", line 937, in etree._Element.findall File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 193, in findall return _compile(path).findall(element) File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 171, in _compile p = Path(path) File "/usr/local/lib/python2.4/site-packages/lxml-1.0.beta-py2.4-freebsd-6.1-RELEASE-i386.egg/lxml/_elementpath.py", line 87, in __init__ raise SyntaxError( SyntaxError: expected path separator ([) -- Best regards, Steve mailto:howe@carcass.dhs.org

6 13

[lxml-dev] XMLFormatter revisited
by Stefan Behnel 22 May '06

22 May '06

Hi all, coming back to what I proposed a while ago, we currently have this: class XMLParser: def __init__(self, **options): self.options = options class HTMLParser: ... doc = ET.parse(source, parser=XMLParser(configuration)) At the time, I brought up an equivalent for output formatting: class XMLFormatter: ... That would give us a nice, symmetric API for input and output options. And it allows you to use sublasses to provide different formats: class XMLPrettyPrinter(XMLFormatter): def __init__(self): self._pretty_print = True class XHTMLFormatter(XMLFormatter): def __init__(self): self._xhtml = True xml = ET.tostring(element, formatter=XMLPrettyPrinter()) ET.parse("myfile.xhtml").write("out.xhtml", formatter=XHTMLFormatter()) After Noah's latest approach in that direction, I think we should adopt this API for 1.0. A version 1.0 is supposed to provide a stable and somewhat future-proof API, and adding keyword arguments to various API functions all the time is not quite what I call future-proof. One reason is that you can't test for the availability of keyword arguments, so adding features at that level is difficult to handle for code that wants to support them as an option. So, I propose to replace the current pretty_print keyword (which only appeared in the beta version anyway) with a new XMLPrettyPrinter class and to provide new features preferably at a subclass level of XMLFormatter (e.g. XHTMLFormatter, etc.). An alternative name would be XMLSerializer, maybe that's more general. We could also leave the pretty_print keyword in as a shortcut, but that would obviously make things a bit more complicated internally. Maybe pretty printing will just become a general keyword of the XMLFormatter class, I guess that would make sense. I will have to check how to make these classes nicely usable internally, but that's a minor problem. We already use a lot of xmlBuffer code, so that should give us a common ground for this API. If there are no objections, I'll start getting my hands on this next week, so if anyone has an opinion on this, please speak up soon. Stefan

1 1

Re: [lxml-dev] Can't change text in comments
by Stefan Behnel 22 May '06

22 May '06

Hi Noah, Noah Slater wrote: > I have noticed that setting the 'text' of a Comment does not seem to > change anything when the Comment is serialised. Oh well, /that/ was an old bug you found there. The code in that area dates from SVN revision 8082 - we're at 27509 now. Changing the text of a comment was never implemented, although ET supports it. The reason (I believe) why Martijn did not implement it at the time (unless he just forgot) is that changing comment texts in libxml2 is a bit tricky. There are no libxml2 functions for doing that and it requires checking the document dictionary to see if the original comment string isn't still in use anywhere else. Great place to add some more segfaults. I implemented this and actually found a couple of other bugs that I fixed (including the test cases that relied on these bugs). What bothers me is that lxml is consistent with ElementTree in that it adds whitespace around comment texts. I have no idea why ElementTree does that in the first place. AFAICT, this happily breaks things like SSI. Maybe Fredrik can enlighten us here? Stefan

4 4

[lxml-dev] lxml crash
by Steve Howe 22 May '06

22 May '06

Hello all, I know that the code is wring, but this crash is happening on my FreeBSD 6.1 and Win XP systems, with lxml 0.9.2: [carcass@/home/howe] python Python 2.4.3 (#2, Apr 17 2006, 14:29:19) [GCC 3.4.4 [FreeBSD] 20050518] on freebsd6 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> a = etree.Element('a') >>> a.append(None) Exception exceptions.TypeError: 'Argument must not be None.' in 'etree._raiseIfNone' ignored zsh: segmentation fault python It should be more friendly by just displaying the message and not crashing, right ? By the way, is there any way to get the lxml version from code ? Some module '__version__' attribute ? Shouldn't it be there ? Something like: lxml.__version__ -- Best regards, Steve mailto:howe@carcass.dhs.org

5 10

Re: [lxml-dev] XMl Processing Instructions
by Stefan Behnel 20 May '06

20 May '06

Hi Noah, Noah Slater wrote: > If I understand your email correctly the behaviour I describe is not > intentional and will be fixed shortly? It has been fixed in the developer version, but there is not yet a release that has the fix. > Just to clarify - I can parse any file, but when I serialise I loose > any processing instructions. This includes the <?xml ... ?> > declaration. I understood that. > This also happens with the ResultTree (?) when I transform using XSLT. > > As an example, I use the DocBook XSLT stylesheets to transform DocBook > XML. This can often set various things up with the processing > instructions - character encoding being the most important. > > When I perform these transformations using ElementTree I loose this > information. > > As a work around at the moment I am using lxml.etree to do the > transformations using UTF-8 as the encoding. I am then using libxml2 > and libxslt to transform the serialized document bytestream a second > time with the only operation being converting from UTF-8 to another > (variable) character encoding. > > This feels quite hackish Well, it /is/ hackish. :) > - and to be honest the whole point of me > moving to lxml was because I find the libxml2 and libxslt bindings > hateful. That's one of the main reasons that drive us in writing lxml - right after users saying "thank you, it's great and helps us do stuff!" :) > To summarise, in an ideal world I would like to be able to transform a > document using XSLT specifying an encoding at transformation time and > have the ResultTree serialise with all processing instructions intact. > Additionally I would like to be able to access these programmatically > - which I don't think is possible at the moment. That's also a feature of the developer version that will eventually become lxml 1.0. See the 'docinfo' feature described here: http://codespeak.net/svn/lxml/trunk/doc/api.txt See here for a complete list of changes since 0.9.2: http://codespeak.net/svn/lxml/trunk/CHANGES.txt > I hope all this makes sense. It does. Thanks for the report. Please use the current SVN trunk version for now, lxml 1.0 is expected to be released this month. See http://codespeak.net/lxml/installation.html and the bottom of the download section in http://codespeak.net/lxml/ on how to check it out of the subversion repository and compile it. Stefan

2 2