Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] building the trunk and test failures
by Martijn Faassen 30 May '06

30 May '06

Hi there, I just tried building the trunk: Python 2.4, Pyrex 0.9.4.1, libxml2 2.6.21 The first thing I noticed is that compilation of the C source code (with 'Make') seems to take extremely long in comparison to the past. What changed to cause that? The next thing is that I get some test failures (included below). The api.txt and the extensions.txt test failures both seem to be triggered by the same problem. What could be going on here? Different behavior between different versions of libxml2? Regards, Martijn python test.py -p -v 459/540 ( 85.0%): Doctest: extensions.txt ====================================================================== ERROR: test_docinfo_public (lxml.tests.test_etree.ETreeOnlyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/lxml/tests/test_etree.py", line 456, in test_docinfo_public tree = etree.parse(StringIO(xml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ====================================================================== FAIL: Doctest: api.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/faassen/working/lxml-trunk/src/doctest.py", line 2187, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for api.txt File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 0 ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 82, in api.txt Failed example: et = etree.parse(StringIO(xhtml)) Exception raised: Traceback (most recent call last): File "/home/faassen/working/lxml-trunk/src/doctest.py", line 1256, in __run compileflags, 1) in test.globs File "<doctest api.txt[19]>", line 1, in ? et = etree.parse(StringIO(xhtml)) File "etree.pyx", line 1483, in etree.parse File "parser.pxi", line 665, in etree._parseDocument File "parser.pxi", line 688, in etree._parseMemoryDocument File "parser.pxi", line 602, in etree._parseDoc File "parser.pxi", line 343, in etree._BaseParser._parseDoc File "parser.pxi", line 423, in etree._handleParseResult File "parser.pxi", line 394, in etree._raiseParseError XMLSyntaxError: switching encoding : no input ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 84, in api.txt Failed example: print docinfo.public_id Expected: -//W3C//DTD XHTML 1.0 Transitional//EN Got: -//W3C//DTD HTML 4.0 Transitional//EN ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 86, in api.txt Failed example: print docinfo.system_url Expected: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd Got: http://www.w3.org/TR/REC-html40/loose.dtd ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 88, in api.txt Failed example: docinfo.doctype == doctype_string Expected: True Got: False ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 91, in api.txt Failed example: print docinfo.xml_version Expected: 1.0 Got: None ---------------------------------------------------------------------- File "/home/faassen/working/lxml-trunk/src/lxml/tests/../../../doc/api.txt", line 93, in api.txt Failed example: print docinfo.encoding Expected: ascii Got: None ---------------------------------------------------------------------- Ran 459 tests in 1.695s FAILED (failures=1, errors=1)

2 5

[lxml-dev] absolute XPath expressions on Elements
by Stefan Behnel 30 May '06

30 May '06

Hi, I implemented a getpath() method for the ElementTree class that returns an XPath expression for a node. While working out test cases for it, however, I realized that the semantics of evaluating absolute XPath expressions (/...) on elements were not clear at all in the current implementation. ET does not allow absolute expressions in Element.findall() and raises a SyntaxError instead. I think we should do the same for Element.xpath() to prevent mistakes like this: >>> a = etree.Element("a") >>> b = etree.SubElement(a, "b") >>> d0 = etree.SubElement(b, "d") >>> c = etree.SubElement(a, "c") >>> d1 = etree.SubElement(c, "d") >>> d2 = etree.SubElement(c, "d") >>> c.xpath("//d") The reasoning is that Elements do not have a root and therefore no absolute starting point for XPath. Only ElementTrees provide the required semantics, so it's perfectly valid to do this instead: >>> ElementTree(c).xpath("//d") Imagine the case where you have many ElementTrees wrapping various elements in a tree. Which one should be the starting point? Remember that documents and their absolute root node are not exposed through the API. The use case that brought me there is this: >>> tree = etree.ElementTree(c) >>> print tree.getpath(d2) /c/d[2] >>> tree.xpath(tree.getpath(d2)) == [d2] # fails! Intuitively, this should work. However, the current implementation fails here, as it starts searching at 'a' rather than 'c' and thus finds nothing. To fix this, we have to switch the root node during XPath evaluation. Doing this for ElementTree.xpath() is ok, but doing this for Element.xpath() also is impossible, as it breaks relative expressions like "..". So I decided to simply special case XPath expressions starting with '/' and raise exceptions for them. I know that this is not sufficient, as absolute paths can be hidden in things like "*[/a]" or "a|/a". But it's hopefully enough to make users aware and to prevent common mistakes. I also added a note in the documentation that the result of absolute expressions is undefined for Elements, I think that's the right way of saying it. I post this here as there will likely be code that breaks because of this change. I already found two test cases in the test suite that used this. It's just too easy to get wrong, so lxml is better off by raising exceptions where it can than just ignoring this problem. Stefan

3 10

[lxml-dev] ElementTree pretty printing (serialisation)
by Noah Slater 30 May '06

30 May '06

Hello again, My constant spamming of this list has finally paid of and I have something to show for all my questions. I have attached the source of a fairly advanced pretty printing serialiser for ElementTree (and ResultTree) objects. The script (after `chmod 755`) can be called from the command line like so: ./prettyprint.py [DOCUMENT] The script can also be imported and used like so: import sys from lxml import etree import prettyprint document = etree.parse(...) serialiser = prettyprint.ElementTreeSerialiser() serialiser.write(document, sys.stdout) I am new to python, and even newer to lxml and the ElementTree API. As a consequence I may be missing some obvious optimisations. This is only my first stab at this problem and I would love any feed back you care to offer. Is pretty printing something that is on the lxml time line - and if not, would my method demonstrated here interest you from an implementation point of view? Thanks so much for your time. Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 10

[lxml-dev] Best Way To Strip A Namespace
by Noah Slater 30 May '06

30 May '06

Hello, Say I have the following document: <box xmlns="http://example.com/box" xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> After I have parsed it into an element tree, how do I change it in a fool proof manner to loose the root namespace so it becomes: <box xmlns:foo="http://example.com/foo"> <foo:bar>baz></foo:bar> </box> This is important for processing my DocBook sources as the XSLT stylesheets cannot handle the namepace on the root element. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 3

[lxml-dev] Threading
by Stefan Behnel 29 May '06

29 May '06

Hi all, I looked a bit at the threading issues and found that there is not much to gain from using threads anyway. Calling code will suffer from the GIL, so it would be up to lxml to release the GIL on long-running operations to benefit from threading. However, releasing the GIL is non-trivial, as most of these operations can call back into the Python API and the interpreter: parsing calls resolvers, as does XSLT. XPath can call extension functions, even the error messages end up in Python objects and lists, etc. So, there are only very few places that could easily be wrapped by ALLOW_THREADS: * everything that traverses trees (which is so fast that the gain may be eaten by the locking overhead) and if error messages can somehow be made working without requiring the GIL: * serialisation to memory or 'real' files * validation (which current does not support custom resolvers) An alternative would be to always create separate Python threads for things like error handling and resolving, but that would require always releasing the GIL whenever these things *might* get called. So, it would be relatively easy to release the GIL in the ElementDepthFirstIterator to have a thread speedup for findall() etc., but everything else will be real work. And since the major overhead is in parsers and serialisers, I don't think it's worth changing this bit just to say "you can gain from threading". Considering this, I changed the FAQ entry on threading to simply state that threading is not supported, with a short explanation that the gain would be marginal without major changes in lxml. Stefan

1 0

[lxml-dev] INPUT: Iniitial FAQ list
by Paul Everitt 29 May '06

29 May '06

Howdy all. This is a thread to collect ideas for an initial FAQ list. Right now we have one on the table: """ How compatible are the findall() and xpath() methods ? findall() don't seem to handle more complicated XPath expressions. Why there is a difference between what they can handle ? """ Other ideas? --Paul

2 2

Re: [lxml-dev] segfault from thread on Windows
by Stefan Behnel 29 May '06

29 May '06

Hi Scott, Scott Haeger wrote: > Here is a small code snippet that demostrates my problem. As you mentioned, I can't reproduce any problems with this under Linux. Couldn't test it under Windows. > The code > works just fine as it is written. However, python crashes if you > comment out the parseFile( ) call from the main thread. I assume that what happens here is that libxml2 misses its thread-local setup. > This seems to > be contrary to what you described about using the default parser from > different threads. > > def parseFile(): > s = "<test><a>entrya</a><b>entryb</b></test>" > sio = StringIO.StringIO (s) > tree = etree.parse(sio) > tree.write(sys.stdout) > sys.stdout.flush() > > parseFile() > t = thread.start_new_thread(parseFile, ()) I understand what you did now. The current way do it this would be something like: def parseFileFromThread(): etree.initThread() # needs the current trunk to work thread_local_parser = etree.XMLParser() s = "<test><a>entrya</a><b>entryb</b></test>" sio = StringIO.StringIO(s) tree = etree.parse(sio, thread_local_parser) tree.write(sys.stdout) parseFile() t1 = thread.start_new_thread(parseFileFromThread, ()) t2 = thread.start_new_thread(parseFileFromThread, ()) 1.0.beta did not have an initThread() function yet (it had an initThreadLogging function, which did part of what the new initThread does). However, I noticed that it is needed to set up some thread-local libxml2 options. This means that you have to call this method from a newly created thread. It doesn't do any harm to call it from the main thread, though, so you can call the above parseFileFromThread also from the main thread. So, hopefully, your problem is solved in the current trunk. I'd be glad if you could test that with the above modifications to your code. I guess I'll have to write up some documentation on this, but it's good to have some feedback first. Stefan

1 0

[lxml-dev] Segmentation Fault
by Noah Slater 29 May '06

29 May '06

Hello guys, I have found a segfault bug with lxml - attached is a file which reproduces it. Hope this helps. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 1

[lxml-dev] Extending lxml.etree._ElementTree
by Noah Slater 28 May '06

28 May '06

Hello again, I would like to extend the _ElementTree class to include some handy methods - write_xml/write_xhtml/write_html are examples. In addition, this should enable me to return extended ElementTrees instead of ResultTrees after an XSLT transform. What would be the best way to go about this. While I can extend the base class I am struggling to understand how I can replace etree.parse() to return these objects. While I am on the subject, what is the reason for ResultTree objects as a pose to ElementTree. Why make the distinction to the user? Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 2

[lxml-dev] segfault on Windows
by Scott Haeger 28 May '06

28 May '06

I am trying to port an application to Windows. It works with no problems on Linux. On Windows, lxml parses a large file with no problems on the main thread. However, I am getting a segfault when it parses the same file from a spawned thread. Has anyone seen the same thing. Python 2.4.2 lxml 1.0 beta from binary built on May 18 Let me know if there is anything else I can provide. Thanks, Scott Haeger

2 1