Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] XMl Processing Instructions
by Noah Slater 20 May '06

20 May '06

Hello, Why I use the write method of the ElementTree class why does it strip out the XML processing insturctions? I would like my documents to start with the processing instruction so I can specify encodings other than UTF-8. Thanks, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 5

[lxml-dev] Encoding Issues
by Noah Slater 20 May '06

20 May '06

Hello, I notice that I can pass an encoding parameter to an elementtree's write method. Two things: 1) When lxml supports xml processing instructions, will these be updated accordingly? I notice that the DocBook XSL will update meta-equiv HTML elements according to the value I pass in here - which is good, except for the following... 2) Why does this let me make up encodings such as "Noahs-Cool-Encoding" without raising an exception. If I was to take a guess I would imagine this falls back on UTF-8 but these is a bug IMO. In case you are wondering I am in the process of writing an XMl based HTTP publishing framework and lxml will be sitting at the very core of how I handle document conversion/manipulation/transformation. The problem with encoding lies within my content negotiation module which will (in a pythonic manner IMO) try to transform the document with each encoding specified in the Accept-Charset header of the client request. If the transform raises an exception we move on to the next one. My previous way of working would raise an exception if I tried an encoding it didn't recognise. While I am aware of how to look up encoding names using the python standard library - I am not sure if this correlates 100% with lxml and additionally I don't feel this extra step should be necessary. Thanks so much. Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

2 2

[lxml-dev] lxml/ET/cET performance comparison in doc/performance.txt
by Stefan Behnel 19 May '06

19 May '06

Hi all, I've re-run the benchmark suite (which works nicely now) and written up some documentation about the performance of lxml in comparison to ET/cET. It's in http://codespeak.net/svn/lxml/trunk/doc/performance.txt If anyone is interested in extending the benchmark suite over other parts of the API or in adding to the comparison, I'd be glad to see some input. :) Have fun, Stefan

1 0

[lxml-dev] Bugs in 1.0.beta
by Olivier Grisel 18 May '06

18 May '06

Hi list, On my box (Ubuntu Linux Dapper Drake on i686 / python2.4) all tests pass but running the bench scripts yields some problems: >>> lxml.etree.LIBXML_COMPILED_VERSION (2, 6, 24) >>> lxml.etree.LIBXSLT_COMPILED_VERSION (1, 1, 15) xpath_extensions_old and xslt_extensions_old trigger the following exceptions: lxe: xpath_extensions_old (-- T1 ) Traceback (most recent call last): File "bench.py", line 652, in ? result = run_bench(bench, *benchmark_setup) File "bench.py", line 606, in run_bench method_call(*args) File "bench.py", line 453, in bench_xpath_extensions_old xpath = self.etree.XPath("child(.)", extensions=extensions) File "xpath.pxi", line 166, in etree.XPath.__init__ File "xpath.pxi", line 53, in etree.XPathEvaluatorBase.__init__ File "xpath.pxi", line 17, in etree._XPathContext.__init__ File "extensions.pxi", line 45, in etree._BaseContext.__init__ TypeError: unindexable object lxe: xslt_extensions_old (-- T1 ) Traceback (most recent call last): File "bench.py", line 652, in ? result = run_bench(bench, *benchmark_setup) File "bench.py", line 606, in run_bench method_call(*args) File "bench.py", line 479, in bench_xslt_extensions_old transform = self.etree.XSLT(tree, extensions) File "xslt.pxi", line 265, in etree.XSLT.__init__ File "xslt.pxi", line 189, in etree._XSLTContext.__init__ File "extensions.pxi", line 45, in etree._BaseContext.__init__ ValueError: unpack sequence of wrong size And append_element and replace_children triggers segmentation faults. The first one crashed after child <Element {bcd}zzzzz at -48285674>. I don't have time to dig further right now. I'll have more time tonight. Best, -- Olivier

2 2

[lxml-dev] lxml 1.0.beta is on cheeseshop
by Stefan Behnel 18 May '06

18 May '06

Hi all, lxml 1.0.beta is available from cheeseshop. This is expected to be the only beta release before the glorious 1.0 will arrive on June 1st (this year). http://cheeseshop.python.org/pypi/lxml The changelog is long and the time-to-market for 1.0 is short, so, as usual, we are happy to receive your bug reports. :) Have fun, Stefan

1 0

[lxml-dev] Lxml 1.0 will be released on Thursday, June 1st
by Stefan Behnel 18 May '06

18 May '06

Hallo everyone, just to keep us from pushing back release dates, Martijn and I have fixed the release date of lxml 1.0 to June 1st. The (long) list of changes for 1.0 is here: http://codespeak.net/svn/lxml/trunk/CHANGES.txt This means for you: If you want a bug-free release ready to install, then come and help us by testing the trunk and reporting any bugs you can find. http://codespeak.net/svn/lxml/trunk Build instructions are here: http://codespeak.net/lxml/build.html It also really helps us if you read the docs and report any difficulties you encounter. The in-development documentation is here: http://codespeak.net/svn/lxml/trunk/doc/ especially http://codespeak.net/svn/lxml/trunk/doc/api.txt http://codespeak.net/svn/lxml/trunk/doc/resolvers.txt and a little bit of http://codespeak.net/svn/lxml/trunk/doc/extensions.txt Any help is appreciated to make lxml 1.0 the best XML tool for Python - ever. :) Have fun, Stefan

4 14

Re: [lxml-dev] What are the equivalents of nowrite, nomkdir options
by Stefan Behnel 18 May '06

18 May '06

Hi Noah, Noah Slater wrote: >> Noah, I'd be glad if you could test it and report back if it works as >> expected. I do not know if the directory creation business works now >> (i.e. if >> directories /are/ created). > > Hi, sorry I can't test until I have a .deb package to install - so I > am guessing I will have to wait until your changes arrive in Debian > unstable. That won't be before 1.0 then, I assume. It's actually not that hard to build Debian packages once they are part of Debian. Get the source .deb via apt and unpack it. Then you can replace the included lxml sources with the current SVN trunk (you may have to make a tgz from it), make sure the Debian package description has the version number you want and then rebuild it (IIRC, the command for that is "dpkg-rebuild" or something in that line). I don't have a Debian system to test, so I can't tell you what exactly you have to do. But there's an even simpler way. You can check out lxml from SVN and build it in-place in the source directory (see doc/build.txt on how to do that). To use it in your program, you can call Python like this: PYTHONPATH=/path/to/lxml-dir/src python myprogram.py To make sure it's the right version, use PYTHONPATH=/path/to/lxml-dir/src python -c \ 'import lxml.etree ; print lxml.etree.__version__' (the '/' makes it one line when you copy and paste it) Stefan

3 3

[lxml-dev] What are the equivalents of nowrite, nomkdir options
by Noah Slater 17 May '06

17 May '06

Hello all, Firstly thanks for such a great library. For a python module I am worryingly excited reading through the documentation. I have had a play around with etree.* and have almost replace 50% of the code in my project thus far to use this - but one concern remains: I cannot figure out from either the docs, snippets of source files I have read or google how I am meant to prevent XSLT from writing to the file system. Using xsltproc from the command line I can use the "--nowrite" and "--nomkdir" switches and would ideally like to be able to replicate this functionality. Thanks so much, Noah -- "Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

4 7

[lxml-dev] new "doc/build.txt" on building lxml from sources
by Stefan Behnel 17 May '06

17 May '06

Hi all, I added a "doc/build.txt" apart from the simplified "INSTALL.txt" to help interested users build lxml from sources (and to prevent normal distrib users from caring about it). It also contains David's procedure on building lxml statically on Windows. http://codespeak.net/svn/lxml/trunk/doc/build.txt It's both in the trunk and the 0.9 branch, so, Martijn, please update the web site from the branch. Regards, Stefan

2 1

[lxml-dev] New CSS style for the lxml website @ codespeak
by Olivier Grisel 16 May '06

16 May '06

Hi list, As I found the current style a bit "too big" I refactored the css to get this: http://champiland.homelinux.net/lxml-newstyle/ If you like it I can check that in the svn. I enclose a patch to list the changes. To Stefan: your work on mail.txt has been done only in branch/lxml-0.9.x and not in trunk. Is it intentional? + it would be nice if your changes could went online. Who is in charge of the publishing process? Regards, -- Olivier Index: style.css =================================================================== --- style.css (revision 27205) +++ style.css (working copy) @@ -1,33 +1,62 @@ body { + /* CSS Hack for IE that does not respect the "margin: auto" rule at the + * document level */ + text-align: center; + padding: 1em; +} + +div.document { + width: 45em; + font: 13px Arial, Verdana, Helvetica, sans-serif; + margin: 1em auto 1em auto; + background-color: white; + color: #222; + text-align: left; +} + +h1.title { background: url(http://codespeak.net/img/codespeak1b.png) no-repeat; - font: 120% Arial, Verdana, Helvetica, sans-serif; - border: 0; - margin: 0.5em 0em 0.5em 0.5em; - padding: 0 0 0 145px; + padding: 20px 0 0 180px; + height: 60px; + font-size: 200%; } -a { - text-decoration: underline; - background-color: transparent; +h1, h2, h3 { + color: #333; + font-weight: bold; } -p { - /*margin: 0.5em 0em 1em 0em;*/ - text-align: left; - line-height: 1.5em; - margin: 0.5em 0em 0em 0em; +h1 { + font-size: 120%; } -p a { - text-decoration: underline; +h2 { + font-size: 110%; } +h3 { + font-size: 105%; +} -p a:active { - color: Red; +a, a:visited { background-color: transparent; + font-weight: bold; + color: Black; + text-decoration: none; } +a:active { + color: Red; + text-decoration: underline; +} + +p { + /*margin: 0.5em 0em 1em 0em;*/ + text-align: justify; + line-height: 1.5em; + margin: 0.5em 0em 0em 0em; +} + hr { clear: both; height: 1px; @@ -35,10 +64,8 @@ background-color: transparent; } - -ul { +ul { line-height: 1.5em; - /*list-style-image: url("bullet.gif"); */ margin-left: 1em; } @@ -47,28 +74,21 @@ margin-left: 0em; } -ul a, ol a { - text-decoration: underline; -} - blockquote { font-family: Times, "Times New Roman", serif; font-style: italic; - font-size: 120%; } code { - font-size: 120%; color: Black; - /*background-color: #dee7ec;*/ background-color: #cccccc; + font-family: Courier, monospace; } pre { - font-size: 120%; - padding: 1em; + padding: 0.5em; border: 1px solid #8cacbb; color: Black; - background-color: #dee7ec; background-color: #cccccc; + font-family: Courier, monospace; } Index: publish.py =================================================================== --- publish.py (revision 27205) +++ publish.py (working copy) @@ -1,9 +1,12 @@ -import os, sys +import os, shutil, sys def publish(dirname, lxml_path, release): if not os.path.exists(dirname): os.mkdir(dirname) - stylesheet_url = 'http://codespeak.net/lxml/style.css' + stylesheet_url = 'style.css' + + shutil.copy(stylesheet_url, dirname) + for name in ['main.txt', 'intro.txt', 'api.txt', 'compatibility.txt', 'extensions.txt', 'namespace_extensions.txt', 'sax.txt']: path = os.path.join(lxml_path, 'doc', name) @@ -22,10 +25,10 @@ os.path.join(dirname, 'index.html')) def rest2html(source_path, dest_path, stylesheet_url): - - command = ('rest2html --stylesheet=%s %s > %s' % + + command = ('rest2html --stylesheet=%s --link-stylesheet %s > %s' % (stylesheet_url, source_path, dest_path)) os.system(command) - + if __name__ == '__main__': publish(sys.argv[1], sys.argv[2], sys.argv[3])

3 9

[lxml-dev] Building lxml on windows: a guide.
by David Sankel 12 May '06

12 May '06

Note: This is my first post on this list. There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down). Unfortunately, there wasn't enough information to make the process of building on windows easy. I wrote this guide to solve both these problems. I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance. I also have a windows installer created for lxml 0.9.1 that is staticly built with iconv-1.9.1, libxml2 2.6.23, libxslt 1.1.15, and zlib 1.2.3. If there is some ftp site or something I can upload it to, in order to give wide access to it, please let me know what to do. David Sankel Building lxml on windows =================== First you'll need to download the latest version of all the required files. Download them all to the same directory. * libxml: Availible from http://codespeak.net/lxml/ * iconv, libxml2, libxslt, and zlib are all availible from xmlsoft.org. The place to go on the ftp site is ftp://xmlsoft.org/libxml2/win32. Your directory should now have something like the following files in it: iconv-1.9.1.win32.zip libxml2-2.6.23.win32.zip libxslt-1.1.15.win32.zip lxml-0.9.1.tgz zlib-1.2.3.win32.zip Now extract each of those files in the _same_ directory. Now you should have something like this: iconv-1.9.1.win32/ iconv-1.9.1.win32.zip libxml2-2.6.23.win32/ libxml2-2.6.23.win32.zip libxslt-1.1.15.win32/ libxslt-1.1.15.win32.zip lxml-0.9.1/ lxml-0.9.1.tgz zlib-1.2.3.win32/ zlib-1.2.3.win32.zip Go to the lxml-0.9.1 directory and edit the Makefile. There should be a section that looks like this:: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = ['-w'] + flags('xslt-config --cflags'), extra_link_args = flags('xslt-config --libs') )], Change it to this (Warning: make sure you are using version numbers that correspond to your downloads) (Note: the _a portion of the libraries means that we are statically linking. If you want to use dlls(why?), link to the dll version of the libraries):: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = [ "-w", "-I..\\libxml2-2.6.23.win32\\include", "-I..\\libxslt-1.1.15.win32\\include", "-I..\\zlib-1.2.3.win32\\include", "-I..\\iconv-1.9.1.win32\\include" ], extra_link_args = [ "..\\libxml2-2.6.23.win32\\lib\\libxml2_a.lib", "..\\libxslt-1.1.15.win32\\lib\\libxslt_a.lib", "..\\zlib-1.2.3.win32\\lib\\zlib.lib", "..\\iconv-1.9.1.win32\\lib\\iconv_a.lib" ] )], Now you should be able to use setup.py and everything should work well. "python setup.py bdist_wininst" will create a windows installer in the pkg directory.

4 12

[lxml-dev] Uche's OT-benchmark on lxml and (c)ElementTree
by Stefan Behnel 12 May '06

12 May '06

Hi all, I just ran a slight variant (doesn't print, builds list) of Uche's OT benchmark on * ElementTree 1.2.6 * cElementTree 1.0.5 * lxml (trunk/SVN 27065, timings for 0.9.2 are similar) The code I used is attached, it runs against the 3.3M ot.xml file from http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip Here's the (somewhat bogus) output on my machine (bench, result, time): bench_ET 120 1.59 bench_cET 120 0.31 bench_lxml_findall 120 0.32 bench_lxml_xpath 120 0.33 bench_lxml_xpath_all 120 0.26 I should note that lxml's findall() uses ElementTree's ElementPath implementation. Here are the real numbers from timeit: # python -m timeit -s "from otbench import *" "bench_ET()" 10 loops, best of 3: 1.36 sec per loop # python -m timeit -s "from otbench import *" "bench_cET()" 10 loops, best of 3: 260 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_findall()" 10 loops, best of 3: 295 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_xpath()" 10 loops, best of 3: 261 msec per loop # python -m timeit -s "from otbench import *" "bench_lxml_xpath_all()" 10 loops, best of 3: 214 msec per loop I think it's pretty interesting how close the timings of cET.findall() and lxml.xpath() are. Regards, Stefan

3 12

[lxml-dev] Python unicode string support in lxml
by Stefan Behnel 12 May '06

12 May '06

Hi all, I had a discussion with Fredrik lately and it convinced me (though not Fredrik) that it would be a good idea to improve the support for Python unicode strings in lxml.etree. I think that unicode strings are the most comfortable way of doing XML I/O from/to strings in Python, so I added support for simply calling unicode() on XML nodes and ElementTrees. It behaves just like tostring(), but always returns Python unicode strings: >>> from lxml import etree >>> uxml = u'<test> \uf8d1 + \uf8d2 </test>' >>> root = etree.XML(uxml) >>> unicode(root) u'<test> \uf8d1 + \uf8d2 </test>' >>> el = etree.Element("test") >>> unicode(el) u'<test/>' >>> subel = etree.SubElement(el, "subtest") >>> unicode(el) u'<test><subtest/></test>' >>> unicode( etree.ElementTree(el) ) u'<test><subtest/></test>' Note that ElementTree does not support this at all. It will raise a parser exception in the XML() call in the second line and return the same generic strings for unicode() as it does for str(). There is a longer doctest in http://codespeak.net/svn/lxml/trunk/doc/api.txt that explains this in more detail. As usual: any comments appreciated. Stefan

6 59

[lxml-dev] tounicode(), again
by Stefan Behnel 11 May '06

11 May '06

Hi all, we had a lengthy discussion yesterday and I guess we found a few use cases where tounicode() makes sense and a few counter-arguments why it might not be a good idea to expose that API at a similarly visible place as tostring(). I'm still convinced that it's a good idea to have that API, but as one of the arguments was that "people who don't understand unicode" (PeWDUUs) would be more likely to write broken code, I added this paragraph to api.txt, in the section that describes the unicode support of lxml. """ Note that the unicode strings returned by ``tounicode()`` never have an XML declaration and therefore do not specify an encoding. This makes it possible to pass them back into the lxml parsers. However, you may have to add a declaration yourself if you want to serialize such a unicode string to a byte stream later. In contrast, the ``tostring()`` function automatically adds a declaration as needed that reflects the encoding of the returned byte string. """ I hope that makes it clear enough for PeWDUUs what the advantage of using tostring() over tounicode() is and that you have to take care what you do with unicode strings. So, I propose leaving the API (and implementation) just as it is now. Regards, Stefan

2 1

[lxml-dev] lxml 0.9.2 released
by Stefan Behnel 10 May '06

10 May '06

Hello everyone, I just uploaded another bug fix release of the 0.9 branch to the Cheese shop. http://cheeseshop.python.org/pypi/lxml http://cheeseshop.python.org/packages/source/l/lxml/lxml-0.9.2.tar.gz This will (hopefully) be the last 0.9.x release before 1.0. It fixes a number of more or less annoying bugs and adds a number of smaller features. All the major, new, fancy features will go into the 1.0 release. Regards, Stefan Features added * Speedup for Element.makeelement(): the new element now reuses the original libxml2 document instead of creating a new empty one * Speedup for reversed() iteration over element children (Py2.4+ only) * ElementTree compatible QName class * RelaxNG and XMLSchema now accept any Element, not only ElementTrees Bugs fixed * str(xslt_result) was broken for XSLT output other than UTF-8 * Memory leak if write_c14n fails to write the file after conversion * Crash in XMLSchema and RelaxNG when passing non-schema documents * Memory leak in RelaxNG() when RelaxNGParseError is raised

1 0

[lxml-dev] The difference between str(xslt_result) and xslt_result.write()
by Matt Patterson 10 May '06

10 May '06

Hi, Is there a difference between the output from lxml.etree._XSLTResultTree.__str__() and lxml.etree._XSLTResultTree.write()? I'm trying to chase some output whitespace issues around, and I'm wondering if there's a difference in how serialisation is handled. Also, str(_XSLTResultTree) is failing for me with errors like "'ascii' codec can't encode character u'\xa9' in position 1608: ordinal not in range(128)" because I have unicode characters in my documents... lxml.etree._XSLTResultTree doesn't define a __unicode__ method so I can't use a unicode coercion to UTF-8 or something like that... All in all though, I'm really enjoying lxml: I spent a long time working with libxml & lbxslt's standard python interfaces and they much more of a pain than lxml! Thanks, Matt -- Matt Patterson | Design & Code <matt at reprocessed org> | http://www.reprocessed.org/

2 2

[lxml-dev] I/O benchmarks
by Stefan Behnel 08 May '06

08 May '06

Hi everyone, have you ever wondered why you should use lxml instead of cElementTree? Here is why. :) Stefan Preparing test suites and trees ... Running benchmark on lxe, ET, cET Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.1180 0.1158 0.1153 0.1178 0.1157 0.1173 T2: 0.1186 0.1202 0.1207 0.1233 0.1236 0.1250 T3: 0.0323 0.0252 0.0250 0.0489 0.0495 0.0492 T4: 0.0005 0.0005 0.0005 0.0010 0.0010 0.0010 ET : -- S- U- -A SA UA T1: 0.2305 0.2887 0.2193 0.2662 0.2872 0.2259 T2: 0.3038 0.3442 0.2823 0.3140 0.2364 0.4051 T3: 0.0534 0.0572 0.0523 0.0583 0.0553 0.0829 T4: 0.0010 0.0008 0.0007 0.0009 0.0008 0.0008 cET: -- S- U- -A SA UA T1: 0.0369 0.0353 0.0371 0.0341 0.0354 0.0345 T2: 0.0370 0.0364 0.0361 0.0371 0.0369 0.0358 T3: 0.0090 0.0091 0.0090 0.0125 0.0174 0.0235 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 lxe: tostring_utf16 (S- T1 ) 25.0344 24.7066 24.6755 msec/pass, best: 24.6755 ET : tostring_utf16 (S- T1 ) 715.0391 668.6494 668.2270 msec/pass, best: 668.2270 cET: tostring_utf16 (S- T1 ) 631.4080 634.0269 629.3236 msec/pass, best: 629.3236 lxe: tostring_utf16 (U- T1 ) 25.3342 25.4860 25.1003 msec/pass, best: 25.1003 ET : tostring_utf16 (U- T1 ) 666.5964 670.9383 664.8268 msec/pass, best: 664.8268 cET: tostring_utf16 (U- T1 ) 628.6623 628.9270 636.1863 msec/pass, best: 628.6623 lxe: tostring_utf16 (S- T2 ) 38.5328 27.5526 28.3839 msec/pass, best: 27.5526 ET : tostring_utf16 (S- T2 ) 696.8423 697.2251 698.0101 msec/pass, best: 696.8423 cET: tostring_utf16 (S- T2 ) 655.5692 652.2847 653.9454 msec/pass, best: 652.2847 lxe: tostring_utf16 (U- T2 ) 26.2363 26.8365 26.6303 msec/pass, best: 26.2363 ET : tostring_utf16 (U- T2 ) 698.3589 697.7249 698.7069 msec/pass, best: 697.7249 cET: tostring_utf16 (U- T2 ) 652.6794 652.6194 652.4247 msec/pass, best: 652.4247 lxe: tostring_utf16 (S- T3 ) 2.9159 2.9798 3.0483 msec/pass, best: 2.9159 ET : tostring_utf16 (S- T3 ) 90.1106 90.2723 90.6772 msec/pass, best: 90.1106 cET: tostring_utf16 (S- T3 ) 75.9803 75.7933 75.6844 msec/pass, best: 75.6844 lxe: tostring_utf16 (U- T3 ) 2.9326 2.9501 3.8947 msec/pass, best: 2.9326 ET : tostring_utf16 (U- T3 ) 90.8508 90.2919 90.4195 msec/pass, best: 90.2919 cET: tostring_utf16 (U- T3 ) 75.8780 75.6118 75.6281 msec/pass, best: 75.6118 lxe: tostring_utf16 (S- T4 ) 0.1184 0.1246 0.1174 msec/pass, best: 0.1174 ET : tostring_utf16 (S- T4 ) 4.0772 4.1340 4.0832 msec/pass, best: 4.0772 cET: tostring_utf16 (S- T4 ) 8.6555 3.7591 3.8131 msec/pass, best: 3.7591 lxe: tostring_utf16 (U- T4 ) 0.1150 0.1142 0.1470 msec/pass, best: 0.1142 ET : tostring_utf16 (U- T4 ) 4.1377 4.1255 4.0694 msec/pass, best: 4.0694 cET: tostring_utf16 (U- T4 ) 8.5306 3.7888 3.8399 msec/pass, best: 3.7888 lxe: tostring_utf8 (S- T1 ) 20.1342 20.8913 22.0570 msec/pass, best: 20.1342 ET : tostring_utf8 (S- T1 ) 658.5292 659.8311 659.2222 msec/pass, best: 658.5292 cET: tostring_utf8 (S- T1 ) 613.7725 616.7276 615.3487 msec/pass, best: 613.7725 lxe: tostring_utf8 (U- T1 ) 22.3571 21.6062 22.3468 msec/pass, best: 21.6062 ET : tostring_utf8 (U- T1 ) 658.4980 658.8312 659.7990 msec/pass, best: 658.4980 cET: tostring_utf8 (U- T1 ) 619.8824 620.7493 618.3270 msec/pass, best: 618.3270 lxe: tostring_utf8 (S- T2 ) 22.8893 23.1579 22.3014 msec/pass, best: 22.3014 ET : tostring_utf8 (S- T2 ) 695.0778 686.7107 686.9514 msec/pass, best: 686.7107 cET: tostring_utf8 (S- T2 ) 644.8464 645.2512 645.4735 msec/pass, best: 644.8464 lxe: tostring_utf8 (U- T2 ) 21.8574 21.5462 21.9939 msec/pass, best: 21.5462 ET : tostring_utf8 (U- T2 ) 689.5708 689.1687 685.5371 msec/pass, best: 685.5371 cET: tostring_utf8 (U- T2 ) 643.3239 644.8895 641.5675 msec/pass, best: 641.5675 lxe: tostring_utf8 (S- T3 ) 2.3862 2.2847 2.3411 msec/pass, best: 2.2847 ET : tostring_utf8 (S- T3 ) 91.7382 89.4661 90.1840 msec/pass, best: 89.4661 cET: tostring_utf8 (S- T3 ) 76.4675 74.7183 74.7682 msec/pass, best: 74.7183 lxe: tostring_utf8 (U- T3 ) 2.3286 2.4161 2.3287 msec/pass, best: 2.3286 ET : tostring_utf8 (U- T3 ) 89.9778 92.0483 91.0223 msec/pass, best: 89.9778 cET: tostring_utf8 (U- T3 ) 74.8817 74.8538 74.9363 msec/pass, best: 74.8538 lxe: tostring_utf8 (S- T4 ) 0.1028 0.1006 0.1058 msec/pass, best: 0.1006 ET : tostring_utf8 (S- T4 ) 8.9541 4.1052 4.0422 msec/pass, best: 4.0422 cET: tostring_utf8 (S- T4 ) 8.5588 3.7166 3.7450 msec/pass, best: 3.7166 lxe: tostring_utf8 (U- T4 ) 0.1178 0.1000 0.1040 msec/pass, best: 0.1000 ET : tostring_utf8 (U- T4 ) 4.0836 4.0845 4.0257 msec/pass, best: 4.0257 cET: tostring_utf8 (U- T4 ) 8.5069 3.7946 3.7505 msec/pass, best: 3.7505 lxe: tostring_utf8_unicode_XML (S- T1 ) 217.3858 194.6370 196.9607 msec/pass, best: 194.6370 ET : tostring_utf8_unicode_XML (S- T1 ) 1044.2196 983.0496 1006.1930 msec/pass, best: 983.0496 cET: tostring_utf8_unicode_XML (S- T1 ) 684.9565 679.0308 674.1449 msec/pass, best: 674.1449 lxe: tostring_utf8_unicode_XML (U- T1 ) 203.8817 203.9280 200.4173 msec/pass, best: 200.4173 ET : tostring_utf8_unicode_XML (U- T1 ) 981.1272 977.5449 978.9895 msec/pass, best: 977.5449 cET: tostring_utf8_unicode_XML (U- T1 ) 653.3920 655.3704 651.0385 msec/pass, best: 651.0385 lxe: tostring_utf8_unicode_XML (S- T2 ) 208.8770 211.5672 210.4380 msec/pass, best: 208.8770 ET : tostring_utf8_unicode_XML (S- T2 ) 1021.3773 1020.3279 1041.1114 msec/pass, best: 1020.3279 cET: tostring_utf8_unicode_XML (S- T2 ) 688.2031 681.9181 680.2590 msec/pass, best: 680.2590 lxe: tostring_utf8_unicode_XML (U- T2 ) 209.5904 210.3374 209.5674 msec/pass, best: 209.5674 ET : tostring_utf8_unicode_XML (U- T2 ) 1027.9991 1022.0539 1022.0318 msec/pass, best: 1022.0318 cET: tostring_utf8_unicode_XML (U- T2 ) 681.4484 678.8596 680.9469 msec/pass, best: 678.8596 lxe: tostring_utf8_unicode_XML (S- T3 ) 12.5315 12.0217 11.9403 msec/pass, best: 11.9403 ET : tostring_utf8_unicode_XML (S- T3 ) 202.9581 203.0447 204.2288 msec/pass, best: 202.9581 cET: tostring_utf8_unicode_XML (S- T3 ) 82.3561 82.4286 82.1372 msec/pass, best: 82.1372 lxe: tostring_utf8_unicode_XML (U- T3 ) 11.6045 11.8548 11.7764 msec/pass, best: 11.6045 ET : tostring_utf8_unicode_XML (U- T3 ) 201.2770 201.0283 202.3813 msec/pass, best: 201.0283 cET: tostring_utf8_unicode_XML (U- T3 ) 82.3942 82.6146 82.3579 msec/pass, best: 82.3579 lxe: tostring_utf8_unicode_XML (S- T4 ) 5.3620 0.4832 0.5012 msec/pass, best: 0.4832 ET : tostring_utf8_unicode_XML (S- T4 ) 6.4547 6.3800 6.3898 msec/pass, best: 6.3800 cET: tostring_utf8_unicode_XML (S- T4 ) 8.8447 3.9611 4.0390 msec/pass, best: 3.9611 lxe: tostring_utf8_unicode_XML (U- T4 ) 5.2781 0.4689 0.4798 msec/pass, best: 0.4689 ET : tostring_utf8_unicode_XML (U- T4 ) 6.5955 6.4135 6.4196 msec/pass, best: 6.4135 cET: tostring_utf8_unicode_XML (U- T4 ) 8.7307 4.0647 4.0202 msec/pass, best: 4.0202 lxe: write_utf8_parse_stringIO (S- T1 ) 198.0141 188.3298 190.1872 msec/pass, best: 188.3298 ET : write_utf8_parse_stringIO (S- T1 ) 1143.8117 1176.7961 1152.1253 msec/pass, best: 1143.8117 cET: write_utf8_parse_stringIO (S- T1 ) 814.6583 847.3877 810.7611 msec/pass, best: 810.7611 lxe: write_utf8_parse_stringIO (U- T1 ) 194.5397 195.0845 194.7964 msec/pass, best: 194.5397 ET : write_utf8_parse_stringIO (U- T1 ) 1153.3167 1142.9091 1146.2344 msec/pass, best: 1142.9091 cET: write_utf8_parse_stringIO (U- T1 ) 810.8662 808.3027 811.9326 msec/pass, best: 808.3027 lxe: write_utf8_parse_stringIO (S- T2 ) 205.9781 202.8390 202.2034 msec/pass, best: 202.2034 ET : write_utf8_parse_stringIO (S- T2 ) 1195.9166 1195.7337 1193.5966 msec/pass, best: 1193.5966 cET: write_utf8_parse_stringIO (S- T2 ) 846.5763 851.8866 848.6367 msec/pass, best: 846.5763 lxe: write_utf8_parse_stringIO (U- T2 ) 203.5371 202.8143 204.4644 msec/pass, best: 202.8143 ET : write_utf8_parse_stringIO (U- T2 ) 1218.4967 1245.6663 1255.3381 msec/pass, best: 1218.4967 cET: write_utf8_parse_stringIO (U- T2 ) 858.2268 849.1432 858.5724 msec/pass, best: 849.1432 lxe: write_utf8_parse_stringIO (S- T3 ) 17.0564 11.5748 12.5815 msec/pass, best: 11.5748 ET : write_utf8_parse_stringIO (S- T3 ) 235.0894 234.2171 233.4405 msec/pass, best: 233.4405 cET: write_utf8_parse_stringIO (S- T3 ) 113.0575 112.7246 112.9978 msec/pass, best: 112.7246 lxe: write_utf8_parse_stringIO (U- T3 ) 11.1004 11.0245 11.3229 msec/pass, best: 11.0245 ET : write_utf8_parse_stringIO (U- T3 ) 238.6545 243.0529 251.3970 msec/pass, best: 238.6545 cET: write_utf8_parse_stringIO (U- T3 ) 122.8126 115.6941 120.9536 msec/pass, best: 115.6941 lxe: write_utf8_parse_stringIO (S- T4 ) 0.4456 0.4642 0.4494 msec/pass, best: 0.4456 ET : write_utf8_parse_stringIO (S- T4 ) 8.0726 8.1274 7.9427 msec/pass, best: 7.9427 cET: write_utf8_parse_stringIO (S- T4 ) 6.0532 5.0335 5.0780 msec/pass, best: 5.0335 lxe: write_utf8_parse_stringIO (U- T4 ) 0.4778 0.4506 0.4657 msec/pass, best: 0.4506 ET : write_utf8_parse_stringIO (U- T4 ) 7.4578 7.3894 8.0762 msec/pass, best: 7.3894 cET: write_utf8_parse_stringIO (U- T4 ) 6.2156 5.5292 5.0383 msec/pass, best: 5.0383

1 0

Re: [lxml-dev] HTMLParser behavior - ill formed HTML
by Stefan Behnel 04 May '06

04 May '06

Hi Dean, Dean Pavlekovic wrote: > On 5/2/06, Stefan Behnel <behnel_ml(a)gkec.informatik.tu-darmstadt.de> wrote: > >> I changed the trunk to always accept ill-formed results if the recover >> option is set and no lxml-internal errors were raised. Please try if that >> helps. I couldn't come up with a sufficiently short example of broken >> HTML where this problem occurs, so I couldn't test it. The examples that >> were tested so far can be found in src/lxml/tests/test_htmlparser.py. > > Unfortunately it's not working and I'm sorry but I didn't have the time to > look at it deeper. It appears that the value of options is changed > somewhere in libxml context during the htmlCtxtReadFile. I printed out the > value of parser's self._parse_options before (97, that is > XML_PARSE_RECOVER_FLAG set) and the value of ctxt.options after the > htmlCtxtReadFile call which was 96 meaning the flag was reset by libxml. Interesting. Could you tell me what version of libxml2 you are using (see below)? My guess is that it's older than 2.6.21. Libxml2 copies the options by hand, so if the RECOVER option is unknown, it will not turn up in the context options. That makes me wonder why recovery worked for Paul on 2.6.16 in the first place... Anyway. I changed the trunk to pass the option explicitly to _handleParseResult so that we no longer rely on libxml2. Please test if this works for you now. I'd be glad if you could come up with a short piece of broken HTML code that triggers the "not well formed" case in recovery mode. That would allow us to set up a test case to check that it actually works (and keeps working). To check which versions you are using, I added module attributes "LXML_VERSION", "LIBXML_VERSION" and "LIBXSLT_VERSION" that carry tuples containing the respective versions used by lxml, e.g. (2, 6, 23) for "2.6.23". "LIBXML_COMPILED_VERSION" and "LIBXSLT_COMPILED_VERSION" show against which versions lxml was compiled. BTW, these attributes are mainly meant for debugging purposes. Since they will first (officially) appear in lxml 1.0, code that wants to use them anyway will have to check if they are available (hasattr or try-except) before accessing them. All of them appeared at the same time, so if one is there, the others will be there, too. > Btw. is there a reason that the XML_... enums are assigned an explicit > value in xmlparser.pxd and the same is not done for HTML_... enums in > htmlparser.pxd? I'm not familiar with Pyrex and wheter it parses c/c++ .h > files to match values... Pyrex doesn't care about the values of enums and C uses the .h files directly, so, since they are just copy&pasted into the .pxd files, they sometimes have numbers and sometimes not. Stefan

2 2

[lxml-dev] 'docinfo' property on ElementTree
by Stefan Behnel 02 May '06

02 May '06

Hi all, I updated the trunk to provide ElementTree objects with access to the document information provided by the parser: DOCTYPE, XML version and original encoding. Paul Everitt had some use cases related to the HTML parser, but I think it's generally a good idea to make this kind of information available. The new API works as follows: >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN" >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url) >>> xml_header = '<?xml version="1.0" encoding="ascii"?>' >>> xhtml = xml_header + doctype_string + '<html><body></body></html>' >>> et = lxml.etree.parse(StringIO(xhtml)) >>> docinfo = et.docinfo >>> print docinfo.public_id -//W3C//DTD XHTML 1.0 Transitional//EN >>> print docinfo.system_url http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd >>> docinfo.doctype == doctype_string True >>> print docinfo.xml_version 1.0 >>> print docinfo.encoding ascii This is backed by a DocInfo object that you can also instantiate on an ElementTree (or Element) by hand. The docinfo property just does it for you. Any of the attributes above may be None if the information is not available. Stefan

1 0

[lxml-dev] HTMLParser behavior - ill formed HTML
by Dean Pavlekovic 02 May '06

02 May '06

Hi all, My name is Dean and I've recently joined the lxml-dev to watch how things are going most notably since the HTML parser has been added to the trunk. We (my colleague and I) have been using lxml in the past for processing of rather large DocBook type XML docs aplit across multiple files and in mixed namespaces without any problems and the experiences with lxml have been nothing but good, so let me first congratulate the developers on a great work lxml is. Lately I've looked into using the lxml trunk revision for analyzing some HTML (not well-formed). I've stumbled upon the feature for which I'm not sure whether it's intended to be. When I feed an ill-formed HTML to a parser via > doc=etree.parse('afile.html', parser=etree.HTMLParser()) #recover=True by default an exception is raised and the function yields no result. When I use libxml2 directly: > doc=libxml2.htmlParseFile('afile.html', None) libxml2 prints out some errors/warnings but I _do_ get a reference to a document, which can normally be used. I've also tried to use the procedure as in parser.pxi i.e. htmlCreateMemoryParserCtxt() and htmlCtxtReadDoc() with the same results. So I tracked this behavior down to calling _handleParseResult(pctxt, result, NULL) at the end of parse* methods in HTMLParser in parser.pxi, note the 'if.ctxt.wellFormed' part:  where the document is destroyed if libxml2's context wellFormed flag is not set. I checked this by calling libxml2 htmlCtxtReadDoc() directly on that document and indeed the wellFormed flag turned out to be 0. Now, shouldn't the HTMLParser also return the document reference in this case if recover=True flag is specified since libxml apparently does not have problems with that. I've checked this by modifying the _handleParseResult with 'accept_ill_formed' argument. If that flag is set, ctxt.wellFormed would be ignored. Also modified _handleParseResult calls in HTMLParser's parse... methods to specify accept flag if 'recover' was set in the constructor. This turned to work just well and I was able to navigate the document with xpath, even the errors (using '&' in href attributes) were corrected. So the question would be is the present behavior correct due to something I'm possibly missing? I think it should be dependent on whether RECOVER flags have been specified or not. Best regards, Dean

2 1