Mailman 3 March 2006 - lxml - The Python XML Toolkit

[lxml-dev] Lxml 0.9 is out!
by Stefan Behnel 21 Mar '06

21 Mar '06

Hello everyone, after almost five months of hacking, lxml 0.9 has finally seen the light of night. :) It has tons of new fancy features and several serious bug fixes. See the ChangeLog for details: http://codespeak.net/lxml/changes-0.9.html and the documentation for examples of the new features: http://codespeak.net/lxml/api.html http://codespeak.net/lxml/sax.html http://codespeak.net/lxml/namespace_extensions.html http://codespeak.net/lxml/extensions.html Installation has just become easier (at least on Linux): http://codespeak.net/lxml/installation.html Give it a try and keep spreading the word! Stefan

3 6

[lxml-dev] Better error reporting
by Stefan Behnel 16 Mar '06

16 Mar '06

Hi, since we often have requests regarding better error reporting in lxml, I implemented a little error log. lxml now keeps a bounded list of output messages from libxml2/xslt (that normally appear on stdout) and provides access to the log entries through its exceptions. LxmlException objects now have an additional attribute "error_log" that contain the last log entries (up to 20 by default). So, if an exception is raised, whoever catches it can print out the error messages from libxml2/xslt to see where things went wrong. This behaviour can be switched on or off at compile time and defaults to off. If off, the attribute simply stores an empty tuple. I hope that comes close to what was asked for. Stefan

2 6

[lxml-dev] New implementation of element.getiterator()
by Stefan Behnel 15 Mar '06

15 Mar '06

Hi all, element.getiterator() now has a real iterator implementation (as opposed to the previous list builder). It is similar to the already existing child iterator in that it keeps Python references to nodes to benefit from Python garbage collection. There are two new iterator classes: * ElementDepthFirstIterator - iterates over a node and all its subelements in depth first pre-order (i.e. document order) * ElementTagFilter - filters out elements from an iterable that do not match a specific tag The combination of both implements the getiterator() functionality. This is less efficient than a single combined iterator could be, as creating Python objects just to throw them away afterwards does not come for free in lxml. However, I think that both iterators are useful in their own right, so I split them up for now, mainly to avoid code duplication. I added the following benchmarks to show the gain compared to the original implementation (see the attached logs): def bench_getiterator(self, root): list(islice(root.getiterator(), 10, 110)) def bench_getiterator_tag(self, root): list(islice(root.getiterator("{b}a"), 3, 10)) def bench_getiterator_tag_all(self, root): list(islice(root.getiterator("{b}a"), 10, 150)) Note that I had to change the tree setup to get sufficiently repetitive tag names for these tests. The second log I attached compares the new implementation to ElementTree and cElementTree. lxml now beats ET in all of the above tests and even cET in most cases. Maybe I should still consider removing the double iterator handicap... Have fun, Stefan Preparing test suites and trees ... Running benchmark on lxe, ET, cET Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.1967 0.1532 0.1738 0.1522 0.1516 0.1511 T2: 0.1615 0.1575 0.1600 0.1603 0.1617 0.1599 T3: 0.0308 0.0294 0.0294 0.0476 0.0494 0.0484 T4: 0.0007 0.0007 0.0007 0.0011 0.0011 0.0012 ET : -- S- U- -A SA UA T1: 0.2303 0.2816 0.2180 0.2715 0.2156 0.2566 T2: 0.2992 0.2313 0.2859 0.2315 0.2774 0.3093 T3: 0.0524 0.0548 0.0519 0.0580 0.0536 0.0594 T4: 0.0009 0.0008 0.0012 0.0008 0.0008 0.0008 cET: -- S- U- -A SA UA T1: 0.0361 0.0354 0.0351 0.0350 0.0373 0.0352 T2: 0.0358 0.0362 0.0375 0.0368 0.0360 0.0364 T3: 0.0086 0.0086 0.0085 0.0122 0.0121 0.0152 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 lxe: getiterator (-- T1 ) 5.4358 5.6172 5.6089 msec/pass, best: 5.4358 ET : getiterator (-- T1 ) 43.8617 43.9253 44.0674 msec/pass, best: 43.8617 cET: getiterator (-- T1 ) 0.3359 0.3485 0.3402 msec/pass, best: 0.3359 lxe: getiterator (-- T2 ) 7.5207 10.2384 9.3765 msec/pass, best: 7.5207 ET : getiterator (-- T2 ) 45.6706 45.9193 46.3493 msec/pass, best: 45.6706 cET: getiterator (-- T2 ) 0.3397 0.3491 0.3451 msec/pass, best: 0.3397 lxe: getiterator (-- T3 ) 0.2269 0.2267 0.2315 msec/pass, best: 0.2267 ET : getiterator (-- T3 ) 11.0083 11.2267 11.2683 msec/pass, best: 11.0083 cET: getiterator (-- T3 ) 0.4715 0.3912 0.3913 msec/pass, best: 0.3912 lxe: getiterator (-- T4 ) 4.5090 0.1437 0.1446 msec/pass, best: 0.1437 ET : getiterator (-- T4 ) 0.2379 0.2461 0.2298 msec/pass, best: 0.2298 cET: getiterator (-- T4 ) 0.2123 0.1953 0.1979 msec/pass, best: 0.1953 lxe: getiterator_tag (-- T1 ) 31.7239 32.0591 32.7678 msec/pass, best: 31.7239 ET : getiterator_tag (-- T1 ) 37.6585 36.3009 36.3366 msec/pass, best: 36.3009 cET: getiterator_tag (-- T1 ) 30.9611 31.2982 32.2575 msec/pass, best: 30.9611 lxe: getiterator_tag (-- T2 ) 0.6885 0.7165 0.7245 msec/pass, best: 0.6885 ET : getiterator_tag (-- T2 ) 37.0500 37.3288 37.3845 msec/pass, best: 37.0500 cET: getiterator_tag (-- T2 ) 0.6012 0.6058 0.6023 msec/pass, best: 0.6012 lxe: getiterator_tag (-- T3 ) 6.5275 6.5976 6.5382 msec/pass, best: 6.5275 ET : getiterator_tag (-- T3 ) 9.1281 9.3763 11.5126 msec/pass, best: 9.1281 cET: getiterator_tag (-- T3 ) 7.5623 7.5396 7.4457 msec/pass, best: 7.4457 lxe: getiterator_tag (-- T4 ) 0.1903 0.1859 0.1840 msec/pass, best: 0.1840 ET : getiterator_tag (-- T4 ) 0.1951 0.1996 0.1955 msec/pass, best: 0.1951 cET: getiterator_tag (-- T4 ) 4.3555 0.1929 0.1904 msec/pass, best: 0.1904 lxe: getiterator_tag_all (-- T1 ) 33.6318 34.9976 33.9690 msec/pass, best: 33.6318 ET : getiterator_tag_all (-- T1 ) 36.3718 36.0605 35.9895 msec/pass, best: 35.9895 cET: getiterator_tag_all (-- T1 ) 31.7803 31.8099 32.6252 msec/pass, best: 31.7803 lxe: getiterator_tag_all (-- T2 ) 34.8316 34.5458 36.4856 msec/pass, best: 34.5458 ET : getiterator_tag_all (-- T2 ) 36.9701 38.3700 37.3110 msec/pass, best: 36.9701 cET: getiterator_tag_all (-- T2 ) 34.0031 32.9882 33.7108 msec/pass, best: 32.9882 lxe: getiterator_tag_all (-- T3 ) 6.6296 6.5843 6.6435 msec/pass, best: 6.5843 ET : getiterator_tag_all (-- T3 ) 9.1701 9.3035 9.5170 msec/pass, best: 9.1701 cET: getiterator_tag_all (-- T3 ) 11.8023 7.2707 7.4327 msec/pass, best: 7.2707 lxe: getiterator_tag_all (-- T4 ) 4.6077 0.1829 0.1837 msec/pass, best: 0.1829 ET : getiterator_tag_all (-- T4 ) 0.1940 0.1847 0.1873 msec/pass, best: 0.1847 cET: getiterator_tag_all (-- T4 ) 0.2014 0.1895 0.1882 msec/pass, best: 0.1882 Preparing test suites and trees ... Running benchmark on lxe Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.2061 0.1642 0.1633 0.1646 0.1645 0.1805 T2: 0.1709 0.1703 0.1699 0.1737 0.1734 0.1740 T3: 0.0324 0.0314 0.0320 0.0504 0.0511 0.0510 T4: 0.0008 0.0008 0.0008 0.0012 0.0012 0.0012 lxe: getiterator (-- T1 ) 80.8973 77.7289 77.4245 msec/pass, best: 77.4245 lxe: getiterator (-- T2 ) 84.2623 85.7539 85.2087 msec/pass, best: 84.2623 lxe: getiterator (-- T3 ) 21.5209 17.3502 18.8891 msec/pass, best: 17.3502 lxe: getiterator (-- T4 ) 0.4163 0.4116 0.4997 msec/pass, best: 0.4116 lxe: getiterator_tag (-- T1 ) 79.2238 81.4558 79.4984 msec/pass, best: 79.2238 lxe: getiterator_tag (-- T2 ) 86.4460 86.6350 86.4704 msec/pass, best: 86.4460 lxe: getiterator_tag (-- T3 ) 22.5506 18.3274 18.5131 msec/pass, best: 18.3274 lxe: getiterator_tag (-- T4 ) 4.6912 0.4282 0.4641 msec/pass, best: 0.4282 lxe: getiterator_tag_all (-- T1 ) 85.4131 83.0776 85.7234 msec/pass, best: 83.0776 lxe: getiterator_tag_all (-- T2 ) 84.7853 83.9738 84.1555 msec/pass, best: 83.9738 lxe: getiterator_tag_all (-- T3 ) 22.0715 17.8837 17.7980 msec/pass, best: 17.7980 lxe: getiterator_tag_all (-- T4 ) 0.4290 0.4228 0.4179 msec/pass, best: 0.4179

1 0

[lxml-dev] XPath error ?
by Fabien SCHWOB 13 Mar '06

13 Mar '06

Hello, I've successfully compiled and installed lxml, but now I have XPath errors. To be precise, the error is that no nodes are returned. I'm using "//a" on "http://www.w3.org/". It's seems to be an namespace error but I haven't found how to give to the .xpath() function the default namespace. Here is my code : ==================== 8< ======================= import lxml.etree import urllib import sys def show_skel(element, ident = 0): print " "*ident, element.tag for node in element: show_skel(node, ident+4) url = "http://www.w3.org" expr_xpath = "//a" f = lxml.etree.StringIO("".join(urllib.urlopen(url).readlines())) doc = lxml.etree.parse(f) r = doc.xpath(expr_xpath) #show_skel(doc.getroot()) print len(r) if (len(r) > 0): for node in r: print node.tag ==================== 8< ======================= Thanks in advance

5 7

[lxml-dev] Fedora Core 4 and lxml
by Fabien SCHWOB 10 Mar '06

10 Mar '06

I'm trying to build lxml on FC4. I've already installed Pyrex and applied the gcc4-small-patch. But when I'm trying to build lxml with 'python setup.py install' or 'make', I get the following error : building 'lxml.etree' extension Traceback (most recent call last): File "setup.py", line 29, in ? cmdclass = {'build_ext': build_pyx} File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/install.py", line 506, in run self.run_command('build') File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/build_ext.py", line 279, in run self.build_extensions() File "/usr/lib/python2.4/distutils/command/build_ext.py", line 405, in build_extensions self.build_extension(ext) File "/usr/lib/python2.4/distutils/command/build_ext.py", line 442, in build_extension sources = self.swig_sources(sources, ext) TypeError: swig_sources() takes exactly 2 arguments (3 given) Does someone has an idea on how to solve this problem ? Thanks in advance. -- Fabien

4 9

[lxml-dev] New benchmark results
by Stefan Behnel 10 Mar '06

10 Mar '06

Here, we go, this is *much* better. I looked over one of the bottlenecks in the API, the _elementFactory function. I found that the main problem was my implementation of the namespace element lookup, which translated to extremely inefficient C code. I rewrote that with explicit Python API calls and that gives us between 10% and 400% speed up in the benchmarks, depending on the API intensiveness of the tests. Note that lxml is faster than ElementTree in the benchmark tree setup now. Now, the main area where lxml is substantially slower than ElementTree is when moving elements between documents by hand. This is rare and can be avoided pretty often, I guess. Note that moving an entire XML tree to a new document in the obvious one-step operation is *not* so expensive in lxml. What matters is the overhead of creating a Python representation of elements, which is purely API related. Everything that runs internally runs mainly at the speed of libxml2. A special beauty here is the deepcopy benchmark. There still is a certain overhead in the API-to-libxml2 mapping, so programs that extensively create elements one after the other are better off with ElementTree and cElementTree. This advantage already diminishes when SubElement is used, for which lxml is now faster than ElementTree. In all other cases, I wouldn't see a major advantage for the two. Especially none that should keep people from using lxml. Depending on the tree size/structure and the operation, lxml and cElementTree can both take the lead. Since the APIs are largely compatible, I see the main decision maker as follows: * If your problem is small and not performance critical, use whatever you like * If you create tons of independent elements, use cElementTree or consider changing your algorithms :) * If your program is mainly API bound, use cElementTree * If your program is dominated by large XML structures, try both lxml and cElementTree * If you often need to copy large trees, consider lxml * If you need features like XPath, XSLT, RNG, custom APIs, ... - use lxml * If you need those features and more speed, send a patch to this list Have fun, Stefan Preparing test suites and trees ... Running benchmark on etree, ElementTree, cElementTree Setup times for trees in seconds: etree : 0.1827, 0.1651, 0.0371, 0.0008 ElementTree : 0.2251, 0.2564, 0.0574, 0.0010 cElementTree : 0.0378, 0.0176, 0.0088, 0.0001 etree append_elements (T1 ) 2.1396 2.2991 2.2673 msec/pass, best: 2.1396 ElementTree append_elements (T1 ) 0.3253 0.3294 0.3451 msec/pass, best: 0.3253 cElementTree append_elements (T1 ) 0.1065 0.0814 0.0795 msec/pass, best: 0.0795 etree append_elements (T2 ) 9.5490 9.8586 9.8201 msec/pass, best: 9.5490 ElementTree append_elements (T2 ) 4.9283 4.9106 4.9252 msec/pass, best: 4.9106 cElementTree append_elements (T2 ) 0.7983 0.8127 0.8101 msec/pass, best: 0.7983 etree append_elements (T3 ) 0.1166 0.1110 0.1099 msec/pass, best: 0.1099 ElementTree append_elements (T3 ) 0.1056 0.1081 0.1097 msec/pass, best: 0.1056 cElementTree append_elements (T3 ) 0.0308 0.0326 0.0314 msec/pass, best: 0.0308 etree append_elements (T4 ) 0.6297 0.4669 0.4662 msec/pass, best: 0.4662 ElementTree append_elements (T4 ) 0.2243 0.2092 0.2179 msec/pass, best: 0.2092 cElementTree append_elements (T4 ) 0.0375 0.0376 0.0355 msec/pass, best: 0.0355 etree append_from_document (T1,T2 ) 17.5267 12.2076 12.4225 msec/pass, best: 12.2076 ElementTree append_from_document (T1,T2 ) 7.2593 8.3792 8.1089 msec/pass, best: 7.2593 cElementTree append_from_document (T1,T2 ) 0.4959 0.5207 0.4878 msec/pass, best: 0.4878 etree append_from_document (T1,T3 ) 0.8263 0.8588 0.8604 msec/pass, best: 0.8263 ElementTree append_from_document (T1,T3 ) 0.0907 0.0806 0.0843 msec/pass, best: 0.0806 cElementTree append_from_document (T1,T3 ) 0.0324 0.0324 0.0348 msec/pass, best: 0.0324 etree append_from_document (T1,T4 ) 1.6374 1.8921 1.7790 msec/pass, best: 1.6374 ElementTree append_from_document (T1,T4 ) 0.1675 0.1767 0.1696 msec/pass, best: 0.1675 cElementTree append_from_document (T1,T4 ) 0.0590 0.0578 0.0561 msec/pass, best: 0.0561 etree append_from_document (T2,T1 ) 46.6267 44.5527 33.6675 msec/pass, best: 33.6675 ElementTree append_from_document (T2,T1 ) 1.6526 1.2971 1.9397 msec/pass, best: 1.2971 cElementTree append_from_document (T2,T1 ) 0.0589 0.0589 0.0579 msec/pass, best: 0.0579 etree append_from_document (T2,T3 ) 0.8258 0.8747 0.8474 msec/pass, best: 0.8258 ElementTree append_from_document (T2,T3 ) 0.0849 0.0896 0.0837 msec/pass, best: 0.0837 cElementTree append_from_document (T2,T3 ) 0.0333 0.0438 0.0326 msec/pass, best: 0.0326 etree append_from_document (T2,T4 ) 1.6711 1.8064 1.8190 msec/pass, best: 1.6711 ElementTree append_from_document (T2,T4 ) 2.0507 2.4498 2.7369 msec/pass, best: 2.0507 cElementTree append_from_document (T2,T4 ) 0.0447 0.0477 0.0477 msec/pass, best: 0.0447 etree append_from_document (T3,T1 ) 23.7482 23.7627 24.5559 msec/pass, best: 23.7482 ElementTree append_from_document (T3,T1 ) 3.3216 3.7194 3.4479 msec/pass, best: 3.3216 cElementTree append_from_document (T3,T1 ) 0.0639 0.1068 0.0728 msec/pass, best: 0.0639 etree append_from_document (T3,T2 ) 9.1862 9.7372 10.1377 msec/pass, best: 9.1862 ElementTree append_from_document (T3,T2 ) 6.2612 7.0192 6.8504 msec/pass, best: 6.2612 cElementTree append_from_document (T3,T2 ) 0.4900 0.5061 0.5034 msec/pass, best: 0.4900 etree append_from_document (T3,T4 ) 0.1238 0.1324 0.1245 msec/pass, best: 0.1238 ElementTree append_from_document (T3,T4 ) 0.1831 0.1520 0.1646 msec/pass, best: 0.1520 cElementTree append_from_document (T3,T4 ) 0.0322 0.0579 0.0326 msec/pass, best: 0.0322 etree append_from_document (T4,T1 ) 23.8607 24.0458 24.7333 msec/pass, best: 23.8607 ElementTree append_from_document (T4,T1 ) 0.1792 0.1784 0.1760 msec/pass, best: 0.1760 cElementTree append_from_document (T4,T1 ) 0.0677 0.0696 0.0694 msec/pass, best: 0.0677 etree append_from_document (T4,T2 ) 9.6456 10.0888 10.2759 msec/pass, best: 9.6456 ElementTree append_from_document (T4,T2 ) 4.9667 5.1865 5.2433 msec/pass, best: 4.9667 cElementTree append_from_document (T4,T2 ) 0.4923 0.5122 1.4466 msec/pass, best: 0.4923 etree append_from_document (T4,T3 ) 0.5469 0.5745 0.5633 msec/pass, best: 0.5469 ElementTree append_from_document (T4,T3 ) 0.0769 0.0754 0.0798 msec/pass, best: 0.0754 cElementTree append_from_document (T4,T3 ) 0.0248 0.0252 0.0253 msec/pass, best: 0.0248 etree clear (T1 ) 6.8760 6.9473 6.9588 msec/pass, best: 6.8760 ElementTree clear (T1 ) 17.0238 17.1312 17.4397 msec/pass, best: 17.0238 cElementTree clear (T1 ) 2.1736 2.1512 2.1850 msec/pass, best: 2.1512 etree clear (T2 ) 6.9850 7.1344 7.1863 msec/pass, best: 6.9850 ElementTree clear (T2 ) 28.4882 17.6843 17.8492 msec/pass, best: 17.6843 cElementTree clear (T2 ) 2.4351 2.4483 2.4590 msec/pass, best: 2.4351 etree clear (T3 ) 0.5403 0.5393 0.5486 msec/pass, best: 0.5393 ElementTree clear (T3 ) 4.3183 4.3355 4.3676 msec/pass, best: 4.3183 cElementTree clear (T3 ) 0.3575 0.3696 0.3694 msec/pass, best: 0.3575 etree clear (T4 ) 0.0243 0.0230 0.0225 msec/pass, best: 0.0225 ElementTree clear (T4 ) 0.0497 0.0468 0.0491 msec/pass, best: 0.0468 cElementTree clear (T4 ) 0.0065 0.0069 0.0066 msec/pass, best: 0.0065 etree create_subelements (T1 ) 0.4447 0.4291 0.4390 msec/pass, best: 0.4291 ElementTree create_subelements (T1 ) 0.4023 0.4029 0.4081 msec/pass, best: 0.4023 cElementTree create_subelements (T1 ) 0.0650 0.0624 0.0642 msec/pass, best: 0.0624 etree create_subelements (T2 ) 6.4463 6.5367 6.4625 msec/pass, best: 6.4463 ElementTree create_subelements (T2 ) 6.4680 6.4867 6.5039 msec/pass, best: 6.4680 cElementTree create_subelements (T2 ) 0.5437 0.5510 0.5644 msec/pass, best: 0.5437 etree create_subelements (T3 ) 0.0982 0.0962 0.0966 msec/pass, best: 0.0962 ElementTree create_subelements (T3 ) 0.1408 0.1207 0.1170 msec/pass, best: 0.1170 cElementTree create_subelements (T3 ) 0.0279 0.0268 0.0265 msec/pass, best: 0.0265 etree create_subelements (T4 ) 0.3160 0.3148 0.3172 msec/pass, best: 0.3148 ElementTree create_subelements (T4 ) 0.3121 0.2866 0.2813 msec/pass, best: 0.2813 cElementTree create_subelements (T4 ) 0.0210 0.0210 0.0212 msec/pass, best: 0.0210 etree deepcopy (T1 ) 17.9121 19.2798 18.9026 msec/pass, best: 17.9121 ElementTree deepcopy (T1 ) 805.1189 805.3489 872.4165 msec/pass, best: 805.1189 cElementTree deepcopy (T1 ) 505.1526 479.0999 474.0529 msec/pass, best: 474.0529 etree deepcopy (T2 ) 30.3835 31.2109 31.1521 msec/pass, best: 30.3835 ElementTree deepcopy (T2 ) 1027.0189 1034.5323 1110.9516 msec/pass, best: 1027.0189 cElementTree deepcopy (T2 ) 534.4865 499.5629 523.3296 msec/pass, best: 499.5629 etree deepcopy (T3 ) 3.2103 3.1447 3.1190 msec/pass, best: 3.1190 ElementTree deepcopy (T3 ) 188.7364 189.6626 192.9635 msec/pass, best: 188.7364 cElementTree deepcopy (T3 ) 124.1634 121.2389 121.2373 msec/pass, best: 121.2373 etree deepcopy (T4 ) 0.6969 0.6899 0.7117 msec/pass, best: 0.6899 ElementTree deepcopy (T4 ) 4.4663 4.3800 4.4279 msec/pass, best: 4.3800 cElementTree deepcopy (T4 ) 3.1183 3.1667 3.0353 msec/pass, best: 3.0353 etree getchildren (T1 ) 21.8075 22.5544 22.9289 msec/pass, best: 21.8075 ElementTree getchildren (T1 ) 0.1317 0.1362 0.1310 msec/pass, best: 0.1310 cElementTree getchildren (T1 ) 0.9512 0.9820 0.9762 msec/pass, best: 0.9512 etree getchildren (T2 ) 20.1090 17.5955 17.8679 msec/pass, best: 17.5955 ElementTree getchildren (T2 ) 1.4564 1.4540 1.4398 msec/pass, best: 1.4398 cElementTree getchildren (T2 ) 1.3277 1.3638 1.3366 msec/pass, best: 1.3277 etree getchildren (T3 ) 0.0357 0.0358 0.0353 msec/pass, best: 0.0353 ElementTree getchildren (T3 ) 0.0633 0.0640 0.0679 msec/pass, best: 0.0633 cElementTree getchildren (T3 ) 0.0240 0.0254 0.0255 msec/pass, best: 0.0240 etree getchildren (T4 ) 0.1035 0.0940 0.0942 msec/pass, best: 0.0940 ElementTree getchildren (T4 ) 0.0671 0.0709 0.0690 msec/pass, best: 0.0671 cElementTree getchildren (T4 ) 0.0276 0.0272 0.0280 msec/pass, best: 0.0272 etree insert_from_document (T1,T2 ) 24.8505 24.1553 25.6988 msec/pass, best: 24.1553 ElementTree insert_from_document (T1,T2 ) 8.7017 9.8994 9.8052 msec/pass, best: 8.7017 cElementTree insert_from_document (T1,T2 ) 1.2120 1.2260 1.2225 msec/pass, best: 1.2120 etree insert_from_document (T1,T3 ) 1.0492 1.1782 1.3616 msec/pass, best: 1.0492 ElementTree insert_from_document (T1,T3 ) 0.0971 0.1012 0.1044 msec/pass, best: 0.0971 cElementTree insert_from_document (T1,T3 ) 0.0451 0.0489 0.0429 msec/pass, best: 0.0429 etree insert_from_document (T1,T4 ) 1.7054 1.8474 1.8518 msec/pass, best: 1.7054 ElementTree insert_from_document (T1,T4 ) 0.2471 0.2510 0.2550 msec/pass, best: 0.2471 cElementTree insert_from_document (T1,T4 ) 0.0815 0.0940 0.0868 msec/pass, best: 0.0815 etree insert_from_document (T2,T1 ) 32.2051 34.0006 34.0085 msec/pass, best: 32.2051 ElementTree insert_from_document (T2,T1 ) 2.2987 3.2959 2.1304 msec/pass, best: 2.1304 cElementTree insert_from_document (T2,T1 ) 0.1201 0.1238 0.1221 msec/pass, best: 0.1201 etree insert_from_document (T2,T3 ) 1.3658 1.2899 1.4140 msec/pass, best: 1.2899 ElementTree insert_from_document (T2,T3 ) 0.1069 0.1094 0.1085 msec/pass, best: 0.1069 cElementTree insert_from_document (T2,T3 ) 0.0470 0.0556 0.0524 msec/pass, best: 0.0470 etree insert_from_document (T2,T4 ) 2.6039 2.7250 2.7508 msec/pass, best: 2.6039 ElementTree insert_from_document (T2,T4 ) 2.2039 2.5299 2.5815 msec/pass, best: 2.2039 cElementTree insert_from_document (T2,T4 ) 0.1059 0.1134 0.1127 msec/pass, best: 0.1059 etree insert_from_document (T3,T1 ) 19.8009 25.0803 23.4647 msec/pass, best: 19.8009 ElementTree insert_from_document (T3,T1 ) 3.0262 4.1957 3.7703 msec/pass, best: 3.0262 cElementTree insert_from_document (T3,T1 ) 0.1040 0.1033 0.0954 msec/pass, best: 0.0954 etree insert_from_document (T3,T2 ) 15.1248 15.4261 16.6505 msec/pass, best: 15.1248 ElementTree insert_from_document (T3,T2 ) 7.8220 8.0915 8.5682 msec/pass, best: 7.8220 cElementTree insert_from_document (T3,T2 ) 1.1612 1.1757 1.2460 msec/pass, best: 1.1612 etree insert_from_document (T3,T4 ) 0.1667 0.4397 0.1828 msec/pass, best: 0.1667 ElementTree insert_from_document (T3,T4 ) 0.2935 0.2366 0.2363 msec/pass, best: 0.2363 cElementTree insert_from_document (T3,T4 ) 0.0566 0.0568 0.0734 msec/pass, best: 0.0566 etree insert_from_document (T4,T1 ) 23.9678 24.8289 24.7205 msec/pass, best: 23.9678 ElementTree insert_from_document (T4,T1 ) 0.2536 0.2725 0.2724 msec/pass, best: 0.2536 cElementTree insert_from_document (T4,T1 ) 0.0970 0.0986 0.0969 msec/pass, best: 0.0969 etree insert_from_document (T4,T2 ) 17.4943 17.6244 18.9083 msec/pass, best: 17.4943 ElementTree insert_from_document (T4,T2 ) 6.4159 6.5594 6.6816 msec/pass, best: 6.4159 cElementTree insert_from_document (T4,T2 ) 1.1943 1.1961 1.2079 msec/pass, best: 1.1943 etree insert_from_document (T4,T3 ) 0.5701 0.5741 0.5685 msec/pass, best: 0.5685 ElementTree insert_from_document (T4,T3 ) 0.1002 0.0967 0.0964 msec/pass, best: 0.0964 cElementTree insert_from_document (T4,T3 ) 0.0336 0.0379 0.0327 msec/pass, best: 0.0327 etree remove_children (T1 ) 6.9888 7.1405 7.1501 msec/pass, best: 6.9888 ElementTree remove_children (T1 ) 10.2381 10.3145 10.3398 msec/pass, best: 10.2381 cElementTree remove_children (T1 ) 1.1740 1.2146 1.2070 msec/pass, best: 1.1740 etree remove_children (T2 ) 8.0086 8.0501 8.0835 msec/pass, best: 8.0086 ElementTree remove_children (T2 ) 171.0074 165.6075 174.0978 msec/pass, best: 165.6075 cElementTree remove_children (T2 ) 4.5207 4.6791 4.7362 msec/pass, best: 4.5207 etree remove_children (T3 ) 0.5668 0.5530 0.5762 msec/pass, best: 0.5530 ElementTree remove_children (T3 ) 3.1857 3.1828 3.2323 msec/pass, best: 3.1828 cElementTree remove_children (T3 ) 0.2588 0.2608 0.2584 msec/pass, best: 0.2584 etree remove_children (T4 ) 0.0684 0.0753 0.0700 msec/pass, best: 0.0684 ElementTree remove_children (T4 ) 0.4579 0.4570 0.4711 msec/pass, best: 0.4570 cElementTree remove_children (T4 ) 0.0209 0.0210 0.0207 msec/pass, best: 0.0207 etree remove_children_reversed (T1 ) 11.9009 11.9145 11.8435 msec/pass, best: 11.8435 ElementTree remove_children_reversed (T1 ) 19.3379 18.8514 20.4126 msec/pass, best: 18.8514 cElementTree remove_children_reversed (T1 ) 2.2264 2.3288 2.2562 msec/pass, best: 2.2264 etree remove_children_reversed (T2 ) 12.9722 14.8116 12.7335 msec/pass, best: 12.7335 ElementTree remove_children_reversed (T2 ) 673.7946 684.9349 669.5603 msec/pass, best: 669.5603 cElementTree remove_children_reversed (T2 ) 13.0890 12.8449 12.9061 msec/pass, best: 12.8449 etree remove_children_reversed (T3 ) 0.5227 0.5268 0.5229 msec/pass, best: 0.5227 ElementTree remove_children_reversed (T3 ) 4.4230 5.2407 4.4353 msec/pass, best: 4.4230 cElementTree remove_children_reversed (T3 ) 0.4222 0.4244 0.4180 msec/pass, best: 0.4180 etree remove_children_reversed (T4 ) 0.0738 0.0731 0.0736 msec/pass, best: 0.0731 ElementTree remove_children_reversed (T4 ) 1.6637 1.6356 1.6363 msec/pass, best: 1.6356 cElementTree remove_children_reversed (T4 ) 0.0510 0.0484 0.0499 msec/pass, best: 0.0484 etree reorder (T1 ) 14.5806 16.2235 15.8373 msec/pass, best: 14.5806 ElementTree reorder (T1 ) 0.8176 2.1794 2.6297 msec/pass, best: 0.8176 cElementTree reorder (T1 ) 0.0634 0.0676 0.0605 msec/pass, best: 0.0605 etree reorder (T2 ) 18.1266 18.8180 18.8672 msec/pass, best: 18.1266 ElementTree reorder (T2 ) 5.5504 5.5097 5.7170 msec/pass, best: 5.5097 cElementTree reorder (T2 ) 1.1481 1.1568 1.1591 msec/pass, best: 1.1481 etree reorder (T3 ) 0.0130 0.0156 0.0155 msec/pass, best: 0.0130 ElementTree reorder (T3 ) 0.0437 0.0459 0.0446 msec/pass, best: 0.0437 cElementTree reorder (T3 ) 0.0244 0.0228 0.0230 msec/pass, best: 0.0228 etree reorder (T4 ) 0.0731 0.0780 0.0734 msec/pass, best: 0.0731 ElementTree reorder (T4 ) 0.1282 0.1242 0.1238 msec/pass, best: 0.1238 cElementTree reorder (T4 ) 0.0217 0.0218 0.0220 msec/pass, best: 0.0217 etree reorder_slice (T1 ) 14.5325 17.0733 15.9148 msec/pass, best: 14.5325 ElementTree reorder_slice (T1 ) 0.5114 1.8668 2.6623 msec/pass, best: 0.5114 cElementTree reorder_slice (T1 ) 0.0629 0.0636 0.0633 msec/pass, best: 0.0629 etree reorder_slice (T2 ) 19.0733 22.6151 20.3017 msec/pass, best: 19.0733 ElementTree reorder_slice (T2 ) 5.5578 5.8006 5.7953 msec/pass, best: 5.5578 cElementTree reorder_slice (T2 ) 1.1598 1.1522 1.1647 msec/pass, best: 1.1522 etree reorder_slice (T3 ) 0.0145 0.0135 0.0133 msec/pass, best: 0.0133 ElementTree reorder_slice (T3 ) 0.0421 0.0435 0.0434 msec/pass, best: 0.0421 cElementTree reorder_slice (T3 ) 0.0225 0.0221 0.0229 msec/pass, best: 0.0221 etree reorder_slice (T4 ) 0.0796 0.0779 0.0786 msec/pass, best: 0.0779 ElementTree reorder_slice (T4 ) 0.1390 0.1334 0.1426 msec/pass, best: 0.1334 cElementTree reorder_slice (T4 ) 0.0259 0.0219 0.0217 msec/pass, best: 0.0217 etree replace_children (T1 ) 7.4184 7.9814 8.2130 msec/pass, best: 7.4184 ElementTree replace_children (T1 ) 19.4359 19.5994 19.5809 msec/pass, best: 19.4359 cElementTree replace_children (T1 ) 2.0112 2.0221 2.0522 msec/pass, best: 2.0112 etree replace_children (T2 ) 17.0823 19.4184 20.3127 msec/pass, best: 17.0823 ElementTree replace_children (T2 ) 26.4215 26.6201 26.9273 msec/pass, best: 26.4215 cElementTree replace_children (T2 ) 2.9001 2.8836 2.8504 msec/pass, best: 2.8504 etree replace_children (T3 ) 0.6777 0.6651 0.6562 msec/pass, best: 0.6562 ElementTree replace_children (T3 ) 5.2446 4.5064 4.6229 msec/pass, best: 4.5064 cElementTree replace_children (T3 ) 0.4712 0.4945 0.4554 msec/pass, best: 0.4554 etree replace_children (T4 ) 0.4540 0.5324 0.4619 msec/pass, best: 0.4540 ElementTree replace_children (T4 ) 0.3328 0.3317 0.3333 msec/pass, best: 0.3317 cElementTree replace_children (T4 ) 0.0388 0.0385 0.0382 msec/pass, best: 0.0382 etree rotate_children (T1 ) 27.4596 27.0203 27.8874 msec/pass, best: 27.0203 ElementTree rotate_children (T1 ) 1.2427 1.2087 1.1850 msec/pass, best: 1.1850 cElementTree rotate_children (T1 ) 0.1627 0.1654 0.1633 msec/pass, best: 0.1627 etree rotate_children (T2 ) 7.2919 7.9870 7.1657 msec/pass, best: 7.1657 ElementTree rotate_children (T2 ) 2.2686 2.5049 1.8473 msec/pass, best: 1.8473 cElementTree rotate_children (T2 ) 0.4213 0.4067 0.4378 msec/pass, best: 0.4067 etree rotate_children (T3 ) 3.9796 3.9531 4.7094 msec/pass, best: 3.9531 ElementTree rotate_children (T3 ) 0.6443 0.5682 0.5796 msec/pass, best: 0.5682 cElementTree rotate_children (T3 ) 0.1355 0.1322 0.1368 msec/pass, best: 0.1322 etree rotate_children (T4 ) 0.3995 0.3204 0.3173 msec/pass, best: 0.3173 ElementTree rotate_children (T4 ) 0.5233 0.5198 0.5278 msec/pass, best: 0.5198 cElementTree rotate_children (T4 ) 0.1210 0.1199 0.1275 msec/pass, best: 0.1199 etree set_attributes (T1 ) 6.0470 6.6676 6.6880 msec/pass, best: 6.0470 ElementTree set_attributes (T1 ) 0.1447 0.1446 0.1479 msec/pass, best: 0.1446 cElementTree set_attributes (T1 ) 0.0691 0.0665 0.0666 msec/pass, best: 0.0665 etree set_attributes (T2 ) 8.8447 9.5652 9.4956 msec/pass, best: 8.8447 ElementTree set_attributes (T2 ) 1.6476 1.6582 1.6447 msec/pass, best: 1.6447 cElementTree set_attributes (T2 ) 0.7479 0.7352 0.7474 msec/pass, best: 0.7352 etree set_attributes (T3 ) 0.0568 0.0928 0.0583 msec/pass, best: 0.0568 ElementTree set_attributes (T3 ) 0.0625 0.0613 0.0658 msec/pass, best: 0.0613 cElementTree set_attributes (T3 ) 0.0272 0.0240 0.0255 msec/pass, best: 0.0240 etree set_attributes (T4 ) 0.1534 0.1562 0.1530 msec/pass, best: 0.1530 ElementTree set_attributes (T4 ) 0.0779 0.0759 0.0762 msec/pass, best: 0.0759 cElementTree set_attributes (T4 ) 0.0268 0.0282 0.0261 msec/pass, best: 0.0261 etree setget_attributes (T1 ) 5.4190 6.6268 6.6813 msec/pass, best: 5.4190 ElementTree setget_attributes (T1 ) 0.2305 0.2438 0.2276 msec/pass, best: 0.2276 cElementTree setget_attributes (T1 ) 0.1521 0.0983 0.0920 msec/pass, best: 0.0920 etree setget_attributes (T2 ) 10.5145 11.2669 11.2211 msec/pass, best: 10.5145 ElementTree setget_attributes (T2 ) 3.2395 3.1300 3.1196 msec/pass, best: 3.1196 cElementTree setget_attributes (T2 ) 1.1320 1.1729 1.1571 msec/pass, best: 1.1320 etree setget_attributes (T3 ) 0.0784 0.0807 0.0913 msec/pass, best: 0.0784 ElementTree setget_attributes (T3 ) 0.0977 0.0835 0.0889 msec/pass, best: 0.0835 cElementTree setget_attributes (T3 ) 0.0295 0.0285 0.0333 msec/pass, best: 0.0285 etree setget_attributes (T4 ) 0.2418 0.2438 0.2392 msec/pass, best: 0.2392 ElementTree setget_attributes (T4 ) 0.1654 0.1779 0.1539 msec/pass, best: 0.1539 cElementTree setget_attributes (T4 ) 0.0476 0.0458 0.0506 msec/pass, best: 0.0458

2 2

[lxml-dev] Call for Contribution: lxml 0.9
by Stefan Behnel 09 Mar '06

09 Mar '06

Hello, everyone! Martijn and I realised that there has not been a release of lxml for quite a while - despite a lot of important changes to the code base. We therefore agreed to prepare the release of lxml 0.9 for *Monday, March 20*. The remaining time will be used to get the trunk ready for a release. It will include the current feature set of the trunk plus the support for extension functions that is currently implemented in the scoder2 branch. The latter still needs a bit of discussion and merging. Any comments on this are appreciated. For you, this announcement means the following. - If you want any features included, make sure you send requests and patches to the list within the next days or try to fix a date when a patch will be available. Remember that we may have to discuss patches before applying them. - If you feel that you especially like or dislike anything about the new features and the way they are made available through APIs, feel free to tell us over the list. - If you feel that anything from past discussions has been left out that should be worth remembering for a new release, speak up now. - We know that there are still a few places where documentation would help users to understand new features. A good way to contribute would therefore be to look through the documentation in the "doc" directory and see if it is understandable or if there is anything you might want to add. - In the same line, usage examples in the form of doctests would be nice to have for more API functions. They also belong into the files in the "doc" directory, which are (or must be) called from the related test cases in src/lxml/tests. Happy coding, Stefan

5 14

[lxml-dev] Benchmark script
by Stefan Behnel 08 Mar '06

08 Mar '06

Hi, as I was at it anyway, I wrote a benchmark script that allows benchmarking simple test cases against different XML trees. Here are the benchmarks I have for now, taken from the new file bench.py in the trunk: def bench_append_from_document(self, tree1, root1, tree2, root2): "1,2" # needs trees 1 and 2 for el in root2: root1.append(root2[0]) def bench_rotate_children(self, tree, root): "1 2" # runs on tree 1 or 2 independently for i in range(100): root[-1] = root[0] def bench_reorder(self, tree, root): "1 2" for i in range(len(root)/2): root[-i] = root[0] The doc strings define the trees (or set of trees) that the benchmark is run against. Calling obviously uses reflection, so you can simply add new methods like the above to have them benchmarked. As an example, here are the results before and after my last patch: ## current trunk: # python bench.py append_from_document (T1,T2 ) 0.0586 0.0584 0.0583 msec/pass, avg: 0.0584 reorder (T1 ) 0.3182 0.3200 0.3183 msec/pass, avg: 0.3188 reorder (T2 ) 0.0951 0.0948 0.0952 msec/pass, avg: 0.0950 rotate_children (T1 ) 14.6780 14.4417 14.3684 msec/pass, avg: 14.4960 rotate_children (T2 ) 0.8209 0.8150 0.8139 msec/pass, avg: 0.8166 ## after applying the patch: # python bench.py append_from_document (T1,T2 ) 0.0582 0.0591 0.0583 msec/pass, avg: 0.0585 reorder (T1 ) 0.2437 0.2629 0.2432 msec/pass, avg: 0.2499 reorder (T2 ) 0.0926 0.0922 0.0923 msec/pass, avg: 0.0924 rotate_children (T1 ) 7.6664 7.6860 7.7656 msec/pass, avg: 7.7060 rotate_children (T2 ) 0.7434 0.7454 0.7599 msec/pass, avg: 0.7496 This is pretty significant in some cases, so I will apply the patch to the trunk. It also tells me that I will have to look at the rotate_children case as it is highly dependent on the type of tree (which it shouldn't be too much, I'd say). So, as usual, feel free to test it and find new benchmark cases for it. Stefan

3 5

[lxml-dev] Restructuring of namespace setting in Element() and SubElement()
by Stefan Behnel 08 Mar '06

08 Mar '06

Hi, just a quick status update. I committed a patch to the trunk (24115) that basically refactors the above functions and the _addNamespaces function. The two are the main API functions (and the oldest grown ones around, I guess), so as I was running a few benchmarks anyway, I wanted to check if I could come up with some tweaks to make them faster. I took a closer look at them and found a couple of potential bugs and inefficient indirections. The result is that (I think) they should be more straight forward to read now and about 10-30% faster according to my benchmark. One of the problems was that name and namespace of the new element were set twice. That was actually good, as they were set incorrectly the first time, so that bug was masked. However, that was neither very readable nor efficient. The new implementation moves the namespace handling and prefix setting into the _addNamespaces function (which I renamed to _setNamespaces). It is therefore combined with the namespace setup from the nsmap dictionary, so that the prefixes and the C namespace structure only have to be generated once. The test cases pass just as before. I also added a few new ones whenever I found sequences of code that made me wonder. So I can somewhat hope I didn't replace the old bugs by new ones... Have fun, Stefan

1 0

[lxml-dev] OT: document('') in xslt
by Brad Clements 06 Mar '06

06 Mar '06

For those who use document('') in stylesheets, especially those xsl files sent to browsers. Using document('') in an xslt file causes Firefox and Mozilla to lock-up. This appears to finally be fixed in today's build: https://bugzilla.mozilla.org/show_bug.cgi?id=205778 What |Removed |Added -------------------------------------------------------------------------- -- Keywords|fixed1.8.0.2 |verified1.8.0.2 ------- Comment #51 from walkerrunner(a)yahoo.com 2006-03-06 13:33 PST ------- Verified on Firefox 1.5.0.2 buildsfrom 20060306 on Windows, Mac and Linux -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM or SKYPE: BKClements

2 1

lxml - The Python XML Toolkit March 2006