[lxml-dev] New benchmark results

Here, we go, this is *much* better.
I looked over one of the bottlenecks in the API, the _elementFactory function. I found that the main problem was my implementation of the namespace element lookup, which translated to extremely inefficient C code. I rewrote that with explicit Python API calls and that gives us between 10% and 400% speed up in the benchmarks, depending on the API intensiveness of the tests. Note that lxml is faster than ElementTree in the benchmark tree setup now.
Now, the main area where lxml is substantially slower than ElementTree is when moving elements between documents by hand. This is rare and can be avoided pretty often, I guess. Note that moving an entire XML tree to a new document in the obvious one-step operation is *not* so expensive in lxml. What matters is the overhead of creating a Python representation of elements, which is purely API related. Everything that runs internally runs mainly at the speed of libxml2. A special beauty here is the deepcopy benchmark.
There still is a certain overhead in the API-to-libxml2 mapping, so programs that extensively create elements one after the other are better off with ElementTree and cElementTree. This advantage already diminishes when SubElement is used, for which lxml is now faster than ElementTree. In all other cases, I wouldn't see a major advantage for the two. Especially none that should keep people from using lxml. Depending on the tree size/structure and the operation, lxml and cElementTree can both take the lead.
Since the APIs are largely compatible, I see the main decision maker as follows:
* If your problem is small and not performance critical, use whatever you like * If you create tons of independent elements, use cElementTree or consider changing your algorithms :) * If your program is mainly API bound, use cElementTree * If your program is dominated by large XML structures, try both lxml and cElementTree * If you often need to copy large trees, consider lxml * If you need features like XPath, XSLT, RNG, custom APIs, ... - use lxml * If you need those features and more speed, send a patch to this list
Have fun, Stefan
Preparing test suites and trees ... Running benchmark on etree, ElementTree, cElementTree
Setup times for trees in seconds: etree : 0.1827, 0.1651, 0.0371, 0.0008 ElementTree : 0.2251, 0.2564, 0.0574, 0.0010 cElementTree : 0.0378, 0.0176, 0.0088, 0.0001
etree append_elements (T1 ) 2.1396 2.2991 2.2673 msec/pass, best: 2.1396 ElementTree append_elements (T1 ) 0.3253 0.3294 0.3451 msec/pass, best: 0.3253 cElementTree append_elements (T1 ) 0.1065 0.0814 0.0795 msec/pass, best: 0.0795
etree append_elements (T2 ) 9.5490 9.8586 9.8201 msec/pass, best: 9.5490 ElementTree append_elements (T2 ) 4.9283 4.9106 4.9252 msec/pass, best: 4.9106 cElementTree append_elements (T2 ) 0.7983 0.8127 0.8101 msec/pass, best: 0.7983
etree append_elements (T3 ) 0.1166 0.1110 0.1099 msec/pass, best: 0.1099 ElementTree append_elements (T3 ) 0.1056 0.1081 0.1097 msec/pass, best: 0.1056 cElementTree append_elements (T3 ) 0.0308 0.0326 0.0314 msec/pass, best: 0.0308
etree append_elements (T4 ) 0.6297 0.4669 0.4662 msec/pass, best: 0.4662 ElementTree append_elements (T4 ) 0.2243 0.2092 0.2179 msec/pass, best: 0.2092 cElementTree append_elements (T4 ) 0.0375 0.0376 0.0355 msec/pass, best: 0.0355
etree append_from_document (T1,T2 ) 17.5267 12.2076 12.4225 msec/pass, best: 12.2076 ElementTree append_from_document (T1,T2 ) 7.2593 8.3792 8.1089 msec/pass, best: 7.2593 cElementTree append_from_document (T1,T2 ) 0.4959 0.5207 0.4878 msec/pass, best: 0.4878
etree append_from_document (T1,T3 ) 0.8263 0.8588 0.8604 msec/pass, best: 0.8263 ElementTree append_from_document (T1,T3 ) 0.0907 0.0806 0.0843 msec/pass, best: 0.0806 cElementTree append_from_document (T1,T3 ) 0.0324 0.0324 0.0348 msec/pass, best: 0.0324
etree append_from_document (T1,T4 ) 1.6374 1.8921 1.7790 msec/pass, best: 1.6374 ElementTree append_from_document (T1,T4 ) 0.1675 0.1767 0.1696 msec/pass, best: 0.1675 cElementTree append_from_document (T1,T4 ) 0.0590 0.0578 0.0561 msec/pass, best: 0.0561
etree append_from_document (T2,T1 ) 46.6267 44.5527 33.6675 msec/pass, best: 33.6675 ElementTree append_from_document (T2,T1 ) 1.6526 1.2971 1.9397 msec/pass, best: 1.2971 cElementTree append_from_document (T2,T1 ) 0.0589 0.0589 0.0579 msec/pass, best: 0.0579
etree append_from_document (T2,T3 ) 0.8258 0.8747 0.8474 msec/pass, best: 0.8258 ElementTree append_from_document (T2,T3 ) 0.0849 0.0896 0.0837 msec/pass, best: 0.0837 cElementTree append_from_document (T2,T3 ) 0.0333 0.0438 0.0326 msec/pass, best: 0.0326
etree append_from_document (T2,T4 ) 1.6711 1.8064 1.8190 msec/pass, best: 1.6711 ElementTree append_from_document (T2,T4 ) 2.0507 2.4498 2.7369 msec/pass, best: 2.0507 cElementTree append_from_document (T2,T4 ) 0.0447 0.0477 0.0477 msec/pass, best: 0.0447
etree append_from_document (T3,T1 ) 23.7482 23.7627 24.5559 msec/pass, best: 23.7482 ElementTree append_from_document (T3,T1 ) 3.3216 3.7194 3.4479 msec/pass, best: 3.3216 cElementTree append_from_document (T3,T1 ) 0.0639 0.1068 0.0728 msec/pass, best: 0.0639
etree append_from_document (T3,T2 ) 9.1862 9.7372 10.1377 msec/pass, best: 9.1862 ElementTree append_from_document (T3,T2 ) 6.2612 7.0192 6.8504 msec/pass, best: 6.2612 cElementTree append_from_document (T3,T2 ) 0.4900 0.5061 0.5034 msec/pass, best: 0.4900
etree append_from_document (T3,T4 ) 0.1238 0.1324 0.1245 msec/pass, best: 0.1238 ElementTree append_from_document (T3,T4 ) 0.1831 0.1520 0.1646 msec/pass, best: 0.1520 cElementTree append_from_document (T3,T4 ) 0.0322 0.0579 0.0326 msec/pass, best: 0.0322
etree append_from_document (T4,T1 ) 23.8607 24.0458 24.7333 msec/pass, best: 23.8607 ElementTree append_from_document (T4,T1 ) 0.1792 0.1784 0.1760 msec/pass, best: 0.1760 cElementTree append_from_document (T4,T1 ) 0.0677 0.0696 0.0694 msec/pass, best: 0.0677
etree append_from_document (T4,T2 ) 9.6456 10.0888 10.2759 msec/pass, best: 9.6456 ElementTree append_from_document (T4,T2 ) 4.9667 5.1865 5.2433 msec/pass, best: 4.9667 cElementTree append_from_document (T4,T2 ) 0.4923 0.5122 1.4466 msec/pass, best: 0.4923
etree append_from_document (T4,T3 ) 0.5469 0.5745 0.5633 msec/pass, best: 0.5469 ElementTree append_from_document (T4,T3 ) 0.0769 0.0754 0.0798 msec/pass, best: 0.0754 cElementTree append_from_document (T4,T3 ) 0.0248 0.0252 0.0253 msec/pass, best: 0.0248
etree clear (T1 ) 6.8760 6.9473 6.9588 msec/pass, best: 6.8760 ElementTree clear (T1 ) 17.0238 17.1312 17.4397 msec/pass, best: 17.0238 cElementTree clear (T1 ) 2.1736 2.1512 2.1850 msec/pass, best: 2.1512
etree clear (T2 ) 6.9850 7.1344 7.1863 msec/pass, best: 6.9850 ElementTree clear (T2 ) 28.4882 17.6843 17.8492 msec/pass, best: 17.6843 cElementTree clear (T2 ) 2.4351 2.4483 2.4590 msec/pass, best: 2.4351
etree clear (T3 ) 0.5403 0.5393 0.5486 msec/pass, best: 0.5393 ElementTree clear (T3 ) 4.3183 4.3355 4.3676 msec/pass, best: 4.3183 cElementTree clear (T3 ) 0.3575 0.3696 0.3694 msec/pass, best: 0.3575
etree clear (T4 ) 0.0243 0.0230 0.0225 msec/pass, best: 0.0225 ElementTree clear (T4 ) 0.0497 0.0468 0.0491 msec/pass, best: 0.0468 cElementTree clear (T4 ) 0.0065 0.0069 0.0066 msec/pass, best: 0.0065
etree create_subelements (T1 ) 0.4447 0.4291 0.4390 msec/pass, best: 0.4291 ElementTree create_subelements (T1 ) 0.4023 0.4029 0.4081 msec/pass, best: 0.4023 cElementTree create_subelements (T1 ) 0.0650 0.0624 0.0642 msec/pass, best: 0.0624
etree create_subelements (T2 ) 6.4463 6.5367 6.4625 msec/pass, best: 6.4463 ElementTree create_subelements (T2 ) 6.4680 6.4867 6.5039 msec/pass, best: 6.4680 cElementTree create_subelements (T2 ) 0.5437 0.5510 0.5644 msec/pass, best: 0.5437
etree create_subelements (T3 ) 0.0982 0.0962 0.0966 msec/pass, best: 0.0962 ElementTree create_subelements (T3 ) 0.1408 0.1207 0.1170 msec/pass, best: 0.1170 cElementTree create_subelements (T3 ) 0.0279 0.0268 0.0265 msec/pass, best: 0.0265
etree create_subelements (T4 ) 0.3160 0.3148 0.3172 msec/pass, best: 0.3148 ElementTree create_subelements (T4 ) 0.3121 0.2866 0.2813 msec/pass, best: 0.2813 cElementTree create_subelements (T4 ) 0.0210 0.0210 0.0212 msec/pass, best: 0.0210
etree deepcopy (T1 ) 17.9121 19.2798 18.9026 msec/pass, best: 17.9121 ElementTree deepcopy (T1 ) 805.1189 805.3489 872.4165 msec/pass, best: 805.1189 cElementTree deepcopy (T1 ) 505.1526 479.0999 474.0529 msec/pass, best: 474.0529
etree deepcopy (T2 ) 30.3835 31.2109 31.1521 msec/pass, best: 30.3835 ElementTree deepcopy (T2 ) 1027.0189 1034.5323 1110.9516 msec/pass, best: 1027.0189 cElementTree deepcopy (T2 ) 534.4865 499.5629 523.3296 msec/pass, best: 499.5629
etree deepcopy (T3 ) 3.2103 3.1447 3.1190 msec/pass, best: 3.1190 ElementTree deepcopy (T3 ) 188.7364 189.6626 192.9635 msec/pass, best: 188.7364 cElementTree deepcopy (T3 ) 124.1634 121.2389 121.2373 msec/pass, best: 121.2373
etree deepcopy (T4 ) 0.6969 0.6899 0.7117 msec/pass, best: 0.6899 ElementTree deepcopy (T4 ) 4.4663 4.3800 4.4279 msec/pass, best: 4.3800 cElementTree deepcopy (T4 ) 3.1183 3.1667 3.0353 msec/pass, best: 3.0353
etree getchildren (T1 ) 21.8075 22.5544 22.9289 msec/pass, best: 21.8075 ElementTree getchildren (T1 ) 0.1317 0.1362 0.1310 msec/pass, best: 0.1310 cElementTree getchildren (T1 ) 0.9512 0.9820 0.9762 msec/pass, best: 0.9512
etree getchildren (T2 ) 20.1090 17.5955 17.8679 msec/pass, best: 17.5955 ElementTree getchildren (T2 ) 1.4564 1.4540 1.4398 msec/pass, best: 1.4398 cElementTree getchildren (T2 ) 1.3277 1.3638 1.3366 msec/pass, best: 1.3277
etree getchildren (T3 ) 0.0357 0.0358 0.0353 msec/pass, best: 0.0353 ElementTree getchildren (T3 ) 0.0633 0.0640 0.0679 msec/pass, best: 0.0633 cElementTree getchildren (T3 ) 0.0240 0.0254 0.0255 msec/pass, best: 0.0240
etree getchildren (T4 ) 0.1035 0.0940 0.0942 msec/pass, best: 0.0940 ElementTree getchildren (T4 ) 0.0671 0.0709 0.0690 msec/pass, best: 0.0671 cElementTree getchildren (T4 ) 0.0276 0.0272 0.0280 msec/pass, best: 0.0272
etree insert_from_document (T1,T2 ) 24.8505 24.1553 25.6988 msec/pass, best: 24.1553 ElementTree insert_from_document (T1,T2 ) 8.7017 9.8994 9.8052 msec/pass, best: 8.7017 cElementTree insert_from_document (T1,T2 ) 1.2120 1.2260 1.2225 msec/pass, best: 1.2120
etree insert_from_document (T1,T3 ) 1.0492 1.1782 1.3616 msec/pass, best: 1.0492 ElementTree insert_from_document (T1,T3 ) 0.0971 0.1012 0.1044 msec/pass, best: 0.0971 cElementTree insert_from_document (T1,T3 ) 0.0451 0.0489 0.0429 msec/pass, best: 0.0429
etree insert_from_document (T1,T4 ) 1.7054 1.8474 1.8518 msec/pass, best: 1.7054 ElementTree insert_from_document (T1,T4 ) 0.2471 0.2510 0.2550 msec/pass, best: 0.2471 cElementTree insert_from_document (T1,T4 ) 0.0815 0.0940 0.0868 msec/pass, best: 0.0815
etree insert_from_document (T2,T1 ) 32.2051 34.0006 34.0085 msec/pass, best: 32.2051 ElementTree insert_from_document (T2,T1 ) 2.2987 3.2959 2.1304 msec/pass, best: 2.1304 cElementTree insert_from_document (T2,T1 ) 0.1201 0.1238 0.1221 msec/pass, best: 0.1201
etree insert_from_document (T2,T3 ) 1.3658 1.2899 1.4140 msec/pass, best: 1.2899 ElementTree insert_from_document (T2,T3 ) 0.1069 0.1094 0.1085 msec/pass, best: 0.1069 cElementTree insert_from_document (T2,T3 ) 0.0470 0.0556 0.0524 msec/pass, best: 0.0470
etree insert_from_document (T2,T4 ) 2.6039 2.7250 2.7508 msec/pass, best: 2.6039 ElementTree insert_from_document (T2,T4 ) 2.2039 2.5299 2.5815 msec/pass, best: 2.2039 cElementTree insert_from_document (T2,T4 ) 0.1059 0.1134 0.1127 msec/pass, best: 0.1059
etree insert_from_document (T3,T1 ) 19.8009 25.0803 23.4647 msec/pass, best: 19.8009 ElementTree insert_from_document (T3,T1 ) 3.0262 4.1957 3.7703 msec/pass, best: 3.0262 cElementTree insert_from_document (T3,T1 ) 0.1040 0.1033 0.0954 msec/pass, best: 0.0954
etree insert_from_document (T3,T2 ) 15.1248 15.4261 16.6505 msec/pass, best: 15.1248 ElementTree insert_from_document (T3,T2 ) 7.8220 8.0915 8.5682 msec/pass, best: 7.8220 cElementTree insert_from_document (T3,T2 ) 1.1612 1.1757 1.2460 msec/pass, best: 1.1612
etree insert_from_document (T3,T4 ) 0.1667 0.4397 0.1828 msec/pass, best: 0.1667 ElementTree insert_from_document (T3,T4 ) 0.2935 0.2366 0.2363 msec/pass, best: 0.2363 cElementTree insert_from_document (T3,T4 ) 0.0566 0.0568 0.0734 msec/pass, best: 0.0566
etree insert_from_document (T4,T1 ) 23.9678 24.8289 24.7205 msec/pass, best: 23.9678 ElementTree insert_from_document (T4,T1 ) 0.2536 0.2725 0.2724 msec/pass, best: 0.2536 cElementTree insert_from_document (T4,T1 ) 0.0970 0.0986 0.0969 msec/pass, best: 0.0969
etree insert_from_document (T4,T2 ) 17.4943 17.6244 18.9083 msec/pass, best: 17.4943 ElementTree insert_from_document (T4,T2 ) 6.4159 6.5594 6.6816 msec/pass, best: 6.4159 cElementTree insert_from_document (T4,T2 ) 1.1943 1.1961 1.2079 msec/pass, best: 1.1943
etree insert_from_document (T4,T3 ) 0.5701 0.5741 0.5685 msec/pass, best: 0.5685 ElementTree insert_from_document (T4,T3 ) 0.1002 0.0967 0.0964 msec/pass, best: 0.0964 cElementTree insert_from_document (T4,T3 ) 0.0336 0.0379 0.0327 msec/pass, best: 0.0327
etree remove_children (T1 ) 6.9888 7.1405 7.1501 msec/pass, best: 6.9888 ElementTree remove_children (T1 ) 10.2381 10.3145 10.3398 msec/pass, best: 10.2381 cElementTree remove_children (T1 ) 1.1740 1.2146 1.2070 msec/pass, best: 1.1740
etree remove_children (T2 ) 8.0086 8.0501 8.0835 msec/pass, best: 8.0086 ElementTree remove_children (T2 ) 171.0074 165.6075 174.0978 msec/pass, best: 165.6075 cElementTree remove_children (T2 ) 4.5207 4.6791 4.7362 msec/pass, best: 4.5207
etree remove_children (T3 ) 0.5668 0.5530 0.5762 msec/pass, best: 0.5530 ElementTree remove_children (T3 ) 3.1857 3.1828 3.2323 msec/pass, best: 3.1828 cElementTree remove_children (T3 ) 0.2588 0.2608 0.2584 msec/pass, best: 0.2584
etree remove_children (T4 ) 0.0684 0.0753 0.0700 msec/pass, best: 0.0684 ElementTree remove_children (T4 ) 0.4579 0.4570 0.4711 msec/pass, best: 0.4570 cElementTree remove_children (T4 ) 0.0209 0.0210 0.0207 msec/pass, best: 0.0207
etree remove_children_reversed (T1 ) 11.9009 11.9145 11.8435 msec/pass, best: 11.8435 ElementTree remove_children_reversed (T1 ) 19.3379 18.8514 20.4126 msec/pass, best: 18.8514 cElementTree remove_children_reversed (T1 ) 2.2264 2.3288 2.2562 msec/pass, best: 2.2264
etree remove_children_reversed (T2 ) 12.9722 14.8116 12.7335 msec/pass, best: 12.7335 ElementTree remove_children_reversed (T2 ) 673.7946 684.9349 669.5603 msec/pass, best: 669.5603 cElementTree remove_children_reversed (T2 ) 13.0890 12.8449 12.9061 msec/pass, best: 12.8449
etree remove_children_reversed (T3 ) 0.5227 0.5268 0.5229 msec/pass, best: 0.5227 ElementTree remove_children_reversed (T3 ) 4.4230 5.2407 4.4353 msec/pass, best: 4.4230 cElementTree remove_children_reversed (T3 ) 0.4222 0.4244 0.4180 msec/pass, best: 0.4180
etree remove_children_reversed (T4 ) 0.0738 0.0731 0.0736 msec/pass, best: 0.0731 ElementTree remove_children_reversed (T4 ) 1.6637 1.6356 1.6363 msec/pass, best: 1.6356 cElementTree remove_children_reversed (T4 ) 0.0510 0.0484 0.0499 msec/pass, best: 0.0484
etree reorder (T1 ) 14.5806 16.2235 15.8373 msec/pass, best: 14.5806 ElementTree reorder (T1 ) 0.8176 2.1794 2.6297 msec/pass, best: 0.8176 cElementTree reorder (T1 ) 0.0634 0.0676 0.0605 msec/pass, best: 0.0605
etree reorder (T2 ) 18.1266 18.8180 18.8672 msec/pass, best: 18.1266 ElementTree reorder (T2 ) 5.5504 5.5097 5.7170 msec/pass, best: 5.5097 cElementTree reorder (T2 ) 1.1481 1.1568 1.1591 msec/pass, best: 1.1481
etree reorder (T3 ) 0.0130 0.0156 0.0155 msec/pass, best: 0.0130 ElementTree reorder (T3 ) 0.0437 0.0459 0.0446 msec/pass, best: 0.0437 cElementTree reorder (T3 ) 0.0244 0.0228 0.0230 msec/pass, best: 0.0228
etree reorder (T4 ) 0.0731 0.0780 0.0734 msec/pass, best: 0.0731 ElementTree reorder (T4 ) 0.1282 0.1242 0.1238 msec/pass, best: 0.1238 cElementTree reorder (T4 ) 0.0217 0.0218 0.0220 msec/pass, best: 0.0217
etree reorder_slice (T1 ) 14.5325 17.0733 15.9148 msec/pass, best: 14.5325 ElementTree reorder_slice (T1 ) 0.5114 1.8668 2.6623 msec/pass, best: 0.5114 cElementTree reorder_slice (T1 ) 0.0629 0.0636 0.0633 msec/pass, best: 0.0629
etree reorder_slice (T2 ) 19.0733 22.6151 20.3017 msec/pass, best: 19.0733 ElementTree reorder_slice (T2 ) 5.5578 5.8006 5.7953 msec/pass, best: 5.5578 cElementTree reorder_slice (T2 ) 1.1598 1.1522 1.1647 msec/pass, best: 1.1522
etree reorder_slice (T3 ) 0.0145 0.0135 0.0133 msec/pass, best: 0.0133 ElementTree reorder_slice (T3 ) 0.0421 0.0435 0.0434 msec/pass, best: 0.0421 cElementTree reorder_slice (T3 ) 0.0225 0.0221 0.0229 msec/pass, best: 0.0221
etree reorder_slice (T4 ) 0.0796 0.0779 0.0786 msec/pass, best: 0.0779 ElementTree reorder_slice (T4 ) 0.1390 0.1334 0.1426 msec/pass, best: 0.1334 cElementTree reorder_slice (T4 ) 0.0259 0.0219 0.0217 msec/pass, best: 0.0217
etree replace_children (T1 ) 7.4184 7.9814 8.2130 msec/pass, best: 7.4184 ElementTree replace_children (T1 ) 19.4359 19.5994 19.5809 msec/pass, best: 19.4359 cElementTree replace_children (T1 ) 2.0112 2.0221 2.0522 msec/pass, best: 2.0112
etree replace_children (T2 ) 17.0823 19.4184 20.3127 msec/pass, best: 17.0823 ElementTree replace_children (T2 ) 26.4215 26.6201 26.9273 msec/pass, best: 26.4215 cElementTree replace_children (T2 ) 2.9001 2.8836 2.8504 msec/pass, best: 2.8504
etree replace_children (T3 ) 0.6777 0.6651 0.6562 msec/pass, best: 0.6562 ElementTree replace_children (T3 ) 5.2446 4.5064 4.6229 msec/pass, best: 4.5064 cElementTree replace_children (T3 ) 0.4712 0.4945 0.4554 msec/pass, best: 0.4554
etree replace_children (T4 ) 0.4540 0.5324 0.4619 msec/pass, best: 0.4540 ElementTree replace_children (T4 ) 0.3328 0.3317 0.3333 msec/pass, best: 0.3317 cElementTree replace_children (T4 ) 0.0388 0.0385 0.0382 msec/pass, best: 0.0382
etree rotate_children (T1 ) 27.4596 27.0203 27.8874 msec/pass, best: 27.0203 ElementTree rotate_children (T1 ) 1.2427 1.2087 1.1850 msec/pass, best: 1.1850 cElementTree rotate_children (T1 ) 0.1627 0.1654 0.1633 msec/pass, best: 0.1627
etree rotate_children (T2 ) 7.2919 7.9870 7.1657 msec/pass, best: 7.1657 ElementTree rotate_children (T2 ) 2.2686 2.5049 1.8473 msec/pass, best: 1.8473 cElementTree rotate_children (T2 ) 0.4213 0.4067 0.4378 msec/pass, best: 0.4067
etree rotate_children (T3 ) 3.9796 3.9531 4.7094 msec/pass, best: 3.9531 ElementTree rotate_children (T3 ) 0.6443 0.5682 0.5796 msec/pass, best: 0.5682 cElementTree rotate_children (T3 ) 0.1355 0.1322 0.1368 msec/pass, best: 0.1322
etree rotate_children (T4 ) 0.3995 0.3204 0.3173 msec/pass, best: 0.3173 ElementTree rotate_children (T4 ) 0.5233 0.5198 0.5278 msec/pass, best: 0.5198 cElementTree rotate_children (T4 ) 0.1210 0.1199 0.1275 msec/pass, best: 0.1199
etree set_attributes (T1 ) 6.0470 6.6676 6.6880 msec/pass, best: 6.0470 ElementTree set_attributes (T1 ) 0.1447 0.1446 0.1479 msec/pass, best: 0.1446 cElementTree set_attributes (T1 ) 0.0691 0.0665 0.0666 msec/pass, best: 0.0665
etree set_attributes (T2 ) 8.8447 9.5652 9.4956 msec/pass, best: 8.8447 ElementTree set_attributes (T2 ) 1.6476 1.6582 1.6447 msec/pass, best: 1.6447 cElementTree set_attributes (T2 ) 0.7479 0.7352 0.7474 msec/pass, best: 0.7352
etree set_attributes (T3 ) 0.0568 0.0928 0.0583 msec/pass, best: 0.0568 ElementTree set_attributes (T3 ) 0.0625 0.0613 0.0658 msec/pass, best: 0.0613 cElementTree set_attributes (T3 ) 0.0272 0.0240 0.0255 msec/pass, best: 0.0240
etree set_attributes (T4 ) 0.1534 0.1562 0.1530 msec/pass, best: 0.1530 ElementTree set_attributes (T4 ) 0.0779 0.0759 0.0762 msec/pass, best: 0.0759 cElementTree set_attributes (T4 ) 0.0268 0.0282 0.0261 msec/pass, best: 0.0261
etree setget_attributes (T1 ) 5.4190 6.6268 6.6813 msec/pass, best: 5.4190 ElementTree setget_attributes (T1 ) 0.2305 0.2438 0.2276 msec/pass, best: 0.2276 cElementTree setget_attributes (T1 ) 0.1521 0.0983 0.0920 msec/pass, best: 0.0920
etree setget_attributes (T2 ) 10.5145 11.2669 11.2211 msec/pass, best: 10.5145 ElementTree setget_attributes (T2 ) 3.2395 3.1300 3.1196 msec/pass, best: 3.1196 cElementTree setget_attributes (T2 ) 1.1320 1.1729 1.1571 msec/pass, best: 1.1320
etree setget_attributes (T3 ) 0.0784 0.0807 0.0913 msec/pass, best: 0.0784 ElementTree setget_attributes (T3 ) 0.0977 0.0835 0.0889 msec/pass, best: 0.0835 cElementTree setget_attributes (T3 ) 0.0295 0.0285 0.0333 msec/pass, best: 0.0285
etree setget_attributes (T4 ) 0.2418 0.2438 0.2392 msec/pass, best: 0.2392 ElementTree setget_attributes (T4 ) 0.1654 0.1779 0.1539 msec/pass, best: 0.1539 cElementTree setget_attributes (T4 ) 0.0476 0.0458 0.0506 msec/pass, best: 0.0458

Stefan Behnel wrote:
A special beauty here is the deepcopy benchmark.
python's deepcopy implementation is horribly slow. I guess I have to do some- thing about that in cET... I currently recommend the following approach for mass- production of fresh trees from a given template:
http://effbot.python-hosting.com/file/stuff/sandbox/elementlib/clone.py
:::
do you have any tag/attribute/text access tests in your benchmark, btw? one big ad- vantage of cElementTree is that cET builds real Python objects on the way in; there's (usually) no extra conversion cost when you access an Element attribute.
</F>

Fredrik Lundh wrote:
Stefan Behnel wrote:
A special beauty here is the deepcopy benchmark.
python's deepcopy implementation is horribly slow. I guess I have to do some- thing about that in cET... I currently recommend the following approach for mass- production of fresh trees from a given template:
http://effbot.python-hosting.com/file/stuff/sandbox/elementlib/clone.py
Thanks, that makes the benchmark suite run considerably faster. You should consider putting something like this into Element.__deepcopy__.
do you have any tag/attribute/text access tests in your benchmark, btw? one big ad- vantage of cElementTree is that cET builds real Python objects on the way in; there's (usually) no extra conversion cost when you access an Element attribute.
I just added some and attached the result. Feel free to send me some more as a patch against
http://codespeak.net/svn/lxml/trunk/bench.py
You can configure how they are called by using a decorator:
def bench_tag(self, root): for child in root: child.tag
@with_text(utext=True, text=True, no_text=True) def bench_text(self, root): for child in root: child.text
The first is called on all trees without attributes and text, the second one is called three times for each tree: without text (-), with ASCII text (S) and with Unicode text (U). Look at the tree setup in bench.py and the results in the attached log to see what I mean.
There is also a decorator @with_attributes(True/False)
Stefan
Preparing test suites and trees ... Running benchmark on lxe, ET, cET
Setup times for trees in seconds: lxe: -- T- U- -A TA UA T1: 0.1907 0.1578 0.1588 0.1592 0.1583 0.1581 T2: 0.1376 0.1388 0.1383 0.1424 0.1421 0.1425 T3: 0.0319 0.0308 0.0322 0.0495 0.0511 0.0500 T4: 0.0007 0.0007 0.0007 0.0012 0.0013 0.0011 ET : -- T- U- -A TA UA T1: 0.2313 0.2763 0.2166 0.2659 0.2128 0.2509 T2: 0.2578 0.1883 0.2500 0.1920 0.2361 0.2668 T3: 0.0512 0.0544 0.0518 0.0573 0.0538 0.0584 T4: 0.0009 0.0008 0.0007 0.0007 0.0008 0.0008 cET: -- T- U- -A TA UA T1: 0.0361 0.0348 0.0355 0.0554 0.0351 0.0360 T2: 0.0143 0.0139 0.0143 0.0149 0.0142 0.0140 T3: 0.0088 0.0088 0.0085 0.0123 0.0121 0.0170 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001
lxe: tag (-- T1 ) 0.1260 0.1212 0.1831 msec/pass, best: 0.1212 ET : tag (-- T1 ) 0.1113 0.1098 0.1132 msec/pass, best: 0.1098 cET: tag (-- T1 ) 0.0406 0.0404 0.0432 msec/pass, best: 0.0404
lxe: tag (-- T2 ) 1.1775 1.2149 1.1726 msec/pass, best: 1.1726 ET : tag (-- T2 ) 1.0539 1.0603 1.0587 msec/pass, best: 1.0539 cET: tag (-- T2 ) 0.3180 0.3406 0.3449 msec/pass, best: 0.3180
lxe: tag (-- T3 ) 0.0353 0.0319 0.0383 msec/pass, best: 0.0319 ET : tag (-- T3 ) 0.0562 0.0675 0.0608 msec/pass, best: 0.0562 cET: tag (-- T3 ) 0.0099 0.0075 0.0077 msec/pass, best: 0.0075
lxe: tag (-- T4 ) 0.0665 0.0510 0.0500 msec/pass, best: 0.0500 ET : tag (-- T4 ) 0.0503 0.0478 0.0482 msec/pass, best: 0.0478 cET: tag (-- T4 ) 0.0158 0.0152 0.0142 msec/pass, best: 0.0142
lxe: text (-- T1 ) 0.1227 0.1120 0.1130 msec/pass, best: 0.1120 ET : text (-- T1 ) 0.1086 0.1273 0.1375 msec/pass, best: 0.1086 cET: text (-- T1 ) 0.0403 0.0407 0.0424 msec/pass, best: 0.0403
lxe: text (S- T1 ) 0.1112 0.1299 0.1150 msec/pass, best: 0.1112 ET : text (S- T1 ) 0.1107 0.1101 0.1113 msec/pass, best: 0.1101 cET: text (S- T1 ) 0.0405 0.0409 0.0404 msec/pass, best: 0.0404
lxe: text (U- T1 ) 0.1124 0.1315 0.1174 msec/pass, best: 0.1124 ET : text (U- T1 ) 0.1152 0.1132 0.1127 msec/pass, best: 0.1127 cET: text (U- T1 ) 0.0448 0.0410 0.0401 msec/pass, best: 0.0401
lxe: text (-- T2 ) 1.0198 1.0201 1.0330 msec/pass, best: 1.0198 ET : text (-- T2 ) 1.0598 1.0430 1.0629 msec/pass, best: 1.0430 cET: text (-- T2 ) 0.3317 0.3290 0.3309 msec/pass, best: 0.3290
lxe: text (S- T2 ) 0.9765 0.9936 1.0045 msec/pass, best: 0.9765 ET : text (S- T2 ) 1.0950 1.0390 1.0646 msec/pass, best: 1.0390 cET: text (S- T2 ) 0.3151 0.3374 0.3238 msec/pass, best: 0.3151
lxe: text (U- T2 ) 0.9843 1.0201 1.0384 msec/pass, best: 0.9843 ET : text (U- T2 ) 1.0933 1.1132 1.0576 msec/pass, best: 1.0576 cET: text (U- T2 ) 0.3155 0.3490 0.3386 msec/pass, best: 0.3155
lxe: text (-- T3 ) 0.0318 0.0316 0.0300 msec/pass, best: 0.0300 ET : text (-- T3 ) 0.0568 0.0580 0.0544 msec/pass, best: 0.0544 cET: text (-- T3 ) 0.0095 0.0070 0.0071 msec/pass, best: 0.0070
lxe: text (S- T3 ) 0.0372 0.0339 0.0323 msec/pass, best: 0.0323 ET : text (S- T3 ) 0.0545 0.0548 0.0573 msec/pass, best: 0.0545 cET: text (S- T3 ) 0.0080 0.0065 0.0082 msec/pass, best: 0.0065
lxe: text (U- T3 ) 0.0349 0.0377 0.0322 msec/pass, best: 0.0322 ET : text (U- T3 ) 0.0537 0.0556 0.0575 msec/pass, best: 0.0537 cET: text (U- T3 ) 0.0104 0.0090 0.0090 msec/pass, best: 0.0090
lxe: text (-- T4 ) 0.0408 0.0433 0.0421 msec/pass, best: 0.0408 ET : text (-- T4 ) 0.3062 0.0502 0.0475 msec/pass, best: 0.0475 cET: text (-- T4 ) 0.0184 0.0152 0.0148 msec/pass, best: 0.0148
lxe: text (S- T4 ) 0.0399 0.0401 0.0416 msec/pass, best: 0.0399 ET : text (S- T4 ) 0.0526 0.0493 0.0490 msec/pass, best: 0.0490 cET: text (S- T4 ) 0.0159 0.0154 0.0152 msec/pass, best: 0.0152
lxe: text (U- T4 ) 0.0422 0.0404 0.0404 msec/pass, best: 0.0404 ET : text (U- T4 ) 0.0499 0.0514 0.0492 msec/pass, best: 0.0492 cET: text (U- T4 ) 0.0151 0.0152 0.0153 msec/pass, best: 0.0151
participants (2)
-
Fredrik Lundh
-
Stefan Behnel