Mailman 3 June 2006 - lxml - The Python XML Toolkit

[lxml-dev] Benchmark results on parse/iterparse
by Stefan Behnel 10 Aug '06

10 Aug '06

Hi all, I extended bench.py to support benchmarks on serialised XML data, so now there are micro-benchmarks for the plain parser performance. I attached some results which are pretty impressive. Expat claims to be the fastest XML parser on earth - and it looks like there's some truth in that statement. cET (1.0.5) beats lxml (trunk and iterparse branch) in virtually every parser benchmark. It's even up to 80% faster for large trees. On the other hand, cET is still much slower for serialisation (up to 30x!), so lxml easily wins the round-trip benchmarks for reading and writing XML and is some 3-4x faster at the end. For the iterparse benchmarks, lxml suffers slightly from its object creation overhead (and maybe also the young implementation), but the overhead is acceptable compared to the overall parse time. As expected, the additional overhead for cET is extremely small here, it's almost as fast as the plain parser. So, for (typical?) iterparse() applications that extract small amounts of data from large XML streams on the fly, cET is pretty much unbeatable. For everything else, lxml is still my favourite. :) Stefan Preparing test suites and trees ... Running benchmark on lxe, cET, ET Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.1323 0.1295 0.1309 0.1400 0.1294 0.1281 T2: 0.1345 0.1326 0.1324 0.1439 0.1443 0.1436 T3: 0.0358 0.0286 0.0290 0.0872 0.0852 0.0867 T4: 0.0007 0.0006 0.0006 0.0018 0.0018 0.0018 cET: -- S- U- -A SA UA T1: 0.0427 0.0433 0.0421 0.0418 0.0426 0.0425 T2: 0.0440 0.0426 0.0448 0.0496 0.0424 0.0445 T3: 0.0109 0.0102 0.0110 0.0185 0.0146 0.0148 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 ET : -- S- U- -A SA UA T1: 0.2288 0.2882 0.2279 0.2701 0.3036 0.2273 T2: 0.2972 0.2352 0.2890 0.3218 0.2337 0.3112 T3: 0.0533 0.0572 0.0534 0.0616 0.0573 0.0949 T4: 0.0009 0.0009 0.0008 0.0008 0.0009 0.0009 lxe: iterparse_stringIO (S-X T1 ) 222.9110 msec/pass, best of ( 227.3209 224.8156 222.9110 ) cET: iterparse_stringIO (S-X T1 ) 38.3651 msec/pass, best of ( 38.6973 38.3651 38.4331 ) ET : iterparse_stringIO (S-X T1 ) 362.5654 msec/pass, best of ( 399.9243 364.9337 362.5654 ) lxe: iterparse_stringIO (SAX T1 ) 223.7216 msec/pass, best of ( 229.2649 225.4670 223.7216 ) cET: iterparse_stringIO (SAX T1 ) 38.7985 msec/pass, best of ( 38.8003 38.7985 39.0118 ) ET : iterparse_stringIO (SAX T1 ) 365.6647 msec/pass, best of ( 365.7858 368.4628 365.6647 ) lxe: iterparse_stringIO (U-X T1 ) 223.2167 msec/pass, best of ( 228.4652 225.6853 223.2167 ) cET: iterparse_stringIO (U-X T1 ) 38.6407 msec/pass, best of ( 38.9079 38.6407 38.7212 ) ET : iterparse_stringIO (U-X T1 ) 361.4145 msec/pass, best of ( 361.5446 362.3198 361.4145 ) lxe: iterparse_stringIO (UAX T1 ) 223.9983 msec/pass, best of ( 230.2715 224.7750 223.9983 ) cET: iterparse_stringIO (UAX T1 ) 38.9739 msec/pass, best of ( 39.1283 38.9739 38.9918 ) ET : iterparse_stringIO (UAX T1 ) 365.9984 msec/pass, best of ( 365.9984 368.7172 367.6013 ) lxe: iterparse_stringIO (S-X T2 ) 232.9033 msec/pass, best of ( 237.3149 233.7498 232.9033 ) cET: iterparse_stringIO (S-X T2 ) 41.1875 msec/pass, best of ( 41.4992 41.2292 41.1875 ) ET : iterparse_stringIO (S-X T2 ) 383.4325 msec/pass, best of ( 383.8255 386.8770 383.4325 ) lxe: iterparse_stringIO (SAX T2 ) 248.6702 msec/pass, best of ( 262.2535 251.9383 248.6702 ) cET: iterparse_stringIO (SAX T2 ) 46.2867 msec/pass, best of ( 46.3457 46.2998 46.2867 ) ET : iterparse_stringIO (SAX T2 ) 406.4786 msec/pass, best of ( 407.3699 406.4786 408.3880 ) lxe: iterparse_stringIO (U-X T2 ) 232.0476 msec/pass, best of ( 239.1271 232.3972 232.0476 ) cET: iterparse_stringIO (U-X T2 ) 41.0636 msec/pass, best of ( 41.0744 41.0636 41.1089 ) ET : iterparse_stringIO (U-X T2 ) 383.7441 msec/pass, best of ( 383.8562 384.8858 383.7441 ) lxe: iterparse_stringIO (UAX T2 ) 250.2190 msec/pass, best of ( 255.8399 250.2190 250.2771 ) cET: iterparse_stringIO (UAX T2 ) 46.3599 msec/pass, best of ( 46.5231 46.3599 47.1866 ) ET : iterparse_stringIO (UAX T2 ) 406.6847 msec/pass, best of ( 410.7563 406.6847 415.3738 ) lxe: iterparse_stringIO (S-X T3 ) 21.0934 msec/pass, best of ( 21.1457 21.0934 21.1917 ) cET: iterparse_stringIO (S-X T3 ) 9.9643 msec/pass, best of ( 9.9643 9.9952 9.9998 ) ET : iterparse_stringIO (S-X T3 ) 119.0730 msec/pass, best of ( 119.0730 119.2219 119.3496 ) lxe: iterparse_stringIO (SAX T3 ) 46.2915 msec/pass, best of ( 51.1358 46.5834 46.2915 ) cET: iterparse_stringIO (SAX T3 ) 41.7721 msec/pass, best of ( 41.7721 43.1893 41.8128 ) ET : iterparse_stringIO (SAX T3 ) 252.5569 msec/pass, best of ( 253.2359 252.5569 257.7912 ) lxe: iterparse_stringIO (U-X T3 ) 21.2006 msec/pass, best of ( 21.6766 21.2006 21.6195 ) cET: iterparse_stringIO (U-X T3 ) 10.1142 msec/pass, best of ( 10.1593 10.1525 10.1142 ) ET : iterparse_stringIO (U-X T3 ) 118.9059 msec/pass, best of ( 119.9316 118.9059 119.0838 ) lxe: iterparse_stringIO (UAX T3 ) 46.0920 msec/pass, best of ( 50.7096 46.3383 46.0920 ) cET: iterparse_stringIO (UAX T3 ) 41.7240 msec/pass, best of ( 41.9300 41.7240 41.9742 ) ET : iterparse_stringIO (UAX T3 ) 253.6689 msec/pass, best of ( 253.6689 253.6885 255.1465 ) lxe: iterparse_stringIO (S-X T4 ) 1.0302 msec/pass, best of ( 1.1030 1.0365 1.0302 ) cET: iterparse_stringIO (S-X T4 ) 0.3072 msec/pass, best of ( 0.3244 0.3115 0.3072 ) ET : iterparse_stringIO (S-X T4 ) 2.8629 msec/pass, best of ( 2.8674 2.8629 2.9106 ) lxe: iterparse_stringIO (SAX T4 ) 1.6899 msec/pass, best of ( 1.6899 1.6941 1.6969 ) cET: iterparse_stringIO (SAX T4 ) 0.9437 msec/pass, best of ( 0.9663 0.9437 0.9659 ) ET : iterparse_stringIO (SAX T4 ) 5.9165 msec/pass, best of ( 5.9764 5.9165 5.9525 ) lxe: iterparse_stringIO (U-X T4 ) 1.0378 msec/pass, best of ( 1.0475 1.0444 1.0378 ) cET: iterparse_stringIO (U-X T4 ) 0.3046 msec/pass, best of ( 0.3141 0.3144 0.3046 ) ET : iterparse_stringIO (U-X T4 ) 2.8697 msec/pass, best of ( 2.8855 2.8799 2.8697 ) lxe: iterparse_stringIO (UAX T4 ) 1.7076 msec/pass, best of ( 1.7076 1.7481 1.7081 ) cET: iterparse_stringIO (UAX T4 ) 0.9469 msec/pass, best of ( 0.9611 0.9520 0.9469 ) ET : iterparse_stringIO (UAX T4 ) 6.0304 msec/pass, best of ( 6.0437 6.0304 6.0345 ) lxe: iterparse_stringIO_clear (S-X T1 ) 234.3113 msec/pass, best of ( 234.3113 246.4326 256.5073 ) cET: iterparse_stringIO_clear (S-X T1 ) 45.1311 msec/pass, best of ( 45.1311 46.6361 45.1353 ) ET : iterparse_stringIO_clear (S-X T1 ) 398.4236 msec/pass, best of ( 411.1450 417.0196 398.4236 ) lxe: iterparse_stringIO_clear (SAX T1 ) 232.9236 msec/pass, best of ( 234.6838 235.7229 232.9236 ) cET: iterparse_stringIO_clear (SAX T1 ) 44.5997 msec/pass, best of ( 44.5997 44.6131 44.6254 ) ET : iterparse_stringIO_clear (SAX T1 ) 391.3037 msec/pass, best of ( 391.3037 392.3368 391.4310 ) lxe: iterparse_stringIO_clear (U-X T1 ) 230.3221 msec/pass, best of ( 240.0502 230.3221 230.4537 ) cET: iterparse_stringIO_clear (U-X T1 ) 44.5020 msec/pass, best of ( 44.5403 44.5163 44.5020 ) ET : iterparse_stringIO_clear (U-X T1 ) 392.5360 msec/pass, best of ( 392.5360 392.7761 392.8283 ) lxe: iterparse_stringIO_clear (UAX T1 ) 232.4709 msec/pass, best of ( 233.1237 232.5471 232.4709 ) cET: iterparse_stringIO_clear (UAX T1 ) 44.6517 msec/pass, best of ( 44.8149 44.6950 44.6517 ) ET : iterparse_stringIO_clear (UAX T1 ) 391.4469 msec/pass, best of ( 393.7384 393.8401 391.4469 ) lxe: iterparse_stringIO_clear (S-X T2 ) 237.8001 msec/pass, best of ( 244.5833 239.2776 237.8001 ) cET: iterparse_stringIO_clear (S-X T2 ) 46.5421 msec/pass, best of ( 46.5820 46.5421 46.6870 ) ET : iterparse_stringIO_clear (S-X T2 ) 404.6504 msec/pass, best of ( 405.8105 404.6504 406.5170 ) lxe: iterparse_stringIO_clear (SAX T2 ) 256.2644 msec/pass, best of ( 256.2644 256.6346 256.5515 ) cET: iterparse_stringIO_clear (SAX T2 ) 51.3243 msec/pass, best of ( 51.4606 51.3311 51.3243 ) ET : iterparse_stringIO_clear (SAX T2 ) 428.6695 msec/pass, best of ( 428.6695 429.9673 429.0031 ) lxe: iterparse_stringIO_clear (U-X T2 ) 238.2270 msec/pass, best of ( 238.2270 238.4732 238.4163 ) cET: iterparse_stringIO_clear (U-X T2 ) 46.8345 msec/pass, best of ( 46.8345 46.9356 47.8009 ) ET : iterparse_stringIO_clear (U-X T2 ) 407.4448 msec/pass, best of ( 407.4448 417.0420 412.9830 ) lxe: iterparse_stringIO_clear (UAX T2 ) 255.1546 msec/pass, best of ( 255.1546 256.3821 258.1754 ) cET: iterparse_stringIO_clear (UAX T2 ) 51.4117 msec/pass, best of ( 51.7307 51.6235 51.4117 ) ET : iterparse_stringIO_clear (UAX T2 ) 427.9256 msec/pass, best of ( 428.1150 435.2808 427.9256 ) lxe: iterparse_stringIO_clear (S-X T3 ) 24.3242 msec/pass, best of ( 24.3841 24.3453 24.3242 ) cET: iterparse_stringIO_clear (S-X T3 ) 11.3448 msec/pass, best of ( 11.5487 11.3886 11.3448 ) ET : iterparse_stringIO_clear (S-X T3 ) 127.7567 msec/pass, best of ( 128.1610 127.7567 128.4989 ) lxe: iterparse_stringIO_clear (SAX T3 ) 53.2374 msec/pass, best of ( 53.4208 53.2374 53.7359 ) cET: iterparse_stringIO_clear (SAX T3 ) 38.3572 msec/pass, best of ( 38.4457 38.3572 38.3621 ) ET : iterparse_stringIO_clear (SAX T3 ) 254.7633 msec/pass, best of ( 257.4664 254.7633 255.7398 ) lxe: iterparse_stringIO_clear (U-X T3 ) 24.6667 msec/pass, best of ( 24.7798 24.6667 24.7138 ) cET: iterparse_stringIO_clear (U-X T3 ) 11.3932 msec/pass, best of ( 11.5611 11.3932 11.5425 ) ET : iterparse_stringIO_clear (U-X T3 ) 127.7254 msec/pass, best of ( 127.8360 127.9421 127.7254 ) lxe: iterparse_stringIO_clear (UAX T3 ) 52.9185 msec/pass, best of ( 52.9185 53.2012 52.9428 ) cET: iterparse_stringIO_clear (UAX T3 ) 38.2394 msec/pass, best of ( 38.4341 38.2394 38.3053 ) ET : iterparse_stringIO_clear (UAX T3 ) 253.7129 msec/pass, best of ( 254.9093 254.8415 253.7129 ) lxe: iterparse_stringIO_clear (S-X T4 ) 1.1246 msec/pass, best of ( 1.2300 1.1246 1.1279 ) cET: iterparse_stringIO_clear (S-X T4 ) 0.3431 msec/pass, best of ( 0.3467 0.3463 0.3431 ) ET : iterparse_stringIO_clear (S-X T4 ) 3.0261 msec/pass, best of ( 3.0261 3.0825 3.0340 ) lxe: iterparse_stringIO_clear (SAX T4 ) 1.8222 msec/pass, best of ( 1.8494 1.8222 1.8429 ) cET: iterparse_stringIO_clear (SAX T4 ) 0.9881 msec/pass, best of ( 0.9933 0.9881 1.0033 ) ET : iterparse_stringIO_clear (SAX T4 ) 6.1015 msec/pass, best of ( 6.1317 6.1015 6.1112 ) lxe: iterparse_stringIO_clear (U-X T4 ) 1.1314 msec/pass, best of ( 1.1320 1.1569 1.1314 ) cET: iterparse_stringIO_clear (U-X T4 ) 0.3512 msec/pass, best of ( 0.3577 0.3564 0.3512 ) ET : iterparse_stringIO_clear (U-X T4 ) 3.0102 msec/pass, best of ( 3.0351 3.0412 3.0102 ) lxe: iterparse_stringIO_clear (UAX T4 ) 1.8182 msec/pass, best of ( 1.8478 1.8182 1.8598 ) cET: iterparse_stringIO_clear (UAX T4 ) 0.9898 msec/pass, best of ( 0.9898 0.9990 0.9943 ) ET : iterparse_stringIO_clear (UAX T4 ) 6.1685 msec/pass, best of ( 6.2379 6.2236 6.1685 ) lxe: parse_stringIO (S-X T1 ) 170.6294 msec/pass, best of ( 173.7078 175.0854 170.6294 ) cET: parse_stringIO (S-X T1 ) 31.2479 msec/pass, best of ( 31.4530 31.2479 31.2610 ) ET : parse_stringIO (S-X T1 ) 320.9497 msec/pass, best of ( 321.8702 322.3815 320.9497 ) lxe: parse_stringIO (SAX T1 ) 171.2800 msec/pass, best of ( 171.2800 175.6441 173.7187 ) cET: parse_stringIO (SAX T1 ) 31.3205 msec/pass, best of ( 32.3134 31.3205 31.3392 ) ET : parse_stringIO (SAX T1 ) 323.7324 msec/pass, best of ( 323.7324 323.7890 328.4510 ) lxe: parse_stringIO (U-X T1 ) 171.6893 msec/pass, best of ( 172.5678 174.2148 171.6893 ) cET: parse_stringIO (U-X T1 ) 31.2387 msec/pass, best of ( 32.1226 31.2387 31.4539 ) ET : parse_stringIO (U-X T1 ) 321.3753 msec/pass, best of ( 323.4447 321.7778 321.3753 ) lxe: parse_stringIO (UAX T1 ) 171.9517 msec/pass, best of ( 171.9517 174.0736 176.9163 ) cET: parse_stringIO (UAX T1 ) 31.5030 msec/pass, best of ( 34.8593 31.5832 31.5030 ) ET : parse_stringIO (UAX T1 ) 323.3877 msec/pass, best of ( 323.6831 323.3877 329.6061 ) lxe: parse_stringIO (S-X T2 ) 179.9828 msec/pass, best of ( 179.9828 181.4266 182.8689 ) cET: parse_stringIO (S-X T2 ) 33.5708 msec/pass, best of ( 34.6295 33.6188 33.5708 ) ET : parse_stringIO (S-X T2 ) 340.2606 msec/pass, best of ( 340.2606 340.5859 342.8782 ) lxe: parse_stringIO (SAX T2 ) 197.7678 msec/pass, best of ( 197.7678 199.4126 199.2439 ) cET: parse_stringIO (SAX T2 ) 38.9390 msec/pass, best of ( 40.0341 38.9390 39.0063 ) ET : parse_stringIO (SAX T2 ) 364.3468 msec/pass, best of ( 365.5197 364.3468 365.6537 ) lxe: parse_stringIO (U-X T2 ) 182.7992 msec/pass, best of ( 182.9719 182.9173 182.7992 ) cET: parse_stringIO (U-X T2 ) 33.7937 msec/pass, best of ( 34.7863 33.8375 33.7937 ) ET : parse_stringIO (U-X T2 ) 340.1892 msec/pass, best of ( 340.1892 341.4799 342.9891 ) lxe: parse_stringIO (UAX T2 ) 197.9534 msec/pass, best of ( 197.9534 199.2566 206.4527 ) cET: parse_stringIO (UAX T2 ) 38.9602 msec/pass, best of ( 40.2117 38.9602 39.0238 ) ET : parse_stringIO (UAX T2 ) 365.2490 msec/pass, best of ( 366.4139 366.0356 365.2490 ) lxe: parse_stringIO (S-X T3 ) 9.3905 msec/pass, best of ( 9.4029 9.3905 9.4458 ) cET: parse_stringIO (S-X T3 ) 7.6507 msec/pass, best of ( 7.6840 7.6507 7.7423 ) ET : parse_stringIO (S-X T3 ) 108.2791 msec/pass, best of ( 108.2791 110.4766 108.3301 ) lxe: parse_stringIO (SAX T3 ) 48.6069 msec/pass, best of ( 48.6069 48.8698 48.9391 ) cET: parse_stringIO (SAX T3 ) 39.8525 msec/pass, best of ( 40.9317 39.8525 39.9056 ) ET : parse_stringIO (SAX T3 ) 237.2957 msec/pass, best of ( 237.5886 237.2957 237.4677 ) lxe: parse_stringIO (U-X T3 ) 9.3709 msec/pass, best of ( 9.4539 9.3835 9.3709 ) cET: parse_stringIO (U-X T3 ) 7.5603 msec/pass, best of ( 7.7213 7.6098 7.5603 ) ET : parse_stringIO (U-X T3 ) 108.1631 msec/pass, best of ( 108.1631 108.8036 108.1696 ) lxe: parse_stringIO (UAX T3 ) 48.6735 msec/pass, best of ( 48.6735 48.8036 48.9602 ) cET: parse_stringIO (UAX T3 ) 39.7455 msec/pass, best of ( 40.0213 39.8740 39.7455 ) ET : parse_stringIO (UAX T3 ) 237.9971 msec/pass, best of ( 237.9971 238.7502 238.3795 ) lxe: parse_stringIO (S-X T4 ) 0.3790 msec/pass, best of ( 0.4000 0.3790 0.3791 ) cET: parse_stringIO (S-X T4 ) 0.2547 msec/pass, best of ( 0.2676 0.2568 0.2547 ) ET : parse_stringIO (S-X T4 ) 2.5935 msec/pass, best of ( 2.5935 2.5990 2.5970 ) lxe: parse_stringIO (SAX T4 ) 1.1852 msec/pass, best of ( 1.2173 1.1852 1.2512 ) cET: parse_stringIO (SAX T4 ) 0.8995 msec/pass, best of ( 0.9106 0.8995 0.9005 ) ET : parse_stringIO (SAX T4 ) 5.5938 msec/pass, best of ( 5.6073 5.6163 5.5938 ) lxe: parse_stringIO (U-X T4 ) 0.3752 msec/pass, best of ( 0.3799 0.3844 0.3752 ) cET: parse_stringIO (U-X T4 ) 0.2558 msec/pass, best of ( 0.2732 0.2599 0.2558 ) ET : parse_stringIO (U-X T4 ) 2.5950 msec/pass, best of ( 2.6326 2.6283 2.5950 ) lxe: parse_stringIO (UAX T4 ) 1.1881 msec/pass, best of ( 1.1881 1.2204 1.2204 ) cET: parse_stringIO (UAX T4 ) 0.8957 msec/pass, best of ( 0.9226 0.8992 0.8957 ) ET : parse_stringIO (UAX T4 ) 5.6234 msec/pass, best of ( 5.6234 5.6872 5.6961 ) lxe: write_utf8_parse_stringIO (S-T T1 ) 189.2201 msec/pass, best of ( 199.9139 189.2201 190.3712 ) cET: write_utf8_parse_stringIO (S-T T1 ) 795.5032 msec/pass, best of ( 844.4007 824.9086 795.5032 ) ET : write_utf8_parse_stringIO (S-T T1 ) 1138.0186 msec/pass, best of ( 1140.2423 1141.4423 1138.0186 ) lxe: write_utf8_parse_stringIO (SAT T1 ) 190.7312 msec/pass, best of ( 192.9438 190.7312 193.4001 ) cET: write_utf8_parse_stringIO (SAT T1 ) 801.8634 msec/pass, best of ( 807.4269 803.6557 801.8634 ) ET : write_utf8_parse_stringIO (SAT T1 ) 1149.8636 msec/pass, best of ( 1149.8636 1150.1960 1161.8435 ) lxe: write_utf8_parse_stringIO (U-T T1 ) 187.6989 msec/pass, best of ( 187.6989 190.4670 190.4346 ) cET: write_utf8_parse_stringIO (U-T T1 ) 797.0220 msec/pass, best of ( 799.1436 797.0220 797.8761 ) ET : write_utf8_parse_stringIO (U-T T1 ) 1139.1940 msec/pass, best of ( 1139.1940 1143.0199 1144.4894 ) lxe: write_utf8_parse_stringIO (UAT T1 ) 192.8713 msec/pass, best of ( 192.8713 194.3054 193.6350 ) cET: write_utf8_parse_stringIO (UAT T1 ) 801.4468 msec/pass, best of ( 801.4468 801.8691 803.9093 ) ET : write_utf8_parse_stringIO (UAT T1 ) 1149.7449 msec/pass, best of ( 1152.1680 1208.7331 1149.7449 ) lxe: write_utf8_parse_stringIO (S-T T2 ) 197.3249 msec/pass, best of ( 198.7984 197.3249 199.2573 ) cET: write_utf8_parse_stringIO (S-T T2 ) 836.6382 msec/pass, best of ( 836.6382 839.7354 838.3641 ) ET : write_utf8_parse_stringIO (S-T T2 ) 1198.1185 msec/pass, best of ( 1201.7853 1200.2708 1198.1185 ) lxe: write_utf8_parse_stringIO (SAT T2 ) 220.1455 msec/pass, best of ( 220.1455 221.5693 221.5681 ) cET: write_utf8_parse_stringIO (SAT T2 ) 933.0841 msec/pass, best of ( 933.7380 937.9234 933.0841 ) ET : write_utf8_parse_stringIO (SAT T2 ) 1311.6351 msec/pass, best of ( 1311.6351 1314.4752 1314.5881 ) lxe: write_utf8_parse_stringIO (U-T T2 ) 200.4080 msec/pass, best of ( 200.4080 203.1036 200.5778 ) cET: write_utf8_parse_stringIO (U-T T2 ) 836.8698 msec/pass, best of ( 836.8698 838.9036 840.1444 ) ET : write_utf8_parse_stringIO (U-T T2 ) 1198.3584 msec/pass, best of ( 1198.3584 1200.3080 1206.1083 ) lxe: write_utf8_parse_stringIO (UAT T2 ) 218.4633 msec/pass, best of ( 222.4844 218.4633 275.0107 ) cET: write_utf8_parse_stringIO (UAT T2 ) 926.8864 msec/pass, best of ( 934.6416 926.8864 930.0010 ) ET : write_utf8_parse_stringIO (UAT T2 ) 1310.0202 msec/pass, best of ( 1312.2715 1317.1640 1310.0202 ) lxe: write_utf8_parse_stringIO (S-T T3 ) 12.0866 msec/pass, best of ( 12.1740 12.0866 12.2349 ) cET: write_utf8_parse_stringIO (S-T T3 ) 109.3471 msec/pass, best of ( 109.3471 109.3881 109.5528 ) ET : write_utf8_parse_stringIO (S-T T3 ) 226.3420 msec/pass, best of ( 226.3420 226.8248 228.9372 ) lxe: write_utf8_parse_stringIO (SAT T3 ) 61.4568 msec/pass, best of ( 61.7457 61.9032 61.4568 ) cET: write_utf8_parse_stringIO (SAT T3 ) 608.1295 msec/pass, best of ( 608.6636 608.1295 613.7108 ) ET : write_utf8_parse_stringIO (SAT T3 ) 825.5513 msec/pass, best of ( 825.8673 827.0674 825.5513 ) lxe: write_utf8_parse_stringIO (U-T T3 ) 12.0974 msec/pass, best of ( 12.1823 12.0974 12.1191 ) cET: write_utf8_parse_stringIO (U-T T3 ) 109.5899 msec/pass, best of ( 109.6426 109.7568 109.5899 ) ET : write_utf8_parse_stringIO (U-T T3 ) 226.3300 msec/pass, best of ( 226.4436 227.5431 226.3300 ) lxe: write_utf8_parse_stringIO (UAT T3 ) 61.9726 msec/pass, best of ( 64.3755 61.9726 62.0165 ) cET: write_utf8_parse_stringIO (UAT T3 ) 609.6428 msec/pass, best of ( 609.6428 609.8895 615.0596 ) ET : write_utf8_parse_stringIO (UAT T3 ) 824.3621 msec/pass, best of ( 824.5392 826.6240 824.3621 ) lxe: write_utf8_parse_stringIO (S-T T4 ) 0.4812 msec/pass, best of ( 0.4814 0.4812 0.4846 ) cET: write_utf8_parse_stringIO (S-T T4 ) 5.3454 msec/pass, best of ( 5.4478 5.3454 5.3874 ) ET : write_utf8_parse_stringIO (S-T T4 ) 7.9776 msec/pass, best of ( 8.0112 7.9776 8.0064 ) lxe: write_utf8_parse_stringIO (SAT T4 ) 1.7338 msec/pass, best of ( 1.7576 1.7409 1.7338 ) cET: write_utf8_parse_stringIO (SAT T4 ) 17.3748 msec/pass, best of ( 17.3826 17.3748 17.3764 ) ET : write_utf8_parse_stringIO (SAT T4 ) 22.7056 msec/pass, best of ( 22.7056 22.7506 22.7812 ) lxe: write_utf8_parse_stringIO (U-T T4 ) 0.4791 msec/pass, best of ( 0.4839 0.4873 0.4791 ) cET: write_utf8_parse_stringIO (U-T T4 ) 5.3264 msec/pass, best of ( 5.3264 5.3404 5.3775 ) ET : write_utf8_parse_stringIO (U-T T4 ) 8.0147 msec/pass, best of ( 8.0221 8.0147 8.0596 ) lxe: write_utf8_parse_stringIO (UAT T4 ) 1.7334 msec/pass, best of ( 1.7661 1.7767 1.7334 ) cET: write_utf8_parse_stringIO (UAT T4 ) 17.3602 msec/pass, best of ( 17.3602 17.3674 17.3672 ) ET : write_utf8_parse_stringIO (UAT T4 ) 22.6152 msec/pass, best of ( 22.6621 22.6913 22.6152 )

3 17

[lxml-dev] Python values in xpath functions
by Petr van Blokland 16 Jul '06

16 Jul '06

Stefan or others, Using XPath Python functions, right now I instantiate a global object that functions can work on. That works fine, however for multi-threading applications in the future that won't work. What do you thinks is the best approach: locking the lxml process while running an XSLT transformation or searching/finding (or waiting for) a way to pass Python variables through to XPath functions? Kind regards, Petr ---------------------------------------------- Petr van Blokland buro(a)petr.com | www.petr.com | +31 15 219 10 40 ----------------------------------------------

2 6

[lxml-dev] let lxml write the ?xml pi
by Andreas Pakulat 11 Jul '06

11 Jul '06

Hi, I think this has been asked before, but I can't find it in my archive (or the official one).. Is it possible to tell lxml to add the <?xml version="1.0" encoding=".."?> when serializing an ElementTree? I couldn't find anything about this in the documentation. Andreas -- You have taken yourself too seriously.

4 9

[lxml-dev] extended XPath function
by Andreas Pakulat 30 Jun '06

30 Jun '06

Hi, before trying to code it I'd like to get opinions about the following. I'd like to add a global xpath function to lxml, that takes an element (the context) and an xpath expression (as string) and returns a list of tuples. Where each tuple has a) 3 parts, the attribute name, value and containing element b) 2 parts, the text and the element that has the text as text or tail c) 1 part, the element itself. I'd like to have this for XPathEvaluator (so I can highlight attribute and text nodes in the tree) and I'm willing to try and implement it in lxml. Andreas -- Your talents will be recognized and suitably rewarded.

4 13

[lxml-dev] installing lxml 1.0.1 on Mac PPC
by Martijn Faassen 30 Jun '06

30 Jun '06

Hi there, I had to install lxml on Mac PPC today (as part of a larger application). I thought, ah, no problem, I got eggs! So, since there isn't a 1.0.2 egg yet for Mac, I used the 1.0.1 one. Unfortunately that had a weird effect: instead of installing it decided it needed the lxml source tarball, downloaded that and tried to compile (failing somewhere on some pyrex error or whatnot, I didn't look into it deeply). What's going on here? I went back and install the lxml-1.0 egg instead of 1.0.1, and that one installed and works fine. The one difference I can identify between the 1.0.1 egg and the 1.0 egg is that the 1.0.1 egg is a fat egg. Does anyone (Olivier?) know what might've triggered this? Did anyone else have this happen to them? Regards, Martijn

4 6

[lxml-dev] lxml 1.0.2 and 1.1alpha released
by Stefan Behnel 27 Jun '06

27 Jun '06

Hi everyone, I just released lxml 1.0.2 mainly with bugfixes and 1.1alpha with loads of new features. So I now hope for no further bug reports on 1.0.2, loads of them for 1.1alpha and some beautiful little eggs for both of them, provided by our helpful egg maintainers. Have fun, Stefan Changes in 1.0.2: Features added * Support for setting a custom default Element class as opposed to namespace specific classes (which still override the default class) Bugs fixed * Rare exceptions in Python list functions were not handled * Parsing accepted unicode strings with XML encoding declaration in certain cases * Parsing 8-bit encoded strings from StringIO objects raised an exception * Module function initThread() was removed - useless (and never worked) * XSLT and parser exception messages include the error line number Features added in 1.1alpha: * Module level iterwalk() function as 'iterparse' for trees * ElementTree compatible iterparse() module function * Element.nsmap property returns a mapping of all namespace prefixes known at the Element to their namespace URI * Reentrant threading support in RelaxNG, XMLSchema and XSLT * Threading support in parsers and serializers: o All in-memory operations (tostring, parse(StringIO), etc.) free the GIL o File operations (on file names) free the GIL o Reading from file-like objects frees the GIL and reacquires it for reading o Serialisation to file-like objects is single-threaded (high lock overhead) * Element iteration over XPath axes: o Element.iterdescendants() iterates over the descendants of an element o Element.iterancestors() iterates over the ancestors of an element (from parent to parent) o Element.itersiblings() iterates over either the following or preceding siblings of an element o Element.iterchildren() iterates over the children of an element in either direction o All iterators support the tag keyword argument to restrict the generated elements * Element.getnext() and Element.getprevious() return the direct siblings of an element

1 1

[lxml-dev] build lxml against libxml2 from $HOME
by Andreas Pakulat 23 Jun '06

23 Jun '06

Hi I've got a small issue with lxml's gcc call when I use libxml2 installed to $HOME. The problem is that the "used" libxml2 is determined by the dynamic linker and defaults to /usr/lib/libxml2.so if I don't set LD_LIBRARY_PATH to point to $HOME/libxml2/lib. Now this is cumbersome and there's a way around: rpath. libxml2's python bindings use the following command for linkage of the .so: gcc -shared .libs/libxml.o .libs/types.o .libs/libxml2-py.o -Wl,--rpath -Wl,/home/andreas/compiling/python/xml/libxml2-2.6.26/.libs -Wl,--rpath -Wl,/home/andreas/libxml2/lib -L/home/andreas/compiling/python/xml/libxml2-2.6.26/.libs ../.libs/libxml2.so -L/usr/lib -Wl,-soname -Wl,libxml2mod.so -o .libs/libxml2mod.so the important path is the -Wl and --rpath stuff. The drawback with rpath is that it needs relinking upon installation, for libxml2's python binding this looks like: gcc -shared .libs/libxml.o .libs/types.o .libs/libxml2-py.o -Wl,--rpath -Wl,/home/andreas/libxml2/lib -L/home/andreas/compiling/python/xml/libxml2-2.6.26/.libs -L/home/andreas/libxml2/lib -lxml2 -L/usr/lib -Wl,-soname -Wl,libxml2mod.so -o .libs/libxml2mod.so So to conclude: I'd be _extremely_ happy if lxml could adapt that and link the etree.so using rpath. Is that possible? Thanks, Andreas -- You get along very well with everyone except animals and people.

2 8

[lxml-dev] iterparse / iterwalk merged, planned releases
by Stefan Behnel 23 Jun '06

23 Jun '06

Hi all, since iterparse() seems to work pretty well, I just added the iterwalk() equivalent for trees and merged the branch into the trunk. Both will therefore become official features of lxml 1.1. I'm planning to release an alpha version next week, so that people can start using the new functions. It will also include threading support and axis iteration. http://codespeak.net/svn/lxml/trunk/CHANGES.txt Additionally, there will be a 1.0.2 soon that fixes the remaining bugs in the 1.0 series. AFAIA (anyone for installing trac?), there was only one open bug report by Petr left that I couldn't reproduce, so unless that actually proves to be a reproducible problem, I won't be able to fix it. Stefan

1 0

[lxml-dev] Remarks on implementing iterparse()
by Stefan Behnel 22 Jun '06

22 Jun '06

Hi all, since I won't have the time to implement iterparse() any time soon, here's a proposal on how it should be implemented, in case someone wants to take a shot at it. "iterparse" will be (or will return) an iterable object, let's call it IterParse for clarity. A class is basically the only way of implementing iterators in Pyrex. For the internal SAX part, IterParse will likely work a lot like lxml.sax.ElementTreeContentHandler. We'd need a custom wrapper to the default libxml2 SAX handler to intercept the parse events (this means implementing C helper functions for the SAX events) /after/ they were processed by libxml2. See xmlSAXVersion (SAX2) on how to retrieve the SAX2 default parser structure. IterParse should pass chunks into the parser and buffer the events it receives. When its __next__() method is called, it returns one event or passes new chunks until there is an event to return. This is needed as IterParse has to convert between libxml2 push (SAX) and Python pull (iter). As for the input to the libxml2 parser, there are two possible ways: one is to pass data chunks in through xmlParseChunk and the other is to use xmlCreateIOParserCtxt and implement xmlInputReadCallback (xmlio.h) to have libxml2 request data by itself. Python events (start, end, start-ns, end-ns) are created as follows: * "*-ns" events must be extracted from the libxml2 xmlSAX2StartElementNs call (passed in arguments "prefix"/"URI" and the char* array "namespaces"). They must be stored on a stack to build the respective "end-ns" events. * "start" is somewhat tricky, as it would be a bad idea to allow modifications of the XML structure during that iterator cycle. Maybe it's enough to document that, but there may be ways to crash lxml with certain tree operations. Note also that care has to be taken to prevent Python from garbage collecting the element before the "end" event. The best way to do that is to store a Python reference to that element on a stack. * "end" is simple then: pop the element from the stack and return it. That's all I can come up with so far. So, if anyone is interested in taking a look at it, I'd be glad to hear about it. :) Stefan

3 4

[lxml-dev] iterparse - finally :)
by Stefan Behnel 22 Jun '06

22 Jun '06

Hi all, I finally decided to take a shot at iterparse() and it turned out to be a matter of hours to implement it. It's very similar to ET's implementation and builds on top of the SAX2 interface of the libxml2 parser (as I proposed in my last post on this topic). It is currently implemented on the iterparse branch: https://codespeak.net/svn/lxml/branch/iterparse/ https://codespeak.net/viewvc/lxml/branch/iterparse/ It passes ET's selftest.py and a few other test cases (including clear()ing up in the end). Anyone out there who has iterparse() code available is invited to run it against the new implementation. If it works better than I currently expect :), it will become part of the lxml 1.1 feature set. Stefan

2 2