Mailman 3 May 2006 - lxml - The Python XML Toolkit

[lxml-dev] The difference between str(xslt_result) and xslt_result.write()
by Matt Patterson 10 May '06

10 May '06

Hi, Is there a difference between the output from lxml.etree._XSLTResultTree.__str__() and lxml.etree._XSLTResultTree.write()? I'm trying to chase some output whitespace issues around, and I'm wondering if there's a difference in how serialisation is handled. Also, str(_XSLTResultTree) is failing for me with errors like "'ascii' codec can't encode character u'\xa9' in position 1608: ordinal not in range(128)" because I have unicode characters in my documents... lxml.etree._XSLTResultTree doesn't define a __unicode__ method so I can't use a unicode coercion to UTF-8 or something like that... All in all though, I'm really enjoying lxml: I spent a long time working with libxml & lbxslt's standard python interfaces and they much more of a pain than lxml! Thanks, Matt -- Matt Patterson | Design & Code <matt at reprocessed org> | http://www.reprocessed.org/

2 2

[lxml-dev] I/O benchmarks
by Stefan Behnel 08 May '06

08 May '06

Hi everyone, have you ever wondered why you should use lxml instead of cElementTree? Here is why. :) Stefan Preparing test suites and trees ... Running benchmark on lxe, ET, cET Setup times for trees in seconds: lxe: -- S- U- -A SA UA T1: 0.1180 0.1158 0.1153 0.1178 0.1157 0.1173 T2: 0.1186 0.1202 0.1207 0.1233 0.1236 0.1250 T3: 0.0323 0.0252 0.0250 0.0489 0.0495 0.0492 T4: 0.0005 0.0005 0.0005 0.0010 0.0010 0.0010 ET : -- S- U- -A SA UA T1: 0.2305 0.2887 0.2193 0.2662 0.2872 0.2259 T2: 0.3038 0.3442 0.2823 0.3140 0.2364 0.4051 T3: 0.0534 0.0572 0.0523 0.0583 0.0553 0.0829 T4: 0.0010 0.0008 0.0007 0.0009 0.0008 0.0008 cET: -- S- U- -A SA UA T1: 0.0369 0.0353 0.0371 0.0341 0.0354 0.0345 T2: 0.0370 0.0364 0.0361 0.0371 0.0369 0.0358 T3: 0.0090 0.0091 0.0090 0.0125 0.0174 0.0235 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 lxe: tostring_utf16 (S- T1 ) 25.0344 24.7066 24.6755 msec/pass, best: 24.6755 ET : tostring_utf16 (S- T1 ) 715.0391 668.6494 668.2270 msec/pass, best: 668.2270 cET: tostring_utf16 (S- T1 ) 631.4080 634.0269 629.3236 msec/pass, best: 629.3236 lxe: tostring_utf16 (U- T1 ) 25.3342 25.4860 25.1003 msec/pass, best: 25.1003 ET : tostring_utf16 (U- T1 ) 666.5964 670.9383 664.8268 msec/pass, best: 664.8268 cET: tostring_utf16 (U- T1 ) 628.6623 628.9270 636.1863 msec/pass, best: 628.6623 lxe: tostring_utf16 (S- T2 ) 38.5328 27.5526 28.3839 msec/pass, best: 27.5526 ET : tostring_utf16 (S- T2 ) 696.8423 697.2251 698.0101 msec/pass, best: 696.8423 cET: tostring_utf16 (S- T2 ) 655.5692 652.2847 653.9454 msec/pass, best: 652.2847 lxe: tostring_utf16 (U- T2 ) 26.2363 26.8365 26.6303 msec/pass, best: 26.2363 ET : tostring_utf16 (U- T2 ) 698.3589 697.7249 698.7069 msec/pass, best: 697.7249 cET: tostring_utf16 (U- T2 ) 652.6794 652.6194 652.4247 msec/pass, best: 652.4247 lxe: tostring_utf16 (S- T3 ) 2.9159 2.9798 3.0483 msec/pass, best: 2.9159 ET : tostring_utf16 (S- T3 ) 90.1106 90.2723 90.6772 msec/pass, best: 90.1106 cET: tostring_utf16 (S- T3 ) 75.9803 75.7933 75.6844 msec/pass, best: 75.6844 lxe: tostring_utf16 (U- T3 ) 2.9326 2.9501 3.8947 msec/pass, best: 2.9326 ET : tostring_utf16 (U- T3 ) 90.8508 90.2919 90.4195 msec/pass, best: 90.2919 cET: tostring_utf16 (U- T3 ) 75.8780 75.6118 75.6281 msec/pass, best: 75.6118 lxe: tostring_utf16 (S- T4 ) 0.1184 0.1246 0.1174 msec/pass, best: 0.1174 ET : tostring_utf16 (S- T4 ) 4.0772 4.1340 4.0832 msec/pass, best: 4.0772 cET: tostring_utf16 (S- T4 ) 8.6555 3.7591 3.8131 msec/pass, best: 3.7591 lxe: tostring_utf16 (U- T4 ) 0.1150 0.1142 0.1470 msec/pass, best: 0.1142 ET : tostring_utf16 (U- T4 ) 4.1377 4.1255 4.0694 msec/pass, best: 4.0694 cET: tostring_utf16 (U- T4 ) 8.5306 3.7888 3.8399 msec/pass, best: 3.7888 lxe: tostring_utf8 (S- T1 ) 20.1342 20.8913 22.0570 msec/pass, best: 20.1342 ET : tostring_utf8 (S- T1 ) 658.5292 659.8311 659.2222 msec/pass, best: 658.5292 cET: tostring_utf8 (S- T1 ) 613.7725 616.7276 615.3487 msec/pass, best: 613.7725 lxe: tostring_utf8 (U- T1 ) 22.3571 21.6062 22.3468 msec/pass, best: 21.6062 ET : tostring_utf8 (U- T1 ) 658.4980 658.8312 659.7990 msec/pass, best: 658.4980 cET: tostring_utf8 (U- T1 ) 619.8824 620.7493 618.3270 msec/pass, best: 618.3270 lxe: tostring_utf8 (S- T2 ) 22.8893 23.1579 22.3014 msec/pass, best: 22.3014 ET : tostring_utf8 (S- T2 ) 695.0778 686.7107 686.9514 msec/pass, best: 686.7107 cET: tostring_utf8 (S- T2 ) 644.8464 645.2512 645.4735 msec/pass, best: 644.8464 lxe: tostring_utf8 (U- T2 ) 21.8574 21.5462 21.9939 msec/pass, best: 21.5462 ET : tostring_utf8 (U- T2 ) 689.5708 689.1687 685.5371 msec/pass, best: 685.5371 cET: tostring_utf8 (U- T2 ) 643.3239 644.8895 641.5675 msec/pass, best: 641.5675 lxe: tostring_utf8 (S- T3 ) 2.3862 2.2847 2.3411 msec/pass, best: 2.2847 ET : tostring_utf8 (S- T3 ) 91.7382 89.4661 90.1840 msec/pass, best: 89.4661 cET: tostring_utf8 (S- T3 ) 76.4675 74.7183 74.7682 msec/pass, best: 74.7183 lxe: tostring_utf8 (U- T3 ) 2.3286 2.4161 2.3287 msec/pass, best: 2.3286 ET : tostring_utf8 (U- T3 ) 89.9778 92.0483 91.0223 msec/pass, best: 89.9778 cET: tostring_utf8 (U- T3 ) 74.8817 74.8538 74.9363 msec/pass, best: 74.8538 lxe: tostring_utf8 (S- T4 ) 0.1028 0.1006 0.1058 msec/pass, best: 0.1006 ET : tostring_utf8 (S- T4 ) 8.9541 4.1052 4.0422 msec/pass, best: 4.0422 cET: tostring_utf8 (S- T4 ) 8.5588 3.7166 3.7450 msec/pass, best: 3.7166 lxe: tostring_utf8 (U- T4 ) 0.1178 0.1000 0.1040 msec/pass, best: 0.1000 ET : tostring_utf8 (U- T4 ) 4.0836 4.0845 4.0257 msec/pass, best: 4.0257 cET: tostring_utf8 (U- T4 ) 8.5069 3.7946 3.7505 msec/pass, best: 3.7505 lxe: tostring_utf8_unicode_XML (S- T1 ) 217.3858 194.6370 196.9607 msec/pass, best: 194.6370 ET : tostring_utf8_unicode_XML (S- T1 ) 1044.2196 983.0496 1006.1930 msec/pass, best: 983.0496 cET: tostring_utf8_unicode_XML (S- T1 ) 684.9565 679.0308 674.1449 msec/pass, best: 674.1449 lxe: tostring_utf8_unicode_XML (U- T1 ) 203.8817 203.9280 200.4173 msec/pass, best: 200.4173 ET : tostring_utf8_unicode_XML (U- T1 ) 981.1272 977.5449 978.9895 msec/pass, best: 977.5449 cET: tostring_utf8_unicode_XML (U- T1 ) 653.3920 655.3704 651.0385 msec/pass, best: 651.0385 lxe: tostring_utf8_unicode_XML (S- T2 ) 208.8770 211.5672 210.4380 msec/pass, best: 208.8770 ET : tostring_utf8_unicode_XML (S- T2 ) 1021.3773 1020.3279 1041.1114 msec/pass, best: 1020.3279 cET: tostring_utf8_unicode_XML (S- T2 ) 688.2031 681.9181 680.2590 msec/pass, best: 680.2590 lxe: tostring_utf8_unicode_XML (U- T2 ) 209.5904 210.3374 209.5674 msec/pass, best: 209.5674 ET : tostring_utf8_unicode_XML (U- T2 ) 1027.9991 1022.0539 1022.0318 msec/pass, best: 1022.0318 cET: tostring_utf8_unicode_XML (U- T2 ) 681.4484 678.8596 680.9469 msec/pass, best: 678.8596 lxe: tostring_utf8_unicode_XML (S- T3 ) 12.5315 12.0217 11.9403 msec/pass, best: 11.9403 ET : tostring_utf8_unicode_XML (S- T3 ) 202.9581 203.0447 204.2288 msec/pass, best: 202.9581 cET: tostring_utf8_unicode_XML (S- T3 ) 82.3561 82.4286 82.1372 msec/pass, best: 82.1372 lxe: tostring_utf8_unicode_XML (U- T3 ) 11.6045 11.8548 11.7764 msec/pass, best: 11.6045 ET : tostring_utf8_unicode_XML (U- T3 ) 201.2770 201.0283 202.3813 msec/pass, best: 201.0283 cET: tostring_utf8_unicode_XML (U- T3 ) 82.3942 82.6146 82.3579 msec/pass, best: 82.3579 lxe: tostring_utf8_unicode_XML (S- T4 ) 5.3620 0.4832 0.5012 msec/pass, best: 0.4832 ET : tostring_utf8_unicode_XML (S- T4 ) 6.4547 6.3800 6.3898 msec/pass, best: 6.3800 cET: tostring_utf8_unicode_XML (S- T4 ) 8.8447 3.9611 4.0390 msec/pass, best: 3.9611 lxe: tostring_utf8_unicode_XML (U- T4 ) 5.2781 0.4689 0.4798 msec/pass, best: 0.4689 ET : tostring_utf8_unicode_XML (U- T4 ) 6.5955 6.4135 6.4196 msec/pass, best: 6.4135 cET: tostring_utf8_unicode_XML (U- T4 ) 8.7307 4.0647 4.0202 msec/pass, best: 4.0202 lxe: write_utf8_parse_stringIO (S- T1 ) 198.0141 188.3298 190.1872 msec/pass, best: 188.3298 ET : write_utf8_parse_stringIO (S- T1 ) 1143.8117 1176.7961 1152.1253 msec/pass, best: 1143.8117 cET: write_utf8_parse_stringIO (S- T1 ) 814.6583 847.3877 810.7611 msec/pass, best: 810.7611 lxe: write_utf8_parse_stringIO (U- T1 ) 194.5397 195.0845 194.7964 msec/pass, best: 194.5397 ET : write_utf8_parse_stringIO (U- T1 ) 1153.3167 1142.9091 1146.2344 msec/pass, best: 1142.9091 cET: write_utf8_parse_stringIO (U- T1 ) 810.8662 808.3027 811.9326 msec/pass, best: 808.3027 lxe: write_utf8_parse_stringIO (S- T2 ) 205.9781 202.8390 202.2034 msec/pass, best: 202.2034 ET : write_utf8_parse_stringIO (S- T2 ) 1195.9166 1195.7337 1193.5966 msec/pass, best: 1193.5966 cET: write_utf8_parse_stringIO (S- T2 ) 846.5763 851.8866 848.6367 msec/pass, best: 846.5763 lxe: write_utf8_parse_stringIO (U- T2 ) 203.5371 202.8143 204.4644 msec/pass, best: 202.8143 ET : write_utf8_parse_stringIO (U- T2 ) 1218.4967 1245.6663 1255.3381 msec/pass, best: 1218.4967 cET: write_utf8_parse_stringIO (U- T2 ) 858.2268 849.1432 858.5724 msec/pass, best: 849.1432 lxe: write_utf8_parse_stringIO (S- T3 ) 17.0564 11.5748 12.5815 msec/pass, best: 11.5748 ET : write_utf8_parse_stringIO (S- T3 ) 235.0894 234.2171 233.4405 msec/pass, best: 233.4405 cET: write_utf8_parse_stringIO (S- T3 ) 113.0575 112.7246 112.9978 msec/pass, best: 112.7246 lxe: write_utf8_parse_stringIO (U- T3 ) 11.1004 11.0245 11.3229 msec/pass, best: 11.0245 ET : write_utf8_parse_stringIO (U- T3 ) 238.6545 243.0529 251.3970 msec/pass, best: 238.6545 cET: write_utf8_parse_stringIO (U- T3 ) 122.8126 115.6941 120.9536 msec/pass, best: 115.6941 lxe: write_utf8_parse_stringIO (S- T4 ) 0.4456 0.4642 0.4494 msec/pass, best: 0.4456 ET : write_utf8_parse_stringIO (S- T4 ) 8.0726 8.1274 7.9427 msec/pass, best: 7.9427 cET: write_utf8_parse_stringIO (S- T4 ) 6.0532 5.0335 5.0780 msec/pass, best: 5.0335 lxe: write_utf8_parse_stringIO (U- T4 ) 0.4778 0.4506 0.4657 msec/pass, best: 0.4506 ET : write_utf8_parse_stringIO (U- T4 ) 7.4578 7.3894 8.0762 msec/pass, best: 7.3894 cET: write_utf8_parse_stringIO (U- T4 ) 6.2156 5.5292 5.0383 msec/pass, best: 5.0383

1 0

Re: [lxml-dev] HTMLParser behavior - ill formed HTML
by Stefan Behnel 04 May '06

04 May '06

Hi Dean, Dean Pavlekovic wrote: > On 5/2/06, Stefan Behnel <behnel_ml(a)gkec.informatik.tu-darmstadt.de> wrote: > >> I changed the trunk to always accept ill-formed results if the recover >> option is set and no lxml-internal errors were raised. Please try if that >> helps. I couldn't come up with a sufficiently short example of broken >> HTML where this problem occurs, so I couldn't test it. The examples that >> were tested so far can be found in src/lxml/tests/test_htmlparser.py. > > Unfortunately it's not working and I'm sorry but I didn't have the time to > look at it deeper. It appears that the value of options is changed > somewhere in libxml context during the htmlCtxtReadFile. I printed out the > value of parser's self._parse_options before (97, that is > XML_PARSE_RECOVER_FLAG set) and the value of ctxt.options after the > htmlCtxtReadFile call which was 96 meaning the flag was reset by libxml. Interesting. Could you tell me what version of libxml2 you are using (see below)? My guess is that it's older than 2.6.21. Libxml2 copies the options by hand, so if the RECOVER option is unknown, it will not turn up in the context options. That makes me wonder why recovery worked for Paul on 2.6.16 in the first place... Anyway. I changed the trunk to pass the option explicitly to _handleParseResult so that we no longer rely on libxml2. Please test if this works for you now. I'd be glad if you could come up with a short piece of broken HTML code that triggers the "not well formed" case in recovery mode. That would allow us to set up a test case to check that it actually works (and keeps working). To check which versions you are using, I added module attributes "LXML_VERSION", "LIBXML_VERSION" and "LIBXSLT_VERSION" that carry tuples containing the respective versions used by lxml, e.g. (2, 6, 23) for "2.6.23". "LIBXML_COMPILED_VERSION" and "LIBXSLT_COMPILED_VERSION" show against which versions lxml was compiled. BTW, these attributes are mainly meant for debugging purposes. Since they will first (officially) appear in lxml 1.0, code that wants to use them anyway will have to check if they are available (hasattr or try-except) before accessing them. All of them appeared at the same time, so if one is there, the others will be there, too. > Btw. is there a reason that the XML_... enums are assigned an explicit > value in xmlparser.pxd and the same is not done for HTML_... enums in > htmlparser.pxd? I'm not familiar with Pyrex and wheter it parses c/c++ .h > files to match values... Pyrex doesn't care about the values of enums and C uses the .h files directly, so, since they are just copy&pasted into the .pxd files, they sometimes have numbers and sometimes not. Stefan

2 2

[lxml-dev] 'docinfo' property on ElementTree
by Stefan Behnel 02 May '06

02 May '06

Hi all, I updated the trunk to provide ElementTree objects with access to the document information provided by the parser: DOCTYPE, XML version and original encoding. Paul Everitt had some use cases related to the HTML parser, but I think it's generally a good idea to make this kind of information available. The new API works as follows: >>> pub_id = "-//W3C//DTD XHTML 1.0 Transitional//EN" >>> sys_url = "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >>> doctype_string = '<!DOCTYPE html PUBLIC "%s" "%s">' % (pub_id, sys_url) >>> xml_header = '<?xml version="1.0" encoding="ascii"?>' >>> xhtml = xml_header + doctype_string + '<html><body></body></html>' >>> et = lxml.etree.parse(StringIO(xhtml)) >>> docinfo = et.docinfo >>> print docinfo.public_id -//W3C//DTD XHTML 1.0 Transitional//EN >>> print docinfo.system_url http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd >>> docinfo.doctype == doctype_string True >>> print docinfo.xml_version 1.0 >>> print docinfo.encoding ascii This is backed by a DocInfo object that you can also instantiate on an ElementTree (or Element) by hand. The docinfo property just does it for you. Any of the attributes above may be None if the information is not available. Stefan

1 0

[lxml-dev] HTMLParser behavior - ill formed HTML
by Dean Pavlekovic 02 May '06

02 May '06

Hi all, My name is Dean and I've recently joined the lxml-dev to watch how things are going most notably since the HTML parser has been added to the trunk. We (my colleague and I) have been using lxml in the past for processing of rather large DocBook type XML docs aplit across multiple files and in mixed namespaces without any problems and the experiences with lxml have been nothing but good, so let me first congratulate the developers on a great work lxml is. Lately I've looked into using the lxml trunk revision for analyzing some HTML (not well-formed). I've stumbled upon the feature for which I'm not sure whether it's intended to be. When I feed an ill-formed HTML to a parser via > doc=etree.parse('afile.html', parser=etree.HTMLParser()) #recover=True by default an exception is raised and the function yields no result. When I use libxml2 directly: > doc=libxml2.htmlParseFile('afile.html', None) libxml2 prints out some errors/warnings but I _do_ get a reference to a document, which can normally be used. I've also tried to use the procedure as in parser.pxi i.e. htmlCreateMemoryParserCtxt() and htmlCtxtReadDoc() with the same results. So I tracked this behavior down to calling _handleParseResult(pctxt, result, NULL) at the end of parse* methods in HTMLParser in parser.pxi, note the 'if.ctxt.wellFormed' part:  where the document is destroyed if libxml2's context wellFormed flag is not set. I checked this by calling libxml2 htmlCtxtReadDoc() directly on that document and indeed the wellFormed flag turned out to be 0. Now, shouldn't the HTMLParser also return the document reference in this case if recover=True flag is specified since libxml apparently does not have problems with that. I've checked this by modifying the _handleParseResult with 'accept_ill_formed' argument. If that flag is set, ctxt.wellFormed would be ignored. Also modified _handleParseResult calls in HTMLParser's parse... methods to specify accept flag if 'recover' was set in the constructor. This turned to work just well and I was able to navigate the document with xpath, even the errors (using '&' in href attributes) were corrected. So the question would be is the present behavior correct due to something I'm possibly missing? I think it should be dependent on whether RECOVER flags have been specified or not. Best regards, Dean

2 1