Mailman 3 June 2013 - lxml - The Python XML Toolkit

Possible bug in lxml.objectify
by Amaury Forgeot d'Arc June 17, 2013

June 17, 2013

Hi, By doing the cffi port, I noticed this suspect line in lxml.objectify.pyx (line 1689): elif c_ns.prefix is not NULL or c_ns.prefix[0] != c'\0': It seems to me that the connector should be "and" instead, otherwise a prefix of "" would yield a typename starting with a colon character... I haven't tried though. But it looks like a bad negation of a similar statement above. -- Amaury Forgeot d'Arc

2 1

New version for PyPy, using cffi
by Amaury Forgeot d'Arc June 17, 2013

June 17, 2013

Hello, I almost finished a port of lxml that uses cffi bindings instead of Cython. The goal is to run it with PyPy, which support for the CPython API is known to be very slow and somewhat incomplete. cffi on the other hand is much more friendly to PyPy's JIT. The code is there: https://github.com/amauryfa/lxml in the "cffi" branch. Note that it works directly from the source tree, no need to compile. You have to use a recent PyPy, ensure that the "lxml" directory can be seen from your PYTHONPATH, and that you have libxml development packages. The conversion -- from .pyx files to pure Python -- turned out to be quite straightforward. And the result is interesting IMO: - Impact on the existing code is minimal (in __init__.py), of course there is a lot of duplication. - Only 5 tests fail in the whole suite (if you except objectify, which is hard to support correctly on a non-refcounting gc) - Performance is on par with the version installed on my laptop, except from some catastrophic benchmarks, probably because of a missed optimization. I did not try to optimize anything, correctness first! My plan is to issue a pull request soon. Some work remains, though, and I need your help here: - adapt setup.py to the new system - fix the remaining tests - Is there a way to reduce the code duplication between .pyx and .py files? So please, try it with PyPy, try it with CPython (will be slower), comment on the code and report issues... Long live lxml! -- Amaury Forgeot d'Arc

3 5

etree.XML() nsmap gives partial namespace
by Sangeeth Saravanaraj June 3, 2013

June 3, 2013

I am using nsmap on etree.XML() to get the list of all the namespaces used in a given xml document. The following code i.e. <code-1> works fine which gives the expected output i.e. <output-1>. The nsmap contains all the namespace that are used in the document. One point to note here is - all the xml namespaces used in this document has been highlighted in the root element itself. <code-1> from lxml import etree xml_str_1 = '''<?xml version='1.0' encoding='UTF-8'?> <ns0:root xmlns:ns0="http://thisisroot.com/rootelem/" xmlns:ns1=" http://thisischild.com/childelem1/" xmlns:ns2=" http://thisischild.com/childelem2/" xmlns:ns3=" http://thisischild.com/childelem3/" xmlns:ns4=" http://thisischild.com/childelem4/"> <ns1:child1>child1</ns1:child1> <ns2:child2>child2</ns2:child2> <ns3:child3>child3</ns3:child3> <ns4:child4>child4</ns4:child4> </ns0:root>''' xml_1 = etree.XML(xml_str_1) ns_1 = xml_1.nsmap print "xml document = ", xml_str_1 print "namespace = ", ns_1 </code-1> <output-1> xml document = <?xml version='1.0' encoding='UTF-8'?> <ns0:root xmlns:ns0="http://thisisroot.com/rootelem/" xmlns:ns1=" http://thisischild.com/childelem1/" xmlns:ns2=" http://thisischild.com/childelem2/" xmlns:ns3=" http://thisischild.com/childelem3/" xmlns:ns4=" http://thisischild.com/childelem4/"> <ns1:child1>child1</ns1:child1> <ns2:child2>child2</ns2:child2> <ns3:child3>child3</ns3:child3> <ns4:child4>child4</ns4:child4> </ns0:root> namespace = {'ns0': 'http://thisisroot.com/rootelem/', 'ns1': ' http://thisischild.com/childelem1/', 'ns2': ' http://thisischild.com/childelem2/', 'ns3': ' http://thisischild.com/childelem3/', 'ns4': ' http://thisischild.com/childelem4/'} </output-1> In the following (<code-2>, I have constructed an xml document using etree.Element() and etree.SubElement() using the same namespace. I used etree.tostring() to get the xml document. One thing to note here is - the xml document that has been generated by etree.tostring() does not have all the namespaces in the root element. A namespace is introduced in an element as and when it is needed. When I used the xml document generated by etree.tostring() in the etree.XML(), the nsmap does not contain all the namespaces used in the xml document instead it contains only the namespace which is highlighted in the root element. <code-2> from lxml import etree root = '{http://thisisroot.com/rootelem/}root' c1 = '{http://thisischild.com/childelem1/}child1' c2 = '{http://thisischild.com/childelem2/}child2' c3 = '{http://thisischild.com/childelem3/}child3' c4 = '{http://thisischild.com/childelem4/}child4' e_root = etree.Element(root) etree.SubElement(e_root, c1).text = 'child1' etree.SubElement(e_root, c2).text = 'child2' etree.SubElement(e_root, c3).text = 'child3' etree.SubElement(e_root, c4).text = 'child4' xml_str_2 = etree.tostring(e_root, pretty_print=True, xml_declaration=True, encoding='UTF-8') xml_2 = etree.XML(xml_str_2) ns_2 = xml_2.nsmap print "xml document = ", xml_str_2 print "namespace = ", ns_2 </code-2> </output-2> xml document = <?xml version='1.0' encoding='UTF-8'?> <ns0:root xmlns:ns0="http://thisisroot.com/rootelem/"> <ns1:child1 xmlns:ns1="http://thisischild.com/childelem1/ ">child1</ns1:child1> <ns2:child2 xmlns:ns2="http://thisischild.com/childelem2/ ">child2</ns2:child2> <ns3:child3 xmlns:ns3="http://thisischild.com/childelem3/ ">child3</ns3:child3> <ns4:child4 xmlns:ns4="http://thisischild.com/childelem4/ ">child4</ns4:child4> </ns0:root> namespace = {'ns0': 'http://thisisroot.com/rootelem/'} </output-2> In the case of <code-2>, is there a way to get all the namespaces used in a given xml document?! Please suggest.. Thank you, Sangeeth

2 1

attributes() in DTD - possible bug()
by Nikolay Amiantov June 3, 2013

June 3, 2013

Hello, I'm trying to iterate over all attributes in DTD. Here is a log from REPL: http://pastebin.com/qfSqYjCB attributes() is empty when using DTD from internalDTD, but they are definitely used (default values for attributes are properly set). Also, when using etree.DTD attributes are properly listed. Is this a bug or there is something that I have missed? P.S.: externalDTD is empty, this seems right to me. Nikolay Amiantov.

1 0

attributes() in DTD - possible bug?
by Nikolay Amiantov June 3, 2013

June 3, 2013

Hello, I'm trying to iterate over all attributes in DTD. Here is a log from REPL: http://pastebin.com/qfSqYjCB attributes() is empty when using DTD from internalDTD, but they are definitely used (default values for attributes are properly set). Also, when using etree.DTD attributes are properly listed. Is this a bug or there is something that I have missed? P.S.: externalDTD is empty, and this seems right to me.

1 0

Changing encoding while parsing an HTML string
by Michael O'Leary June 2, 2013

June 2, 2013

If I understand how the XML parser works, if it parses an XML string that begins with a header line like <?xml version="1.0" encoding="utf-8"?> it will parse the rest of the XML string using the encoding of the encoding it has found. So first of all, does the XML parser do this? If so, I'd like to know if something like this can be done, either automatically or manually, with the HTML parser. I have been parsing HTML strings that have been obtained from a random sampling of web sites, and I am wondering if there is a way to do something similar with these HTML strings. Typically, these HTML strings do not start with a header like, but most of them do have a meta tag of the form <meta charset="utf-8"> or <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" > I wouldn't expect that the HTML parser reads these tags and automatically sets the encoding to be used while parsing the rest of the HTML string (but I might as well ask: does it?). I could include code in my target's start tag handler that checks meta tags to see if they contain encoding information and calls the parser to set its encoding, but as far as I can see there isn't a way to set the encoding for an HTML parser after it has been created. That is, it looks like you can do this parser = etree.HTMLParser(target=my_target, encoding=my_encoding) but not this parser = etree.HTMLParser(target=my_target) .... figure out what value my_encoding should have .... parser.set_encoding(my_encoding) So I guess my underlying question is, how can I parse HTML strings and give the parser the best chance to use the right encoding for the part of the HTML string that is in the <body> .... </body> section of it? Thanks, Mike

2 1