Mailman 3 May 2013 - lxml - The Python XML Toolkit

lxml object architecture quandary
by John W. Shipman May 21, 2013

May 21, 2013

I have a design problem involving ElementTree. I'd appreciate any suggestions. The problem is posted online here: http://johnwshipman.blogspot.com/ Best regards, John Shipman (john(a)nmt.edu), Applications Specialist New Mexico Tech Computer Center, Speare 146, Socorro, NM 87801 (575) 835-5735, http://www.nmt.edu/~john ``Let's go outside and commiserate with nature.'' --Dave Farber

3 3

Question on 1) encoding/decoding and 2) importing xsd
by Sangeeth Saravanaraj May 16, 2013

May 16, 2013

I have the following snippet of code which validates the given xml data against a given xsd. <code> from lxml import etree from io import StringIO, BytesIO with open("schema1.xsd", "r") as schema_file: schemaContent = schema_file.read() with open("data.xml", "r") as xml_file: xmlContent = xml_file.read() xmlschema_doc = etree.parse(StringIO(schemaContent.decode('iso8859_1'))) xml_doc = etree.parse(StringIO(xmlContent.decode('iso8859_1'))) xmlschema = etree.XMLSchema(xmlschema_doc) print xmlschema.validate(xml_doc) </code> The above code usually works fine but not under the following scenarios. *#1) ValueError: Unicode strings with encoding declaration are not supported.* I see ValueError when the .xsd or .xml file begins with an 'xml' element which contains the 'encoding' attribute as follows <?xml version="1.0" encoding="UTF-8"?> When I remove the 'encoding' attribute, the problem vanishes. For example: <?xml version="1.0"?> I referred the Python Standard Encodings - http://docs.python.org/2/library/codecs.html#standard-encodings and tried the following but couldn't get the xml/xsd data decoded when 'xml' element contains 'encoding' attribute. schemaContent.decode(encoding='UTF-8', errors='ignore') schemaContent.decode(encoding='ISO-8859-1', errors='ignore') What am I missing here?! *#2 lxml.etree.XMLSchemaParseError: failed to load external entity "schema2.xsd"* Say I have the following directory structure: proj_home |-- validate.py `-- schema |-- schema1.xsd `-- schema2.xsd Say proj_home/schema/schema1.xsd imports schema2.xsd (which is present proj_home/schema/) as follows: <xs:import namespace="namespace url goes here" schemaLocation="schema2.xsd"/> Now, when the validate.py reads proj_home/schema/schema1.xsd and uses that in etree.XMLSchema(), it raises lxml.etree.XMLSchemaParseError as schema2.xsd is not present in proj_home but present in proj_home/schema directory. If I copy schema2.xsd to proj_home (in the same directory where validate.py exists) then it works fine! How should I make etree.XMLSchema() to load the "import" files from a particular directory (in this case proj_home/schema) instead of looking at the current working directory? Thank you, Sangeeth

3 2

Parsing problem
by Arnaud RIVET May 15, 2013

May 15, 2013

Hi, I am new with LXML and I have a problem after parsing my element : its architecture seems to have changed if I remove (or replace) the last child. Here is my code and a screen of the console output. I have looked up for solution but I still can't figure what I am doing wrong. I would really appreciate someone's help ! (I am using LXML 3.2.1 with Python 2.6 on Windows) ############################################################################################# from lxml import etree from copy import deepcopy def Write( file, element ): f = open( file, 'w' ) f.write( etree.tostring( element, xml_declaration=True, encoding="ISO-8859-1", pretty_print = True ) ) f.close() return 1 def ReadAndReturn( file ): lookup = etree.ElementDefaultClassLookup() parser = etree.XMLParser(recover = True) parser.set_element_class_lookup( lookup ) mainTree = etree.parse( file, parser ) return mainTree # create a root element with 3 children root = etree.Element( "root" ) root.append( etree.Element( "child1" ) ) child2 = etree.SubElement( root, "child2" ) child2.text = 'CHILD2' child3 = etree.SubElement( root, "child3" ) child3.text = 'CHILD3' print "\n--- INITIAL ROOT ---" print( etree.tostring( root, pretty_print=True ) ) # remove last child root2 = deepcopy( root ) root2.remove( root2[2] ) print "--- ROOT WITHOUT LAST CHILD / BEFORE WRITING ---" print( etree.tostring( root2, pretty_print=True ) ) # write initial root (3 children) and read the file filename = 'test.tst' status = Write( filename, root ) tree = ReadAndReturn( filename ) # remove last child from the read element root3 = deepcopy( tree.getroot() ) root3.remove( root3[2] ) print "--- ROOT WITHOUT LAST CHILD / AFTER WRITING AND PARSING ---" print( etree.tostring( root3, pretty_print=True ) ) #############################################################################################

3 2

lxml 3.2.1 released
by Stefan Behnel May 11, 2013

May 11, 2013

Hi everyone, I just sent out lxml 3.2.1 as the first major bug-fix release for the stable 3.2 release series. You can get it from PyPI: http://pypi.python.org/pypi/lxml/3.2.1 The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.2.1.tgz Signature: http://lxml.de/files/lxml-3.2.1.tgz.asc Changelog: http://lxml.de/3.2/changes-3.2.1.html GitHub: https://github.com/lxml/lxml/commit/90afe19b11c4fd1df8998b254d8a60b550aa56d0 This release was built using Cython 0.19.1. Note that the build no longer uses Cython, even if it is installed. Recompilation of the sources has to be requested explicitly with the setup.py option "--with-cython". If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.2.1 (2013-05-11) ================== Features added -------------- * The methods ``apply_templates()`` and ``process_children()`` of XSLT extension elements have gained two new boolean options ``elements_only`` and ``remove_blank_text`` that discard either all strings or whitespace-only strings from the result list. Bugs fixed ---------- * When moving Elements to another tree, the namespace cleanup mechanism no longer drops namespace prefixes from attributes for which it finds a default namespace declaration, to prevent them from appearing as unnamespaced attributes after serialisation. * Returning non-type objects from a custom class lookup method could lead to a crash. * Instantiating and using subtypes of Comments and ProcessingInstructions crashed.

1 0

Patch for losing namespace on attributes
by Tyler Spivey May 11, 2013

May 11, 2013

I tried patching this. I'm not sure this is the right way of doing things, but it passes the existing tests as well as the one added for it. diff -r 452a2705665b -r 41fee8a923a1 src/lxml/proxy.pxi --- a/src/lxml/proxy.pxi Sun Apr 28 20:42:33 2013 +0200 +++ b/src/lxml/proxy.pxi Fri May 03 17:07:35 2013 -0700 @@ -287,7 +287,7 @@ while c_nsdef[0] is not NULL: c_ns = tree.xmlSearchNsByHref( c_element.doc, c_element.parent, c_nsdef[0].href) - if c_ns is NULL: + if c_ns is NULL or c_ns.prefix is NULL: # new namespace href => keep and cache the ns declaration _appendToNsCache(c_ns_cache, c_nsdef[0], c_nsdef[0]) c_nsdef = &c_nsdef[0].next @@ -340,6 +340,7 @@ cdef xmlNs* c_ns_next cdef xmlNs* c_nsdef cdef xmlNs* c_del_ns_list + cdef xmlNs* c_default_ns = NULL cdef size_t i, proxy_count = 0 if not tree._isElementOrXInclude(c_element): @@ -366,12 +367,18 @@ # 2) make sure the namespaces of an element and its attributes # are declared in this document (i.e. on the node or its parents) + #Find the default namespace for this element in either the node or its parents + c_default_ns = tree.xmlSearchNs(c_source_doc, c_element, NULL) + c_node = c_element while c_node is not NULL: if c_node.ns is not NULL: for i from 0 <= i < c_ns_cache.last: if c_node.ns is c_ns_cache.old[i]: - c_node.ns = c_ns_cache.new[i] + if c_default_ns is not NULL and c_node is c_element and tree.xmlStrcmp(c_ns_cache.new[i].href, c_default_ns.href) == 0: + c_node.ns = c_default_ns + else: + c_node.ns = c_ns_cache.new[i] break else: # not in cache => find a replacement from this document diff -r 452a2705665b -r 41fee8a923a1 src/lxml/tests/test_etree.py --- a/src/lxml/tests/test_etree.py Sun Apr 28 20:42:33 2013 +0200 +++ b/src/lxml/tests/test_etree.py Fri May 03 17:07:35 2013 -0700 @@ -2165,6 +2165,18 @@ _bytes('<foo xmlns="F"><bar xmlns:ns="NS" xmlns="B"><ns:baz/></bar></foo>'), self.etree.tostring(root)) + def test_append_attribute_namespace(self): + etree = self.etree + + root = etree.Element('{http://test/ns}root', nsmap={None: 'http://test/ns'}) + sub = etree.Element('{http://test/ns}sub', nsmap={'test': 'http://test/ns'}) + sub.attrib['{http://test/ns}attr'] = 'value' + self.assertEqual(sub.attrib['{http://test/ns}attr'], 'value') + root.append(sub) + self.assertEqual( + _bytes('<root xmlns="http://test/ns"><sub xmlns:test="http://test/ns" test:attr="value"/></root>'), + etree.tostring(root)) + def test_element_nsmap(self): etree = self.etree

2 5

Comments causing malloc/bus error/segmentation fault
by Ed Singleton May 10, 2013

May 10, 2013

Apologies if this isn't the right place to send this, but I'm not sure what to do with it. I put it on StackOverflow where it has been pretty much ignored (if anyone would prefer to answer it there). http://stackoverflow.com/questions/16459539/malloc-and-bus-error-in-python-… I've been having lots of problems with comments crashing my interpreter in lxml. A simplest case that causes malloc is: ``` import lxml import lxml.builder import lxml.html print "Causing malloc" builder = lxml.builder.ElementMaker() el = builder.div() el.append(lxml.html.HtmlComment("foo")) ``` I've tested this on python2.6 and python3.2 (with just a change to the print statements). It causes segmentation faults on python3.2 A more full example that also demonstrates more of what I've been trying to do is: ``` import lxml import lxml.builder import lxml.html import lxml.etree print "Causing Bus Error" class HtmlElement(lxml.html.HtmlElement): pass class HtmlElementLookup(lxml.html.HtmlElementClassLookup): def lookup(self, node_type, document, namespace, name): if node_type == 'comment': return lxml.etree.Comment else: return HtmlElement parser = lxml.html.HTMLParser() parser.set_element_class_lookup(HtmlElementLookup()) _HTMLBuilder = lxml.builder.ElementMaker( makeelement=parser.makeelement, typemap={ int: lambda e, i: str(i)}) class HtmlBuilder(object): def __getattr__(self, key): return getattr(_HTMLBuilder, key.lower()) builder = HtmlBuilder() el = builder.div() el.append(lxml.etree.Comment("foo")) for i in el: print i ``` This causes a bus error in python2.6 and a segmentation fault in python3.2. I'm not really sure how to proceed with this. I'd either like to get round it somehow (I can quite imagine I'm using it incorrectly), or fix it somehow. Thanks Ed

2 3

Re: [lxml] bidirectional python <-> xml transformations?
by Stefan Behnel May 7, 2013

May 7, 2013

Hi, please keep this on-list. Eric Levy, 08.05.2013 07:52: > On 05/08/2013 01:36 AM, Stefan Behnel wrote: >> Eric Levy, 08.05.2013 06:58: >>> It seems my initial message was not explicit enough. I was aware that the >>> output of the XPath query is basic Python objects, while the input required >>> for the XSLT transformation is a tree of objects defined by the lxml API. >>> The point that I wanted to make, though I admit I did not state it >>> outright, was that one expects, within in the context of the same software >>> library, that the same type of data is represented by the same datatypes. >>> So if I have XML data here, and XML data there, they are hopefully the same >>> data types (unless the library explicitly supports multiple representations >>> of the data). The reason for this expectation of consistency, or course, >>> is so that output from one operation can be input of another. >> >> That doesn't mean that you can pass anything into anything. The API that >> you are using for retrieving values is an API that can return different >> types of objects, whereas the API that you are using to construct the tree >> is an API that only handles Elements. You are appending to an Element here, >> and the ElementTree API reserves that operation to something that can be >> understood as children, i.e. Elements. >> >> When you pass an integer into something that expects an iterable, are you >> surprised to get an error? > > Obviously not, but my point concerns library design. If I wrote a library > to read and write to the system clock, would you expect the read feature to > process a long, and the write feature to return a string? And if they did > so, wouldn't you at least expect conversion functions as part of the library? Please read again what I wrote. >> I am not criticizing or second-guessing the design choices underlying >>> lxml. The purpose of the original post was to say: "I am trying to make >>> the input of one operation the output of another, all within the same >>> library, and it is surprising that this does not work. What, then, is the >>> intended way to deal with this issue within lxml?" >>> >>> I should also added that, as it is beginning to appear that there is no >>> straightforward solution, I have attempted another approach, where I >>> programmatically select the desired node rather than using an XPath >>> expression. I am able to do this, but when I try to recursively apply >>> templates to the selected node, I am seeing now is that the >>> apply_templates() routine does not work as it should, according to my >>> understanding of the documentation. Instead of processing descendent >>> elements as though they were in the original document, I am seeing that the >>> entire subtree is flattened to be text only (elements removed). >>> >>> I attached a sample case to demonstrate the issue. >>> >>> So again I am asking for help from the community, hoping you will have >>> useful suggestions. >> >> Yes, that's the way XSLT works. By default, it only copies text content >> when you call <xsl:apply-templates />. If you want to apply templates also >> to elements, you have to implement that. It's common to have a fallback >> "copy all" template to copy anything over that doesn't match any specific >> template. > > That's true for the input document, yes. It's how XSLT works, regardless of the source of the data. If you use the document() function in your stylesheet, it will do exactly the same thing. Everything else would be horribly inconsistent. And apply_templates() works *exactly* like <xsl:apply-templates />, for the same reason. > But the example in the lxml > documentation, reproduced below, shows that apply_templates() is being > being called with the parameter "child". That's the child of the > "self_node", which I can only assume is node of the extension, which > resides in the template document, not the input document. Again, the source doesn't matter. > So to me this > indicates that the child should be processed as templates. The rules in > this case are that all nodes are copied verbatim except XSLT elements (and > the children of certain XSLT elements). If I misunderstood, then I would > have to suggest that the documentation is misleading. Ah, so that's the source of the confusion. You are misreading apply_templates() as <xsl:call-template />. No, that's not what it is. I think what you really want is to explicitly add some content to the output_parent. You have both in your hands, so use the ET API to do that in exactly the way you want it to be. Or add a straight-forward recursive <xsl:copy> to your stylesheet to do it for you and call apply_templates(). Stefan

1 0

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 7, 2013

May 7, 2013

On 05/07/2013 12:59 AM, Stefan Behnel wrote: > Hi, > > one more remark. > > Stefan Behnel, 07.05.2013 06:45: >> Eric Levy, 07.05.2013 01:42: >>> On 05/06/2013 07:04 PM, Simon Sapin wrote: >>>> Le 07/05/2013 00:52, Eric Levy a écrit : >>>>> results[0] is type str which, like type list, causes append() to cause >>>>> an exception further up the stack. The contents of results[0] is the >>>>> string analogous to the XPath text() function of the query results. In >>>>> other words, it's a flattened representation of the result tree. >>>> >>>> I’m really not familiar with XSLT, but the API docs indicate that >>>> apply_templates() returns a list of elements and strings. I don’t know >>>> what determines which. >>>> >>>> http://lxml.de/api/ >>>> http://lxml.de/api/lxml.etree.XSLTExtension-class.html#apply_templates >>>> >>>> There is also an output_parent parameter that could be useful to you. >>>> >>>> >>>> But more generally, to resolve this kind of issues: look at the precise >>>> message of exceptions, be mindful of the types of your various values, >>>> and double-check docs (both tutorials and API) for hints. >>>> >>>> I feel like I’ve helped you more than enough with this now. I hope that >>>> you got the general idea of the debugging process to do it on your own. >>> >>> Well, I'm sorry that you feel that you've given too much help >> >> I think that what Simon meant is that he's given you all the help he could, >> without knowing your code and without having enough information to >> reproduce your problem. >> >> >>> but the >>> issue is not that I don't know how to debug my own work. The issue is that >>> either I am misunderstanding the lxml API, or the implementation is working >>> incorrectly. I had tried variations of the them, such as using the first >>> element of the XPath results rather than the list, but nothing worked. I >>> wrote my original message so that someone from the development team could >>> help me determine which is the case: my misuse of the library, or a bug >>> inside it. The core of the issue is that XPath generates one or more XML >>> trees, and XSLT templates operate on an XML tree to create an output tree. >>> Yet, when try to connect an XPath result to an XSLT input, then send the >>> XSLT results to the console, all in a very simple way, I don't have >>> success. Using the output_parent also causes the nodes to be flattened, >>> and additionally seems to cause the output to be not well-formed. >>> Hopefully I can understand why all my attempts are failing, and what needs >>> to happen for me to be successful. Thanks greatly to anyone who can help. >> >> From reading though this thread (and from looking at your initial code >> example), I get the impression that Simon is right. You do not seem to be >> aware of what types you pass into the API functions. The append() method >> only works for Elements, and extend() is not more than a little helper that >> appends a list of Elements, one after the other. >> >> My guess is that your input is not an Element (for append()) or a list of >> Elements (for extend()), but a list of different objects, perhaps >> containing other lists and/or strings. Please make sure that's not the >> case. If it is, you'll have to fix up your input by manually building the >> desired tree(s) that you want to append. > > You may want to take a look at the lxml.builder package for this. > >> lxml simply cannot know what it >> should do with a list of arbitrary objects. That's totally ambiguous input, >> so it refuses to guess and gives you an appropriate error. >> >> If you need more help with debugging your problem, please provide more >> information about what you are doing, specifically about the data you are >> trying to process and the kind of XPath query you are using to collect the >> input data. Otherwise, you make it impossible for others to guess what >> might be happening. > > Stefan It seems my initial message was not explicit enough. I was aware that the output of the XPath query is basic Python objects, while the input required for the XSLT transformation is a tree of objects defined by the lxml API. The point that I wanted to make, though I admit I did not state it outright, was that one expects, within in the context of the same software library, that the same type of data is represented by the same datatypes. So if I have XML data here, and XML data there, they are hopefully the same data types (unless the library explicitly supports multiple representations of the data). The reason for this expectation of consistency, or course, is so that output from one operation can be input of another. I am not criticizing or second-guessing the design choices underlying lxml. The purpose of the original post was to say: "I am trying to make the input of one operation the output of another, all within the same library, and it is surprising that this does not work. What, then, is the intended way to deal with this issue within lxml?" I should also added that, as it is beginning to appear that there is no straightforward solution, I have attempted another approach, where I programmatically select the desired node rather than using an XPath expression. I am able to do this, but when I try to recursively apply templates to the selected node, I am seeing now is that the apply_templates() routine does not work as it should, according to my understanding of the documentation. Instead of processing descendent elements as though they were in the original document, I am seeing that the entire subtree is flattened to be text only (elements removed). I attached a sample case to demonstrate the issue. So again I am asking for help from the community, hoping you will have useful suggestions. Eric Levy Eric Levy

2 1

UnicodeDecodeError in lxml
by Joaquin Cuenca Abela May 7, 2013

May 7, 2013

Hi, we are parsing lots of html files with lxml, and we're facing some problems with their encoding. Short story, we want to parse an HTML with an encoding even if the HTML is not fully correctly encoded. For example, we may have a 500K file correctly encoded in UTF-8 but with a single word encoded in iso-latin-1. lxml gives a UnicodeDecodeError when we do: self._html = lxml.html.fromstring( html_text, parser=lxml.html.HTMLParser( encoding='utf-8', remove_blank_text=True, remove_comments=True, remove_pis=True)) It seems the only way around it is to sanitize the text before sending it to lxml, which is a bit slow: self._html = lxml.html.fromstring( html_text.decode('utf-8', errors='ignore'), parser=lxml.html.HTMLParser( encoding='unicode', remove_blank_text=True, remove_comments=True, remove_pis=True)) This fails on documents that have an xml encoding declaration (<?xml version="1.0" encoding="utf-8"?>), so I can't use it blindly. I have to encode the unicode string to utf-8 before I pass it to lxml: self._html = lxml.html.fromstring( html_text.decode('utf-8', errors='ignore').encode('utf-8'), parser=lxml.html.HTMLParser( encoding=unicode, remove_blank_text=True, remove_comments=True, remove_pis=True)) But this decoding / reencoding seems (and usually it's) unnecessary. Is there any known way to parse a html with some potential encoding errors other than decoding and reencoding the html? Thanks! -- Joaquin Cuenca Abela

2 2

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

Well, I'm sorry that you feel that you've given too much help, but the issue is not that I don't know how to debug my own work. The issue is that either I am misunderstanding the lxml API, or the implementation is working incorrectly. I had tried variations of the them, such as using the first element of the XPath results rather than the list, but nothing worked. I wrote my original message so that someone from the development team could help me determine which is the case: my misuse of the library, or a bug inside it. The core of the issue is that XPath generates one or more XML trees, and XSLT templates operate on an XML tree to create an output tree. Yet, when try to connect an XPath result to an XSLT input, then send the XSLT results to the console, all in a very simple way, I don't have success. Using the output_parent also causes the nodes to be flattened, and additionally seems to cause the output to be not well-formed. Hopefully I can understand why all my attempts are failing, and what needs to happen for me to be successful. Thanks greatly to anyone who can help. On 05/06/2013 07:04 PM, Simon Sapin wrote: > Le 07/05/2013 00:52, Eric Levy a écrit : >> results[0] is type str which, like type list, causes append() to cause >> an exception further up the stack. The contents of results[0] is the >> string analogous to the XPath text() function of the query results. In >> other words, it's a flattened representation of the result tree. > > I’m really not familiar with XSLT, but the API docs indicate that > apply_templates() returns a list of elements and strings. I don’t know > what determines which. > > http://lxml.de/api/ > http://lxml.de/api/lxml.etree.XSLTExtension-class.html#apply_templates > > There is also an output_parent parameter that could be useful to you. > > > But more generally, to resolve this kind of issues: look at the precise > message of exceptions, be mindful of the types of your various values, > and double-check docs (both tutorials and API) for hints. > > I feel like I’ve helped you more than enough with this now. I hope that > you got the general idea of the debugging process to do it on your own. > > Cheers,

2 3