Mailman 3 May 2013 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 11, 2023

April 11, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 10, 2023

April 10, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

lxml.objectify drops leading whitespace
by Wolfgang Schnerring Oct. 2, 2013

Oct. 2, 2013

Hello, lxml.objectify drops leading whitespace before a tag: >>> lxml.etree.tostring(lxml.objectify.fromstring(' <a>foo</a>')) '<a>foo</a>' while lxml.etree handles the same situation just fine: >>> lxml.etree.tostring(lxml.etree.fromstring(' <a>foo</a>')) ' <a>foo</a>' What's the reason for the different behaviour? Is this intentional? Most importantly, how can I make lxml.objectify preserve whitespace correctly? Thanks for your help, Wolfgang

2 2

Changing encoding while parsing an HTML string
by Michael O'Leary June 2, 2013

June 2, 2013

If I understand how the XML parser works, if it parses an XML string that begins with a header line like <?xml version="1.0" encoding="utf-8"?> it will parse the rest of the XML string using the encoding of the encoding it has found. So first of all, does the XML parser do this? If so, I'd like to know if something like this can be done, either automatically or manually, with the HTML parser. I have been parsing HTML strings that have been obtained from a random sampling of web sites, and I am wondering if there is a way to do something similar with these HTML strings. Typically, these HTML strings do not start with a header like, but most of them do have a meta tag of the form <meta charset="utf-8"> or <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" > I wouldn't expect that the HTML parser reads these tags and automatically sets the encoding to be used while parsing the rest of the HTML string (but I might as well ask: does it?). I could include code in my target's start tag handler that checks meta tags to see if they contain encoding information and calls the parser to set its encoding, but as far as I can see there isn't a way to set the encoding for an HTML parser after it has been created. That is, it looks like you can do this parser = etree.HTMLParser(target=my_target, encoding=my_encoding) but not this parser = etree.HTMLParser(target=my_target) .... figure out what value my_encoding should have .... parser.set_encoding(my_encoding) So I guess my underlying question is, how can I parse HTML strings and give the parser the best chance to use the right encoding for the part of the HTML string that is in the <body> .... </body> section of it? Thanks, Mike

2 1

Re: [lxml] wrong ELF class
by Wolfgang Buesser May 27, 2013

May 27, 2013

>Le 27/05/2013 01:55, Wolfgang Buesser a écrit : >> I have installed python-lxml-2.3.4-6.1.1.x86_64.rpm >> >> […] >> >> Could it be possible, that the architecture in the RPM does not match >> 64-Bit (despite it's name)? >It depends, where did you get this RPM file? >> How can I check? >Try running this command: >file /usr/lib/python2.7/site-packages/lxml/etree.so >Cheers, >-- >Simon Sapin Hi Simon, Hi Holger, with your help I found out that the rpm was actually i586. I don't know how that happened. As far as I remember it was part of official SuSE release. Anyway I replaced with an rpm from rpm-find and now my application is running. Thanks Wolfgang

1 0

wrong ELF class
by Wolfgang Buesser May 27, 2013

May 27, 2013

Hello, I am running SUSE 12.3 on an AMD-64 System. I have installed python-lxml-2.3.4-6.1.1.x86_64.rpm When lxml gets called from gnuradio-companion I get the following error-message: ImportError: /usr/lib/python2.7/site-packages/lxml/etree.so: wrong ELF class: ELFCLASS32 Could it be possible, that the architecture in the RPM does not match 64-Bit (despite it's name)? How can I check? How can I get a working python-lxml for my system? Thanks Wolfgang

3 2

Error seen when running unit tests
by Lindykid May 25, 2013

May 25, 2013

Deal lxml Developers, I want to use your library for some projects that I will need XML parsing and XSD verification. I am attempting to install lxml 3.2.1 onto a RHEL 6.1 server which has python 2.6.6 installed. I did not use pip to build and install lxml, rather I used the 'python setup.py install'. Everything seemed to build and install fine. I was able to run through a few tutorials with out any issue. My problem came when I decided to try and run the unit tests that you have provided. I get one failure (see bottom of this E-Mail) when I run the 'make test' or 'python test.py -p -v' command. I have tried this on lxml versions of 3.1.2, 3.2.0, and 3.2.1 and I see the same test error on each release. I have attempted to sower the web, and the lxml mailing list archives to see if I could find any information on what is causing this test to fail, but I have come up empty. I am hoping that one of you can tell me what I am doing wrong, or what I can to to get this test to pass. Thanks so much for your time, - Chris # make test Skipping tests in lxml.cssselect - external cssselect package is not installed Comparing with ElementTree 1.2.6 TESTED VERSION: 3.2.1 Python: (2, 6, 6, 'final', 0) lxml.etree: (3, 2, 1, 0) libxml used: (2, 7, 6) libxml compiled: (2, 7, 6) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 26) 1479/1479 (100.0%): Doctest: xpathxslt.txt ====================================================================== ERROR: test_feed_parser_unicode (lxml.tests.test_elementtree.ETreeTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.6/unittest.py", line 278, in run testMethod() File "/root/chris/lxml-3.2.1/src/lxml/tests/test_elementtree.py", line 3444, in test_feed_parser_unicode parser.feed(_str('<ro')) File "parser.pxi", line 1177, in lxml.etree._FeedParser.feed (src/lxml/lxml.etree.c:90354) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResult (src/lxml/lxml.etree.c:84648) File "parser.pxi", line 676, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:85816) File "parser.pxi", line 627, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:85308) XMLSyntaxError: None ---------------------------------------------------------------------- Ran 1479 tests in 14.117s FAILED (errors=1)

2 1

lxml object architecture quandary
by John W. Shipman May 21, 2013

May 21, 2013

I have a design problem involving ElementTree. I'd appreciate any suggestions. The problem is posted online here: http://johnwshipman.blogspot.com/ Best regards, John Shipman (john(a)nmt.edu), Applications Specialist New Mexico Tech Computer Center, Speare 146, Socorro, NM 87801 (575) 835-5735, http://www.nmt.edu/~john ``Let's go outside and commiserate with nature.'' --Dave Farber

3 3

Question on 1) encoding/decoding and 2) importing xsd
by Sangeeth Saravanaraj May 16, 2013

May 16, 2013

I have the following snippet of code which validates the given xml data against a given xsd. <code> from lxml import etree from io import StringIO, BytesIO with open("schema1.xsd", "r") as schema_file: schemaContent = schema_file.read() with open("data.xml", "r") as xml_file: xmlContent = xml_file.read() xmlschema_doc = etree.parse(StringIO(schemaContent.decode('iso8859_1'))) xml_doc = etree.parse(StringIO(xmlContent.decode('iso8859_1'))) xmlschema = etree.XMLSchema(xmlschema_doc) print xmlschema.validate(xml_doc) </code> The above code usually works fine but not under the following scenarios. *#1) ValueError: Unicode strings with encoding declaration are not supported.* I see ValueError when the .xsd or .xml file begins with an 'xml' element which contains the 'encoding' attribute as follows <?xml version="1.0" encoding="UTF-8"?> When I remove the 'encoding' attribute, the problem vanishes. For example: <?xml version="1.0"?> I referred the Python Standard Encodings - http://docs.python.org/2/library/codecs.html#standard-encodings and tried the following but couldn't get the xml/xsd data decoded when 'xml' element contains 'encoding' attribute. schemaContent.decode(encoding='UTF-8', errors='ignore') schemaContent.decode(encoding='ISO-8859-1', errors='ignore') What am I missing here?! *#2 lxml.etree.XMLSchemaParseError: failed to load external entity "schema2.xsd"* Say I have the following directory structure: proj_home |-- validate.py `-- schema |-- schema1.xsd `-- schema2.xsd Say proj_home/schema/schema1.xsd imports schema2.xsd (which is present proj_home/schema/) as follows: <xs:import namespace="namespace url goes here" schemaLocation="schema2.xsd"/> Now, when the validate.py reads proj_home/schema/schema1.xsd and uses that in etree.XMLSchema(), it raises lxml.etree.XMLSchemaParseError as schema2.xsd is not present in proj_home but present in proj_home/schema directory. If I copy schema2.xsd to proj_home (in the same directory where validate.py exists) then it works fine! How should I make etree.XMLSchema() to load the "import" files from a particular directory (in this case proj_home/schema) instead of looking at the current working directory? Thank you, Sangeeth

3 2

Parsing problem
by Arnaud RIVET May 15, 2013

May 15, 2013

Hi, I am new with LXML and I have a problem after parsing my element : its architecture seems to have changed if I remove (or replace) the last child. Here is my code and a screen of the console output. I have looked up for solution but I still can't figure what I am doing wrong. I would really appreciate someone's help ! (I am using LXML 3.2.1 with Python 2.6 on Windows) ############################################################################################# from lxml import etree from copy import deepcopy def Write( file, element ): f = open( file, 'w' ) f.write( etree.tostring( element, xml_declaration=True, encoding="ISO-8859-1", pretty_print = True ) ) f.close() return 1 def ReadAndReturn( file ): lookup = etree.ElementDefaultClassLookup() parser = etree.XMLParser(recover = True) parser.set_element_class_lookup( lookup ) mainTree = etree.parse( file, parser ) return mainTree # create a root element with 3 children root = etree.Element( "root" ) root.append( etree.Element( "child1" ) ) child2 = etree.SubElement( root, "child2" ) child2.text = 'CHILD2' child3 = etree.SubElement( root, "child3" ) child3.text = 'CHILD3' print "\n--- INITIAL ROOT ---" print( etree.tostring( root, pretty_print=True ) ) # remove last child root2 = deepcopy( root ) root2.remove( root2[2] ) print "--- ROOT WITHOUT LAST CHILD / BEFORE WRITING ---" print( etree.tostring( root2, pretty_print=True ) ) # write initial root (3 children) and read the file filename = 'test.tst' status = Write( filename, root ) tree = ReadAndReturn( filename ) # remove last child from the read element root3 = deepcopy( tree.getroot() ) root3.remove( root3[2] ) print "--- ROOT WITHOUT LAST CHILD / AFTER WRITING AND PARSING ---" print( etree.tostring( root3, pretty_print=True ) ) #############################################################################################

3 2

lxml 3.2.1 released
by Stefan Behnel May 11, 2013

May 11, 2013

Hi everyone, I just sent out lxml 3.2.1 as the first major bug-fix release for the stable 3.2 release series. You can get it from PyPI: http://pypi.python.org/pypi/lxml/3.2.1 The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.2.1.tgz Signature: http://lxml.de/files/lxml-3.2.1.tgz.asc Changelog: http://lxml.de/3.2/changes-3.2.1.html GitHub: https://github.com/lxml/lxml/commit/90afe19b11c4fd1df8998b254d8a60b550aa56d0 This release was built using Cython 0.19.1. Note that the build no longer uses Cython, even if it is installed. Recompilation of the sources has to be requested explicitly with the setup.py option "--with-cython". If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.2.1 (2013-05-11) ================== Features added -------------- * The methods ``apply_templates()`` and ``process_children()`` of XSLT extension elements have gained two new boolean options ``elements_only`` and ``remove_blank_text`` that discard either all strings or whitespace-only strings from the result list. Bugs fixed ---------- * When moving Elements to another tree, the namespace cleanup mechanism no longer drops namespace prefixes from attributes for which it finds a default namespace declaration, to prevent them from appearing as unnamespaced attributes after serialisation. * Returning non-type objects from a custom class lookup method could lead to a crash. * Instantiating and using subtypes of Comments and ProcessingInstructions crashed.

1 0

Patch for losing namespace on attributes
by Tyler Spivey May 11, 2013

May 11, 2013

I tried patching this. I'm not sure this is the right way of doing things, but it passes the existing tests as well as the one added for it. diff -r 452a2705665b -r 41fee8a923a1 src/lxml/proxy.pxi --- a/src/lxml/proxy.pxi Sun Apr 28 20:42:33 2013 +0200 +++ b/src/lxml/proxy.pxi Fri May 03 17:07:35 2013 -0700 @@ -287,7 +287,7 @@ while c_nsdef[0] is not NULL: c_ns = tree.xmlSearchNsByHref( c_element.doc, c_element.parent, c_nsdef[0].href) - if c_ns is NULL: + if c_ns is NULL or c_ns.prefix is NULL: # new namespace href => keep and cache the ns declaration _appendToNsCache(c_ns_cache, c_nsdef[0], c_nsdef[0]) c_nsdef = &c_nsdef[0].next @@ -340,6 +340,7 @@ cdef xmlNs* c_ns_next cdef xmlNs* c_nsdef cdef xmlNs* c_del_ns_list + cdef xmlNs* c_default_ns = NULL cdef size_t i, proxy_count = 0 if not tree._isElementOrXInclude(c_element): @@ -366,12 +367,18 @@ # 2) make sure the namespaces of an element and its attributes # are declared in this document (i.e. on the node or its parents) + #Find the default namespace for this element in either the node or its parents + c_default_ns = tree.xmlSearchNs(c_source_doc, c_element, NULL) + c_node = c_element while c_node is not NULL: if c_node.ns is not NULL: for i from 0 <= i < c_ns_cache.last: if c_node.ns is c_ns_cache.old[i]: - c_node.ns = c_ns_cache.new[i] + if c_default_ns is not NULL and c_node is c_element and tree.xmlStrcmp(c_ns_cache.new[i].href, c_default_ns.href) == 0: + c_node.ns = c_default_ns + else: + c_node.ns = c_ns_cache.new[i] break else: # not in cache => find a replacement from this document diff -r 452a2705665b -r 41fee8a923a1 src/lxml/tests/test_etree.py --- a/src/lxml/tests/test_etree.py Sun Apr 28 20:42:33 2013 +0200 +++ b/src/lxml/tests/test_etree.py Fri May 03 17:07:35 2013 -0700 @@ -2165,6 +2165,18 @@ _bytes('<foo xmlns="F"><bar xmlns:ns="NS" xmlns="B"><ns:baz/></bar></foo>'), self.etree.tostring(root)) + def test_append_attribute_namespace(self): + etree = self.etree + + root = etree.Element('{http://test/ns}root', nsmap={None: 'http://test/ns'}) + sub = etree.Element('{http://test/ns}sub', nsmap={'test': 'http://test/ns'}) + sub.attrib['{http://test/ns}attr'] = 'value' + self.assertEqual(sub.attrib['{http://test/ns}attr'], 'value') + root.append(sub) + self.assertEqual( + _bytes('<root xmlns="http://test/ns"></root>'), + etree.tostring(root)) + def test_element_nsmap(self): etree = self.etree

2 5

Comments causing malloc/bus error/segmentation fault
by Ed Singleton May 10, 2013

May 10, 2013

Apologies if this isn't the right place to send this, but I'm not sure what to do with it. I put it on StackOverflow where it has been pretty much ignored (if anyone would prefer to answer it there). http://stackoverflow.com/questions/16459539/malloc-and-bus-error-in-python-… I've been having lots of problems with comments crashing my interpreter in lxml. A simplest case that causes malloc is: ``` import lxml import lxml.builder import lxml.html print "Causing malloc" builder = lxml.builder.ElementMaker() el = builder.div() el.append(lxml.html.HtmlComment("foo")) ``` I've tested this on python2.6 and python3.2 (with just a change to the print statements). It causes segmentation faults on python3.2 A more full example that also demonstrates more of what I've been trying to do is: ``` import lxml import lxml.builder import lxml.html import lxml.etree print "Causing Bus Error" class HtmlElement(lxml.html.HtmlElement): pass class HtmlElementLookup(lxml.html.HtmlElementClassLookup): def lookup(self, node_type, document, namespace, name): if node_type == 'comment': return lxml.etree.Comment else: return HtmlElement parser = lxml.html.HTMLParser() parser.set_element_class_lookup(HtmlElementLookup()) _HTMLBuilder = lxml.builder.ElementMaker( makeelement=parser.makeelement, typemap={ int: lambda e, i: str(i)}) class HtmlBuilder(object): def __getattr__(self, key): return getattr(_HTMLBuilder, key.lower()) builder = HtmlBuilder() el = builder.div() el.append(lxml.etree.Comment("foo")) for i in el: print i ``` This causes a bus error in python2.6 and a segmentation fault in python3.2. I'm not really sure how to proceed with this. I'd either like to get round it somehow (I can quite imagine I'm using it incorrectly), or fix it somehow. Thanks Ed

2 3

Re: [lxml] bidirectional python <-> xml transformations?
by Stefan Behnel May 8, 2013

May 8, 2013

Hi, please keep this on-list. Eric Levy, 08.05.2013 07:52: > On 05/08/2013 01:36 AM, Stefan Behnel wrote: >> Eric Levy, 08.05.2013 06:58: >>> It seems my initial message was not explicit enough. I was aware that the >>> output of the XPath query is basic Python objects, while the input required >>> for the XSLT transformation is a tree of objects defined by the lxml API. >>> The point that I wanted to make, though I admit I did not state it >>> outright, was that one expects, within in the context of the same software >>> library, that the same type of data is represented by the same datatypes. >>> So if I have XML data here, and XML data there, they are hopefully the same >>> data types (unless the library explicitly supports multiple representations >>> of the data). The reason for this expectation of consistency, or course, >>> is so that output from one operation can be input of another. >> >> That doesn't mean that you can pass anything into anything. The API that >> you are using for retrieving values is an API that can return different >> types of objects, whereas the API that you are using to construct the tree >> is an API that only handles Elements. You are appending to an Element here, >> and the ElementTree API reserves that operation to something that can be >> understood as children, i.e. Elements. >> >> When you pass an integer into something that expects an iterable, are you >> surprised to get an error? > > Obviously not, but my point concerns library design. If I wrote a library > to read and write to the system clock, would you expect the read feature to > process a long, and the write feature to return a string? And if they did > so, wouldn't you at least expect conversion functions as part of the library? Please read again what I wrote. >> I am not criticizing or second-guessing the design choices underlying >>> lxml. The purpose of the original post was to say: "I am trying to make >>> the input of one operation the output of another, all within the same >>> library, and it is surprising that this does not work. What, then, is the >>> intended way to deal with this issue within lxml?" >>> >>> I should also added that, as it is beginning to appear that there is no >>> straightforward solution, I have attempted another approach, where I >>> programmatically select the desired node rather than using an XPath >>> expression. I am able to do this, but when I try to recursively apply >>> templates to the selected node, I am seeing now is that the >>> apply_templates() routine does not work as it should, according to my >>> understanding of the documentation. Instead of processing descendent >>> elements as though they were in the original document, I am seeing that the >>> entire subtree is flattened to be text only (elements removed). >>> >>> I attached a sample case to demonstrate the issue. >>> >>> So again I am asking for help from the community, hoping you will have >>> useful suggestions. >> >> Yes, that's the way XSLT works. By default, it only copies text content >> when you call <xsl:apply-templates />. If you want to apply templates also >> to elements, you have to implement that. It's common to have a fallback >> "copy all" template to copy anything over that doesn't match any specific >> template. > > That's true for the input document, yes. It's how XSLT works, regardless of the source of the data. If you use the document() function in your stylesheet, it will do exactly the same thing. Everything else would be horribly inconsistent. And apply_templates() works *exactly* like <xsl:apply-templates />, for the same reason. > But the example in the lxml > documentation, reproduced below, shows that apply_templates() is being > being called with the parameter "child". That's the child of the > "self_node", which I can only assume is node of the extension, which > resides in the template document, not the input document. Again, the source doesn't matter. > So to me this > indicates that the child should be processed as templates. The rules in > this case are that all nodes are copied verbatim except XSLT elements (and > the children of certain XSLT elements). If I misunderstood, then I would > have to suggest that the documentation is misleading. Ah, so that's the source of the confusion. You are misreading apply_templates() as <xsl:call-template />. No, that's not what it is. I think what you really want is to explicitly add some content to the output_parent. You have both in your hands, so use the ET API to do that in exactly the way you want it to be. Or add a straight-forward recursive <xsl:copy> to your stylesheet to do it for you and call apply_templates(). Stefan

1 0

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 8, 2013

May 8, 2013

On 05/07/2013 12:59 AM, Stefan Behnel wrote: > Hi, > > one more remark. > > Stefan Behnel, 07.05.2013 06:45: >> Eric Levy, 07.05.2013 01:42: >>> On 05/06/2013 07:04 PM, Simon Sapin wrote: >>>> Le 07/05/2013 00:52, Eric Levy a écrit : >>>>> results[0] is type str which, like type list, causes append() to cause >>>>> an exception further up the stack. The contents of results[0] is the >>>>> string analogous to the XPath text() function of the query results. In >>>>> other words, it's a flattened representation of the result tree. >>>> >>>> I’m really not familiar with XSLT, but the API docs indicate that >>>> apply_templates() returns a list of elements and strings. I don’t know >>>> what determines which. >>>> >>>> http://lxml.de/api/ >>>> http://lxml.de/api/lxml.etree.XSLTExtension-class.html#apply_templates >>>> >>>> There is also an output_parent parameter that could be useful to you. >>>> >>>> >>>> But more generally, to resolve this kind of issues: look at the precise >>>> message of exceptions, be mindful of the types of your various values, >>>> and double-check docs (both tutorials and API) for hints. >>>> >>>> I feel like I’ve helped you more than enough with this now. I hope that >>>> you got the general idea of the debugging process to do it on your own. >>> >>> Well, I'm sorry that you feel that you've given too much help >> >> I think that what Simon meant is that he's given you all the help he could, >> without knowing your code and without having enough information to >> reproduce your problem. >> >> >>> but the >>> issue is not that I don't know how to debug my own work. The issue is that >>> either I am misunderstanding the lxml API, or the implementation is working >>> incorrectly. I had tried variations of the them, such as using the first >>> element of the XPath results rather than the list, but nothing worked. I >>> wrote my original message so that someone from the development team could >>> help me determine which is the case: my misuse of the library, or a bug >>> inside it. The core of the issue is that XPath generates one or more XML >>> trees, and XSLT templates operate on an XML tree to create an output tree. >>> Yet, when try to connect an XPath result to an XSLT input, then send the >>> XSLT results to the console, all in a very simple way, I don't have >>> success. Using the output_parent also causes the nodes to be flattened, >>> and additionally seems to cause the output to be not well-formed. >>> Hopefully I can understand why all my attempts are failing, and what needs >>> to happen for me to be successful. Thanks greatly to anyone who can help. >> >> From reading though this thread (and from looking at your initial code >> example), I get the impression that Simon is right. You do not seem to be >> aware of what types you pass into the API functions. The append() method >> only works for Elements, and extend() is not more than a little helper that >> appends a list of Elements, one after the other. >> >> My guess is that your input is not an Element (for append()) or a list of >> Elements (for extend()), but a list of different objects, perhaps >> containing other lists and/or strings. Please make sure that's not the >> case. If it is, you'll have to fix up your input by manually building the >> desired tree(s) that you want to append. > > You may want to take a look at the lxml.builder package for this. > >> lxml simply cannot know what it >> should do with a list of arbitrary objects. That's totally ambiguous input, >> so it refuses to guess and gives you an appropriate error. >> >> If you need more help with debugging your problem, please provide more >> information about what you are doing, specifically about the data you are >> trying to process and the kind of XPath query you are using to collect the >> input data. Otherwise, you make it impossible for others to guess what >> might be happening. > > Stefan It seems my initial message was not explicit enough. I was aware that the output of the XPath query is basic Python objects, while the input required for the XSLT transformation is a tree of objects defined by the lxml API. The point that I wanted to make, though I admit I did not state it outright, was that one expects, within in the context of the same software library, that the same type of data is represented by the same datatypes. So if I have XML data here, and XML data there, they are hopefully the same data types (unless the library explicitly supports multiple representations of the data). The reason for this expectation of consistency, or course, is so that output from one operation can be input of another. I am not criticizing or second-guessing the design choices underlying lxml. The purpose of the original post was to say: "I am trying to make the input of one operation the output of another, all within the same library, and it is surprising that this does not work. What, then, is the intended way to deal with this issue within lxml?" I should also added that, as it is beginning to appear that there is no straightforward solution, I have attempted another approach, where I programmatically select the desired node rather than using an XPath expression. I am able to do this, but when I try to recursively apply templates to the selected node, I am seeing now is that the apply_templates() routine does not work as it should, according to my understanding of the documentation. Instead of processing descendent elements as though they were in the original document, I am seeing that the entire subtree is flattened to be text only (elements removed). I attached a sample case to demonstrate the issue. So again I am asking for help from the community, hoping you will have useful suggestions. Eric Levy Eric Levy

2 1

UnicodeDecodeError in lxml
by Joaquin Cuenca Abela May 7, 2013

May 7, 2013

Hi, we are parsing lots of html files with lxml, and we're facing some problems with their encoding. Short story, we want to parse an HTML with an encoding even if the HTML is not fully correctly encoded. For example, we may have a 500K file correctly encoded in UTF-8 but with a single word encoded in iso-latin-1. lxml gives a UnicodeDecodeError when we do: self._html = lxml.html.fromstring( html_text, parser=lxml.html.HTMLParser( encoding='utf-8', remove_blank_text=True, remove_comments=True, remove_pis=True)) It seems the only way around it is to sanitize the text before sending it to lxml, which is a bit slow: self._html = lxml.html.fromstring( html_text.decode('utf-8', errors='ignore'), parser=lxml.html.HTMLParser( encoding='unicode', remove_blank_text=True, remove_comments=True, remove_pis=True)) This fails on documents that have an xml encoding declaration (<?xml version="1.0" encoding="utf-8"?>), so I can't use it blindly. I have to encode the unicode string to utf-8 before I pass it to lxml: self._html = lxml.html.fromstring( html_text.decode('utf-8', errors='ignore').encode('utf-8'), parser=lxml.html.HTMLParser( encoding=unicode, remove_blank_text=True, remove_comments=True, remove_pis=True)) But this decoding / reencoding seems (and usually it's) unnecessary. Is there any known way to parse a html with some potential encoding errors other than decoding and reencoding the html? Thanks! -- Joaquin Cuenca Abela

2 2

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 7, 2013

May 7, 2013

Well, I'm sorry that you feel that you've given too much help, but the issue is not that I don't know how to debug my own work. The issue is that either I am misunderstanding the lxml API, or the implementation is working incorrectly. I had tried variations of the them, such as using the first element of the XPath results rather than the list, but nothing worked. I wrote my original message so that someone from the development team could help me determine which is the case: my misuse of the library, or a bug inside it. The core of the issue is that XPath generates one or more XML trees, and XSLT templates operate on an XML tree to create an output tree. Yet, when try to connect an XPath result to an XSLT input, then send the XSLT results to the console, all in a very simple way, I don't have success. Using the output_parent also causes the nodes to be flattened, and additionally seems to cause the output to be not well-formed. Hopefully I can understand why all my attempts are failing, and what needs to happen for me to be successful. Thanks greatly to anyone who can help. On 05/06/2013 07:04 PM, Simon Sapin wrote: > Le 07/05/2013 00:52, Eric Levy a écrit : >> results[0] is type str which, like type list, causes append() to cause >> an exception further up the stack. The contents of results[0] is the >> string analogous to the XPath text() function of the query results. In >> other words, it's a flattened representation of the result tree. > > I’m really not familiar with XSLT, but the API docs indicate that > apply_templates() returns a list of elements and strings. I don’t know > what determines which. > > http://lxml.de/api/ > http://lxml.de/api/lxml.etree.XSLTExtension-class.html#apply_templates > > There is also an output_parent parameter that could be useful to you. > > > But more generally, to resolve this kind of issues: look at the precise > message of exceptions, be mindful of the types of your various values, > and double-check docs (both tutorials and API) for hints. > > I feel like I’ve helped you more than enough with this now. I hope that > you got the general idea of the debugging process to do it on your own. > > Cheers,

2 3

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

results[0] is type str which, like type list, causes append() to cause an exception further up the stack. The contents of results[0] is the string analogous to the XPath text() function of the query results. In other words, it's a flattened representation of the result tree. On 05/06/2013 06:36 PM, Simon Sapin wrote: > Le 07/05/2013 00:19, Eric Levy a écrit : >> I want to append the first item in the list. I tried passing string[0] >> to apply_templates(), and then passing results[0] to append(), but >> regardless of whether I am passing the entire list or only the first >> element, the stack trace looks the same, with an exception coming from >> lxml.etree.c. > > And the message is still "TypeError: invalid argument type <type > 'list'>"? What does results[0] and type(results[0]) look like? (Try > using pdb or adding print statements.) >

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

I want to append the first item in the list. I tried passing string[0] to apply_templates(), and then passing results[0] to append(), but regardless of whether I am passing the entire list or only the first element, the stack trace looks the same, with an exception coming from lxml.etree.c. On 05/06/2013 04:55 PM, Eric P. Levy wrote: > As stated, I want to append the first item, but I'm not seeing how your suggestion differs from what I have. > > Simon Sapin <simon.sapin(a)exyr.org> wrote: > > Le 06/05/2013 22:07, Eric Levy a écrit : >> Hopefully this will help. Note the callback occurs about halfway up the >> stack. >> >> Traceback (most recent call last): >> File "index.py", line 107, in <module> >> rslt_tree = trans(in_tree, mobile="'" + mobile_str + "'") >> File "xslt.pxi", line 582, in lxml.etree.XSLT.__call__ >> (src/lxml/lxml.etree.c:139795) >> File "lxml.etree.pyx", line 294, in >> lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9359) >> File "xsltext.pxi", line 195, in lxml.etree._callExtensionElement >> (src/lxml/lxml.etree.c:144738) >> File "index.py", line 100, in execute >> output_parent.append(results) >> File "readonlytree.pxi", line 475, in >> lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69888) >> File "readonlytree.pxi", line 481, in >> lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69807) >> File "readonlytree.pxi", line 534, in lxml.etree._roNodeOf >> (src/lxml/lxml.etree.c:70448) >> TypeError: invalid argument type <type 'list'> > > It seems this is the same issue as before: apply_templates() returns a > list of results; element objects behave kind of like a list of their > children and the .append() method expects a child element object rather > than a list. > > Try one of these, depending on what you mean: > > output_parent.append(results[0]) > output_parent.extend(results) > > (append takes one item to add, the list grows by one; extend takes an > iterable of items, the list grows by the length of the iterable.) >

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

Hopefully this will help. Note the callback occurs about halfway up the stack. Traceback (most recent call last): File "index.py", line 107, in <module> rslt_tree = trans(in_tree, mobile="'" + mobile_str + "'") File "xslt.pxi", line 582, in lxml.etree.XSLT.__call__ (src/lxml/lxml.etree.c:139795) File "lxml.etree.pyx", line 294, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9359) File "xsltext.pxi", line 195, in lxml.etree._callExtensionElement (src/lxml/lxml.etree.c:144738) File "index.py", line 100, in execute output_parent.append(results) File "readonlytree.pxi", line 475, in lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69888) File "readonlytree.pxi", line 481, in lxml.etree._AppendOnlyElementProxy.append (src/lxml/lxml.etree.c:69807) File "readonlytree.pxi", line 534, in lxml.etree._roNodeOf (src/lxml/lxml.etree.c:70448) TypeError: invalid argument type <type 'list'> On 05/06/2013 04:01 PM, Simon Sapin wrote: > Le 06/05/2013 21:51, Eric Levy a écrit : >> I may have put the issue in a confusing way, because I used the term >> "string", but I am actually trying to capture a an entire results tree >> (or the first item if there is a list of results), and if possible, >> recursively apply XSLT templates to those results. Consequently, I >> cannot use the XPath text() function because this flattens the results, >> rather than keeping them as a tree. I should also note that in the line >> reading "results = self.apply_templates(context, string)", I tried >> changing "string" to "string[0]", but the effect was the same. >> >> I hope this makes clearer what I am trying to do. > > Ok, so forget the part about strings and text. Can you copy the full > traceback you get with string[0]? > > Cheers,

2 1

Re: [lxml] bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

I may have put the issue in a confusing way, because I used the term "string", but I am actually trying to capture a an entire results tree (or the first item if there is a list of results), and if possible, recursively apply XSLT templates to those results. Consequently, I cannot use the XPath text() function because this flattens the results, rather than keeping them as a tree. I should also note that in the line reading "results = self.apply_templates(context, string)", I tried changing "string" to "string[0]", but the effect was the same. I hope this makes clearer what I am trying to do. On 05/06/2013 03:37 PM, Simon Sapin wrote: > Le 06/05/2013 18:41, Eric Levy a écrit : >> Currently, the apply_templates call is failing, with the following error: >>> TypeError: invalid argument type <type 'list'> > > Hi, > > That seems pretty self-explanatory. XPath returns a list of results, and > according to the doc[1] example apply_templates() expects a single > element object. > > [1] http://lxml.de/extensions.html > > If your XPath expression is constructed such that there is exactly one > result, use `foo[0]` to extract it from the list. If you use `text()` in > XPath or something such that results are strings rather than element > objects, you might need to construct a dummy object to pass to > apply_templates(): > > el = lxml.etree.Element('dummy') > el.text = xpath_results[0] >

2 1

bidirectional python <-> xml transformations?
by Eric Levy May 6, 2013

May 6, 2013

Hello, I am experience an issue writing data python data that had been generated by an XPath query. It appears that lxml uses some pythonic magic to transform between Python objects and XML data. It also appears, unfortunately that the same data that was generated by the library cannot be written back to XML. Specifically, I am seeking to develop a simple web page that is based on lxml applying XSLT templates. Human readable text, as well as high-level formatting information, is kept in XML documents. XSLT templates transform the documents to HTML pages. Because internationalization is necessary, text is kept separate documents. The particular document used depends on the locale, which is determined at run time by the Python application. To satisfy these requirements, I have attempted to use the XSLT extensions feature of lxml. Before processing begins, I parse the XML document containing human-readable text. I then perform the transformation on the main XML document, passing an extension callback that is meant to read one of the attribute values of the extension XSLT element, and use the value to generate an XPath query. The goal for the extension element to evaluate to the result tree of the XPath query on the auxiliary document. Additionally, markup in the results tree from the query should be processed according to the template rules. To illustrate, consider this snippet: in_tree = etree.parse("document.xml") tpl_tree = etree.parse("template.xslt") strings_tree = etree.parse("aux_doc.xml") class StringsElement(XSLTExtension): def execute(self, context, self_node, input_node, output_parent): id_ = self_node.attrib["id"] string = strings_tree.xpath(generate_xpath_query(id_)) results = self.apply_templates(context, string) output_parent.append(results) xslt_exts = { ( "mynamespace", "string" ) : StringElement() } template = etree.XSLT(tpl_tree, extensions = xslt_exts) rslt_tree = trans(in_tree) rslt_tree.write(sys.stdout) Currently, the apply_templates call is failing, with the following error: TypeError: invalid argument type <type 'list'> I have been struggling with this issue and appreciate your help. Sincerely, Eric Levy

2 2

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) May 3, 2013

May 3, 2013

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Namespaced attributes losing their namespace
by Tyler Spivey May 1, 2013

May 1, 2013

In the below test script, I'm not getting the result I expect. Is this a bug in lxml? If not, what am I doing wrong? I'm using lxml 3.1.1 on win32, downloaded from PyPi. What I get: <root xmlns="url1"> <test attrib="value"/> </root> I expect attrib to not lose its namespace, as shown when I re-parse the output: >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'attrib': 'value'} If I swap the append and the attribute setting, things work: <test xmlns:ns0="url1" ns0:attrib="value"/> >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'{url1}attrib': 'value'} If this is intended behaviour, I'll have to see how to fit that into my design, since my functions return elements ready for appending. from lxml import etree nsmap = {None: 'url1'} root = etree.Element('{url1}root', nsmap=nsmap) test = etree.Element('{url1}test') test.attrib['{url1}attrib'] = 'value' root.append(test) print etree.tostring(root, encoding='utf-8', pretty_print=True)

2 6