Mailman 3 December 2017 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 11, 2023

April 11, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 10, 2023

April 10, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

docinfo.doctype don't return the original doctype
by Kim Grønborg Nielsen Dec. 28, 2017

Dec. 28, 2017

I’m using following code to extract DOCTYPE with python 2.7 and lxml 4.1.1: from lxml import tree from StringIO import StringIO if __name__ == '__main__': doc = etree.parse(StringIO('''<?xml version="1.0"?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration xmlns:log4j = "http://jakarta.apache.org/log4j/" debug="false"> <a>tasty</a> </log4j:configuration>''')) print "Type: {}\n".format(doc.docinfo.doctype) But it returns: Type: <!DOCTYPE configuration SYSTEM "log4j.dtd”> And not, as I expected: Type: <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> Is it a bug in lxml? Is there a workaround for getting what I’m expecting? =:-) Kim Grønborg Nielsen M kgn+lxml(a)network-it.dk

2 3

lxml.objectify with xml elements containing hyphens ('-')
by Grazhdani Jonian Dec. 27, 2017

Dec. 27, 2017

Hello, We have been using lxml.objectify for a while for our xml work in python and I really appreciate its neat way of converting xml Elements to python objects and vice-versa. That worked great until we needed to add support for an xml-based standard which extensively uses hyphens in its element names (Specifically the MDX from ASAM https://www.asam.net/standards/detail/mdx/wiki/ ). The resulting lxml.objectify objects have a lot of attributes with hyphens which makes their use in python very cumbersome, since using hyphens in variable names in forbidden in python and results in a syntax error. Googling pointed me to use getattr() / setattr() instead of the dot notation, which I have done, also in combination with xpath expressions to avoid ridiculous getattr() chains. It does do the job but the resulting code is a nightmare to work with and maintain (See example code as post scriptum below). I kindly ask those of you who have extensive experience with hyphened elements to provide me some guidance on what would be the best way to proceed from here. The alternatives I see in front of me are: 1 Convert all element names to use another character (e.g. underscore) instead of hyphen before objectify parsing, and then converting back before exporting. 2 Continue with getattr() / setattr() 3 give up objectify Obviously (and unfortunately) changing the standard is not an option. If you have any other approaches not listed above I would be very happy if you could share them with me! I am seriously considering giving alternative 1 a try and figured out I would need to convert both the schema and the xml to make objectify work. Any tips on doing that in a smart way would be also welcome. Happy holidays and thank you for your help! Jonian Grazhdani EMS Software Architect Powertrain Control System Development Scania CV AB SE-151 87 Södertälje, Sweden tel: +46 (0)8 553 53427 e-mail: jonian.grazhdani(a)scania.com<mailto:jonian.grazhdani@scania.com> P.S. Example code with two implementations: NAMESPACE = "http://www.asam.net/schema/MDX/r1.3" E = objectify.ElementMaker(annotate=False, namespace=NAMESPACE, nsmap={ None: NAMESPACE, 'xsd': 'http://www.w3.org/2001/XMLSchema', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}) # Implementation 1: dot notation which results in a syntax error self.sw_feature = E.SW-FEATURE( E.SHORT-NAME(self.short_name), E.CATEGORY("FCT"), E.SW-FEATURE-OWNED-ELEMENT-SETS( E.SW-FEATURE-OWNED-ELEMENT-SET( E.SW-FEATURE-ELEMENTS() ), E.SW-FEATURE-INTERFACES(), ID = self.id ) if len(self.owned_elements['SW-VARIABLE-REFS']) > 0: self.sw_feature.SW-FEATURE-OWNED-ELEMENT-SETS.SW-FEATURE-OWNED-ELEMENT-SET.SW-FEATURE-ELEMENTS.SW-VARIABLE-REFS = E.SW-VARIABLE-REFS() self.sw_feature.SW-FEATURE-OWNED-ELEMENT-SETS.SW-FEATURE-OWNED-ELEMENT-SET.SW-FEATURE-ELEMENTS.SW-VARIABLE-REFS = self.owned_elements['SW-VARIABLE-REFS'] # Implementation 2: getattr / setattr with the occasional xpath self.sw_feature = getattr(E,'SW-FEATURE')( getattr(E,'SHORT-NAME')(self.short_name), E.CATEGORY("FCT"), getattr(E,'SW-FEATURE-OWNED-ELEMENT-SETS')( getattr(E,'SW-FEATURE-OWNED-ELEMENT-SET')( getattr(E,'SW-FEATURE-ELEMENTS')() ) ), getattr(E,'SW-FEATURE-INTERFACES')(), ID = self.id ) mdx_sw_feature_elements = self.sw_feature.find("./{ns}SW-FEATURE-OWNED-ELEMENT-SETS/{ns}SW-FEATURE-OWNED-ELEMENT-SET/{ns}SW-FEATURE-ELEMENTS".format(ns="{" + NAMESPACE + "}")) if self.owned_elements['SW-VARIABLE-REFS']: setattr(mdx_sw_feature_elements, 'SW-VARIABLE-REFS', getattr(E, 'SW-VARIABLE-REFS')()) mdx_sw_refs = getattr(mdx_sw_feature_elements, 'SW-VARIABLE-REFS') #set the objects children (e.g. SW-VARIABLE-REFS.SW-VARIABLE-REF) setattr('SW-VARIABLE-REFS', 'SW-VARIABLE-REF', self.owned_elements['SW-VARIABLE-REFS'])

2 1

What is the difference between XML() and fromstring() in etree?
by Peng Yu Dec. 20, 2017

Dec. 20, 2017

I am trying to find out the difference between XML() and fromstring() in etree. But it seems that they are the same. Does anybody know if what is the main difference between them? Thanks. $ cat ./main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: from lxml import etree tree = etree.fromstring('<a>text</a>') print type(tree) tree = etree.XML('<a>text</a>') print type(tree) $ ./main.py <type 'lxml.etree._Element'> <type 'lxml.etree._Element'> Help on cython_function_or_method in module lxml.etree: XML(...) XML(text, parser=None, base_url=None) Parses an XML document or fragment from a string constant. Returns the root node (or the result returned by a parser target). This function can be used to embed "XML literals" in Python code, like in >>> root = XML("<root><test/></root>") >>> print(root.tag) root To override the parser with a different ``XMLParser`` you can pass it to the ``parser`` keyword argument. The ``base_url`` keyword argument allows to set the original base URL of the document to support relative Paths when looking up external entities (DTD, XInclude, ...). Help on cython_function_or_method in module lxml.etree: fromstring(...) fromstring(text, parser=None, base_url=None) Parses an XML document or fragment from a string. Returns the root node (or the result returned by a parser target). To override the default parser with a different parser you can pass it to the ``parser`` keyword argument. The ``base_url`` keyword argument allows to set the original base URL of the document to support relative Paths when looking up external entities (DTD, XInclude, ...). -- Regards, Peng

2 3

What is the difference between etree.XML and etree.HTML?
by Peng Yu Dec. 20, 2017

Dec. 20, 2017

Hi, In the following example, XML and HTML work equally well. Does anybody has an example showing when they will be different? Thanks. from lxml import etree tree = etree.XML('<foo><bar>abc</bar></foo>') tree = etree.HTML('<foo><bar>abc</bar></foo>') print type(tree) r = tree.xpath('//bar') print [x.tag for x in r] -- Regards, Peng

2 1

Which version of xpath does lxml support?
by Peng Yu Dec. 20, 2017

Dec. 20, 2017

Hi, I don't find this in the document. Maybe I miss it? Does lxml support xpath version 1 or 2? Thanks. -- Regards, Peng

3 2

How find() works for a node returned by xpath()?
by Peng Yu Dec. 18, 2017

Dec. 18, 2017

Hi, I am not able to find the 'rab' node using the following code. Could anybody help me understand why it is the case? $ cat ./main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: from lxml import etree import sys from StringIO import StringIO tree = etree.parse(StringIO('<foo><bar><rab>abc</rab></bar><mybar>xyz</mybar></foo>')) r = tree.xpath('/foo') print len(r) print r[0].find('bar').tag print r[0].find('rab') print r[0].find('mybar') print r[0].find('nobar') $ ./main.py 1 bar None <Element mybar at 0x1082416c8> None -- Regards, Peng

3 4

How use find() with an absolute path?
by Peng Yu Dec. 17, 2017

Dec. 17, 2017

I thought etree.find() was the same as etree.xpath()[0]. But find('/foo/bar') does not show any result. Could anybody know why it is the case? Thanks. $ cat main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: from lxml import etree from StringIO import StringIO tree = etree.parse(StringIO('<foo><bar>abc</bar></foo>')) print tree.find('bar') print tree.xpath('bar')[0] print tree.find('/foo/bar') print tree.xpath('/foo/bar')[0] $ ./main.py <Element bar at 0x10aff8a70> <Element bar at 0x10aff8a70> None <Element bar at 0x10aff8680> -- Regards, Peng

2 1

odd behaviour of defacultdictin lxml script
by Martin Mueller Dec. 15, 2017

Dec. 15, 2017

I’m puzzled about the behavior of defaultdict in an lxml script and wonder whether there is a bug, undefined behavior, or (most likely) that I don’t know what I’m doing. I work with linguistically annotated data, where every word has attributes, e.g. <w lemma=”love”, pos=”vvd”, reg=”lov’d”>lou’d</w>. There are errors in those data. To catch the most common errors I loop through the text and turn the attribute values into tuples, .e.g “love/vvd/loved/lou’d”. A review of them idetnfies a list of tuples that are always wrong regardless of context. So I correct those tuples, and create a dictionary where the key is the faulty tuple and the value a tuple with the corrected values. Now I loop through the text , look for occurrences of the faulty tuples and adjust the attribute values with the values of the correct one. So there is code that goes like this: If bad_tuple in tuplefixer: values = tuplefixer[ana_value] valuelist = values.split('/') if len(valuelist) == 2: lemmavalue = valuelist[0] posvalue = valuelist[1] w.set('lemma', lemmavalue) w.set('pos', posvalue) ..... This works., though there may be more elegant ways of doing it. But I want to create an additional step and keep track of how often a particular correction happens. For this I create a default dictionary whose key is a tuple of the corrected values and the value is the count that increments by one with every occurrence. So I add a line of code that goes like changedict[values] += 1 I’ve done this often and successfully, but in this script it doesn’t work. The line does not have its intended effect of adding an item to a default dictionary, and it prevents the correction of the tuple. I don’t know why. The only difference from other uses of this technique is that in this case the value of one dictionary becomes the key of a default dictionary. I have always assumed that adding something to a dictionary has no impact on the underlying data. Am I wrong on that point?

3 2

Re: [lxml] odd behaviour of defacultdictin lxml script
by Holger Joukl Dec. 15, 2017

Dec. 15, 2017

Martin, (taking this to the list as others might have better ideas) > I’ve been thinking about what you said about attributes. I know > their order is arbitrary. In my dictionary of values, the values are > a list, and I know that the lemma comes first and the POS tag second > etc. I could turn the list into a dictionary which would be more > explicit. If the list of values were long that would almost > certainly be the better way to go. Doing it as a list seems a little > simpler. Do you know whether there is a performance difference? On > a fast Mac, lxml is very fast. It seems that for just about > operation, the creation of the original tree is the most expensive > thing in terms of time, and if the document is longer than 100MB— > some 17th century theological tracts are like that—that can take a > minute. Looping through it after wards is very fast. With regard to performance you'll always need to measure. You might want to look at the timeit module for that. That said my first hunch would be to use the elements attrib dict (or a subset if not all element attributes are of interest for your comparisons) as the key for your lookup dictionaries. You can't do this directly as dictionaries aren't hashable (which is necessary for becoming dictionary keys). As Python doesn't have a built-in hashable "frozen-dict" I'd probably go for s.th. like this: >>> # I tend to use lxml.objectify for simplicity but you can just as well use etree >>> from lxml import etree, objectify >>> root = objectify.fromstring("""<root><w lemma="love" pos="vvd" reg="lov'd">lou'd</w></root>""") >>> print objectify.dump(root) # Note: objectify.dump is very readable but can be slow root = None [ObjectifiedElement] w = "lou'd" [StringElement] * lemma = 'love' * pos = 'vvd' * reg = "lov'd" >>> # with etree.fromstring()-construction you'd need to do s.th. like >>> # root[0], root.find('w'), root.xpath(...) to get at the w-element here >>> w = root.w >>> w.items() # list of attribute (key-value)-tuples [('lemma', 'love'), ('pos', 'vvd'), ('reg', "lov'd")] >>> attributes = frozenset(root.w.items()) # a hashable representation for our attributes >>> attributes2 = frozenset(reversed(root.w.items())) # just for demo purposes >>> attributes == attributes2 # order is irrelevant True >>> hash(attributes) == hash(attributes2) # order is irrelevant for hashing True >>> # attributes frozenset can be used as dict key >>> tuplefixer = {} >>> tuplefixer[attributes] = {'lemma': 'love', 'reg': "loved", 'pos': 'vvd'} >>> # Sidenote: frozenset can easily be converted back to a dict >>> dict(attributes) {'lemma': 'love', 'reg': "lov'd", 'pos': 'vvd'} >>> >>> # So you could do s.th. like... >>> bad_attributes = frozenset({'lemma': 'love', 'pos': 'vvd', 'reg': "lov'd"}) >>> tuplefixer = { ... bad_attributes: {'lemma': 'love', 'pos': 'vvd', 'reg': "loved"} ... } >>> >>> attributes = frozenset(w.attrib) >>> tuplefixer[attributes] {'lemma': 'love', 'pos': 'vvd', 'reg': 'loved'} >>> w.attrib.update(tuplefixer[attributes]) # update all attribute entries in one go >>> # note the fixed "reg" attribute >>> print objectify.dump(root) root = None [ObjectifiedElement] w = "lou'd" [StringElement] * lemma = 'love' * pos = 'vvd' * reg = 'loved' >>> print etree.tostring(root, pretty_print=True) <root> <w lemma="love" pos="vvd" reg="loved">lou'd</w> </root> >>> Hope this helps, H. Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart

1 0

Copyright symbol changed to HTML entity
by Lanam, Jeff Dec. 12, 2017

Dec. 12, 2017

I have a Python program that reads XML files and modifies the version attributes. Some of these files also have a copyright notice, with the copyright symbol (c). The lxml package turns these into the HTML entity &#169. Is there a way to prevent this? I tried using the resolve_entities parameter of the XMLParser function, but that has no effect. I've tried with both Python 2.7 and 3.6.3. The program below is for Python 3. # coding: utf-8 import os import glob import argparse from lxml import etree xParser = etree.XMLParser(strip_cdata=False, resolve_entities=False) etree.set_default_parser(xParser) someXML ='<node version="1.0.1"><copyright>Copyright (c) 2017 by me</copyright></node>' doc = etree.fromstring(someXML) print(someXML) print(etree.tostring(doc)) It prints out: <node version="1.0.1"><copyright>Copyright c 2017 by me</copyright></node> b'<node version="1.0.1"><copyright>Copyright © 2017 by me</copyright></node>' I have also posted this question to Stack Overflow. https://stackoverflow.com/questions/47779890/lxml-python-package-changes-co… Thanks for any suggestions. Jeff Lanam Software Designer NonStop Enterprise Division jeff.lanam(a)hpe.com +1 650 386 4703 Skype Palo Alto, CA hpe.com @jefflanam [HPE logo]<http://www.hpe.com/>

2 2