Mailman 3 December 2017 - lxml - The Python XML Toolkit

How find() works for a node returned by xpath()?
by Peng Yu Dec. 18, 2017

Dec. 18, 2017

Hi, I am not able to find the 'rab' node using the following code. Could anybody help me understand why it is the case? $ cat ./main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: from lxml import etree import sys from StringIO import StringIO tree = etree.parse(StringIO('<foo><bar><rab>abc</rab></bar><mybar>xyz</mybar></foo>')) r = tree.xpath('/foo') print len(r) print r[0].find('bar').tag print r[0].find('rab') print r[0].find('mybar') print r[0].find('nobar') $ ./main.py 1 bar None <Element mybar at 0x1082416c8> None -- Regards, Peng

3 4

How use find() with an absolute path?
by Peng Yu Dec. 17, 2017

Dec. 17, 2017

I thought etree.find() was the same as etree.xpath()[0]. But find('/foo/bar') does not show any result. Could anybody know why it is the case? Thanks. $ cat main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: from lxml import etree from StringIO import StringIO tree = etree.parse(StringIO('<foo><bar>abc</bar></foo>')) print tree.find('bar') print tree.xpath('bar')[0] print tree.find('/foo/bar') print tree.xpath('/foo/bar')[0] $ ./main.py <Element bar at 0x10aff8a70> <Element bar at 0x10aff8a70> None <Element bar at 0x10aff8680> -- Regards, Peng

2 1

odd behaviour of defacultdictin lxml script
by Martin Mueller Dec. 15, 2017

Dec. 15, 2017

I’m puzzled about the behavior of defaultdict in an lxml script and wonder whether there is a bug, undefined behavior, or (most likely) that I don’t know what I’m doing. I work with linguistically annotated data, where every word has attributes, e.g. <w lemma=”love”, pos=”vvd”, reg=”lov’d”>lou’d</w>. There are errors in those data. To catch the most common errors I loop through the text and turn the attribute values into tuples, .e.g “love/vvd/loved/lou’d”. A review of them idetnfies a list of tuples that are always wrong regardless of context. So I correct those tuples, and create a dictionary where the key is the faulty tuple and the value a tuple with the corrected values. Now I loop through the text , look for occurrences of the faulty tuples and adjust the attribute values with the values of the correct one. So there is code that goes like this: If bad_tuple in tuplefixer: values = tuplefixer[ana_value] valuelist = values.split('/') if len(valuelist) == 2: lemmavalue = valuelist[0] posvalue = valuelist[1] w.set('lemma', lemmavalue) w.set('pos', posvalue) ..... This works., though there may be more elegant ways of doing it. But I want to create an additional step and keep track of how often a particular correction happens. For this I create a default dictionary whose key is a tuple of the corrected values and the value is the count that increments by one with every occurrence. So I add a line of code that goes like changedict[values] += 1 I’ve done this often and successfully, but in this script it doesn’t work. The line does not have its intended effect of adding an item to a default dictionary, and it prevents the correction of the tuple. I don’t know why. The only difference from other uses of this technique is that in this case the value of one dictionary becomes the key of a default dictionary. I have always assumed that adding something to a dictionary has no impact on the underlying data. Am I wrong on that point?

3 2

Re: [lxml] odd behaviour of defacultdictin lxml script
by Holger Joukl Dec. 15, 2017

Dec. 15, 2017

Martin, (taking this to the list as others might have better ideas) > I’ve been thinking about what you said about attributes. I know > their order is arbitrary. In my dictionary of values, the values are > a list, and I know that the lemma comes first and the POS tag second > etc. I could turn the list into a dictionary which would be more > explicit. If the list of values were long that would almost > certainly be the better way to go. Doing it as a list seems a little > simpler. Do you know whether there is a performance difference? On > a fast Mac, lxml is very fast. It seems that for just about > operation, the creation of the original tree is the most expensive > thing in terms of time, and if the document is longer than 100MB— > some 17th century theological tracts are like that—that can take a > minute. Looping through it after wards is very fast. With regard to performance you'll always need to measure. You might want to look at the timeit module for that. That said my first hunch would be to use the elements attrib dict (or a subset if not all element attributes are of interest for your comparisons) as the key for your lookup dictionaries. You can't do this directly as dictionaries aren't hashable (which is necessary for becoming dictionary keys). As Python doesn't have a built-in hashable "frozen-dict" I'd probably go for s.th. like this: >>> # I tend to use lxml.objectify for simplicity but you can just as well use etree >>> from lxml import etree, objectify >>> root = objectify.fromstring("""<root><w lemma="love" pos="vvd" reg="lov'd">lou'd</w></root>""") >>> print objectify.dump(root) # Note: objectify.dump is very readable but can be slow root = None [ObjectifiedElement] w = "lou'd" [StringElement] * lemma = 'love' * pos = 'vvd' * reg = "lov'd" >>> # with etree.fromstring()-construction you'd need to do s.th. like >>> # root[0], root.find('w'), root.xpath(...) to get at the w-element here >>> w = root.w >>> w.items() # list of attribute (key-value)-tuples [('lemma', 'love'), ('pos', 'vvd'), ('reg', "lov'd")] >>> attributes = frozenset(root.w.items()) # a hashable representation for our attributes >>> attributes2 = frozenset(reversed(root.w.items())) # just for demo purposes >>> attributes == attributes2 # order is irrelevant True >>> hash(attributes) == hash(attributes2) # order is irrelevant for hashing True >>> # attributes frozenset can be used as dict key >>> tuplefixer = {} >>> tuplefixer[attributes] = {'lemma': 'love', 'reg': "loved", 'pos': 'vvd'} >>> # Sidenote: frozenset can easily be converted back to a dict >>> dict(attributes) {'lemma': 'love', 'reg': "lov'd", 'pos': 'vvd'} >>> >>> # So you could do s.th. like... >>> bad_attributes = frozenset({'lemma': 'love', 'pos': 'vvd', 'reg': "lov'd"}) >>> tuplefixer = { ... bad_attributes: {'lemma': 'love', 'pos': 'vvd', 'reg': "loved"} ... } >>> >>> attributes = frozenset(w.attrib) >>> tuplefixer[attributes] {'lemma': 'love', 'pos': 'vvd', 'reg': 'loved'} >>> w.attrib.update(tuplefixer[attributes]) # update all attribute entries in one go >>> # note the fixed "reg" attribute >>> print objectify.dump(root) root = None [ObjectifiedElement] w = "lou'd" [StringElement] * lemma = 'love' * pos = 'vvd' * reg = 'loved' >>> print etree.tostring(root, pretty_print=True) <root> <w lemma="love" pos="vvd" reg="loved">lou'd</w> </root> >>> Hope this helps, H. Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart

1 0

Copyright symbol changed to HTML entity
by Lanam, Jeff Dec. 12, 2017

Dec. 12, 2017

I have a Python program that reads XML files and modifies the version attributes. Some of these files also have a copyright notice, with the copyright symbol (c). The lxml package turns these into the HTML entity &#169. Is there a way to prevent this? I tried using the resolve_entities parameter of the XMLParser function, but that has no effect. I've tried with both Python 2.7 and 3.6.3. The program below is for Python 3. # coding: utf-8 import os import glob import argparse from lxml import etree xParser = etree.XMLParser(strip_cdata=False, resolve_entities=False) etree.set_default_parser(xParser) someXML ='<node version="1.0.1"><copyright>Copyright (c) 2017 by me</copyright></node>' doc = etree.fromstring(someXML) print(someXML) print(etree.tostring(doc)) It prints out: <node version="1.0.1"><copyright>Copyright c 2017 by me</copyright></node> b'<node version="1.0.1"><copyright>Copyright © 2017 by me</copyright></node>' I have also posted this question to Stack Overflow. https://stackoverflow.com/questions/47779890/lxml-python-package-changes-co… Thanks for any suggestions. Jeff Lanam Software Designer NonStop Enterprise Division jeff.lanam(a)hpe.com +1 650 386 4703 Skype Palo Alto, CA hpe.com @jefflanam [HPE logo]<http://www.hpe.com/>

2 2