Mailman 3 - lxml - The Python XML Toolkit

How to update the `lxml` view of a tree modified at the `libxml2` level?
by Dieter Maurer Aug. 8, 2022

Aug. 8, 2022

In `dm.xmlsec.binding` ("https://pypi.org/project/dm.xmlsec.binding/"), I integrate Aleksey Sanin's XML security library ("http://www.aleksey.com/xmlsec") with `lxml`, both `libxml2` based. Recently, one of its encryption/decryption tests started to fail. This test uses a tree of the form: <root> ... <to_be_encrypted>...</to_be_encrypted> ... </root> The encryption transforms it (directly at the `libxml2` level) to the form <root> ... <encrypted>...… [View More]

1 0

Re: Why does it fail cleaning GPX file?
by Holger.Joukl＠LBBW.de July 22, 2022

July 22, 2022

Hi, > I run this script to remove unneeded elements. > > For some reason, the input file is left as-is, when I try to get rid of the <metadata> > block; If works as expected when I ignore that element. > > Any idea why? The reason is that lxml (sensibly) uses fully qualified tag names in Clark notation (see http://www.jclark.com/xml/xmlns.htm) > > ========== INPUT.GPX > > <?xml version="1.0" encoding="UTF-8"?> > <gpx version="1.1" creator="… [View More]GPSBabel - http://www.acme.com" > xmlns="http://www.topografix.com/GPX/1/1" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> > <metadata> > <time>2022-07-21T20:29:48.309Z</time> > <bounds minlat="44.456597300" minlon="3.007453400" > maxlat="45.803699000" maxlon="5.251047400"/> > </metadata> > <wpt lat="45.042569200" lon="5.040802200"> > <name>way/4749044</name> > <cmt>landuse=cemetery</cmt> > <desc>landuse=cemetery</desc> > <link href="http://osm.org/browse/way/4749044"/> > </wpt> > </gpx> Here, your actual xml elements are e.g. "{http://www.topografix.com/GPX/1/1}cmt" (in Clark notation), not plain "cmt", since all elements without a namespace prefix due to this namespace declaration xmlns="http://www.topografix.com/GPX/1/1" belong to said namespace. So you must iter() over the proper fully-qualified names. Here's a slightly adapted version of your example code that hopefully shows what's going on and how to do that: # modified element cleaning sample xmldata = """<?xml version="1.0"?> <gpx version="1.1" creator="GPSBabel - http://www.acme.com" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <metadata> ┆<time>2022-07-21T20:29:48.309Z</time> ┆<bounds minlat="44.456597300" minlon="3.007453400" maxlat="45.803699000" maxlon="5.251047400"/> </metadata> <wpt lat="45.042569200" lon="5.040802200"> ┆<name>way/4749044</name> ┆<cmt>landuse=cemetery</cmt> ┆<desc>landuse=cemetery</desc> ┆<link href="http://osm.org/browse/way/4749044"/> </wpt> </gpx> """ import lxml.etree as et parser = et.XMLParser(remove_blank_text=True, strip_cdata=False) root = et.fromstring(xmldata, parser) # Show the fully qualified tag names (Clark-Notation) for el in root.iter(): print(el.tag) # Remove by fully-qualified Clark tag name or using a wildcard, if appropriate # (unqualified link will not get removed here since this doesn't exist - left in for # demo purposes) for el in root.iter('{http://www.topografix.com/GPX/1/1}cmt','{*}desc','link'): ┆parent = el.getparent() ┆parent.remove(el) print(et.tostring(root, pretty_print=True, encoding='unicode')) (I skipped the file reading/writing for my convenience; you might also want to look at the pathlib standard library module for you file/path name constructions, which is nice for handling such stuff, sometimes) HTH, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz. [View Less]

2 1

Why does it fail cleaning GPX file?
by Gilles July 21, 2022

July 21, 2022

Hello, I run this script to remove unneeded elements. For some reason, the input file is left as-is, when I try to get rid of the <metadata> block; If works as expected when I ignore that element. Any idea why? Thank you. ========== INPUT.GPX <?xml version="1.0" encoding="UTF-8"?> <gpx version="1.1" creator="GPSBabel - http://www.acme.com" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <metadata> &… [View More]

1 0

Tested value modified in validation error message
by Bob Kline July 16, 2022

July 16, 2022

Before I file a ticket, I wanted to check here to find out if this is a known problem (I searched the bug tracker, but didn't see any matches). Our customer is using a commercial XML editor and the editor's implementation of the DOM API sometimes garbles whitespace inside text nodes when serializing XML documents for the document's xml property. This causes schema validation errors when we send that serialization off to be validated against the document type's schema using lxml. The problem we … [View More]

4 10

Iterparse raises TypeError on attempt to clean up preceding siblings
by Parfait Gasana June 26, 2022

June 26, 2022

Hello, While following the iterparse / modifying the tree docs (https://lxml.de/4.5/parsing.html#modifying-the-tree) in attempt to clean up preceding siblings, code raises a TypeError: 'NoneType' object does not support item deletion when the XML contains a comment preceding the root element. Please advise on a workaround for this use case if this is not a potential bug. See below reproducible example. Thank you for your time and review. ```python from io import BytesIO import lxml.etree as … [View More]

3 2

tostring() returning possible invalid XML. Is this a bug?
by brian.bird＠trustpayments.com June 7, 2022

June 7, 2022

In more recent versions of lxml the tostring() method can return extra text after the closing tag of the node I've passed to it. So instead of returning b'<form action="action1">\n</form>\n' it returns b'<form action="action1">\n</form>\n</body>\n</html>\n' Here's a (python3) script along with two outputs, one from a machine running lxml 4.6.5 and one running 4.8.0. NOTE the output only changes if the DOCTYPE line is left in the "html" variable. import sys … [View More]from lxml import etree print("%-20s: %s" % ('Python', sys.version_info)) print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION)) html = """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <body> <form action="action1"> </form> </body> </html> """ parser = etree.XMLParser() doc = etree.fromstring(html, parser=parser) nodeList = doc.xpath("//form") print(etree.tostring(nodeList[0])) This is the output I would expect to see: Python : sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) lxml.etree : (4, 6, 5, 0) libxml used : (2, 9, 10) libxml compiled : (2, 9, 10) libxslt used : (1, 1, 34) libxslt compiled : (1, 1, 34) b'<form action="action1">\n</form>\n' #<------ Notice how the tostring() has returned the opening and closing <form> node (as I expected) This is the output I get when I upgrade: Python : sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) lxml.etree : (4, 8, 0, 0) libxml used : (2, 9, 12) libxml compiled : (2, 9, 12) libxslt used : (1, 1, 34) libxslt compiled : (1, 1, 34) b'<form action="action1">\n</form>\n</body>\n</html>\n' #<-------- Notice how the tostring() has returned extra text after the closing </form> tag Is this a bug? Or is this expected behaviour if the DOCTYPE is defined in the html passed to etree.fromstring()? ie. Is there a valid reason why tostring() might return an invalid XML byte string? I've seen the same behaviour in lxml 4.7.1 but I've not tried 4.9.0 as it's not in my repo yet. Any help appreciated! If this is a deliberate change I've got quite a lot of legacy code that will need updating to cope. [View Less]

3 2

Build problems on Python 3.11
by Charlie Clark June 1, 2022

June 1, 2022

Hi, while I don't see this locally, I'm getting problems on my CI with the Docker Image: ``` Compile failed: command '/usr/bin/gcc' failed with exit code 1 cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitw7u6s7rr.c -o tmp/xmlXPathInitw7u6s7rr.o cc tmp/xmlXPathInitw7u6s7rr.o -lxml2 -o a.out error: command '/usr/bin/gcc' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem … [View More]

2 3

Re: : Re: getparent() fails with "AttributeError: 'list' object has no attribute 'getparent' "
by Holger.Joukl＠LBBW.de May 30, 2022

May 30, 2022

> Through trial and error, it looks like xpath() returns an array, even if only one > element is found in the tree. Behold the power of the great docs: https://lxml.de/xpathxslt.html#xpath "[...] XPath return values The return value types of XPath evaluations vary, depending on the XPath expression used: - True or False, when the XPath expression has a boolean result - a float, when the XPath expression has a numeric result (integer or float) - a 'smart' string (as … [View More]

2 1

getparent() fails with "AttributeError: 'list' object has no attribute 'getparent' "
by codecomplete＠free.fr May 29, 2022

May 29, 2022

Hello, I'd like to find and replace an element in an HTML file. I can't figure out why getparent() doesn't work as expected: ============== import lxml.html from lxml.html import builder as E import lxml.etree as et import lxml.etree as et parser = et.HTMLParser(remove_blank_text=True,recover=True) with open("template.tmpl") as tempfile: template_tree = et.parse(tempfile, parser=parser) template_root = template_tree.getroot() #BAD FOO = template_root.find('.//FOO') FOO = template_root.… [View More]

1 2

[HTML] How to get text of attribute?
by Gilles May 28, 2022

May 28, 2022

Hello, My XPath skills being what they are… in an HTML file, I can't figure out how to grab the text of the second attribute in a meta element: ================== with open("input.html") as tempfile: parser = et.HTMLParser(remove_blank_text=True,recover=True) currfile_tree = et.parse(tempfile, parser=parser) currfile_root = currfile_tree.getroot() """ <head> <… [View More]

3 3