Mailman 3 March 2021 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 10 Apr '23

10 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 09 Apr '23

09 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 30 Aug '22

30 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Accessing variables from within an XSL file
by Adrian Bool 05 Apr '21

05 Apr '21

Hi, I'm trying to extract from data from XSL files on an IBM DataPower system. I can't change the XSLT files - and therefore need to work around their current structure to get the info I need. I'm doing this from within Python 3.7 using etree.__version__ 4.3.3. Here are my imports, a dummy XML object and a cutdown version of the XSL file from the IBM system: from lxml import etree from textwrap import dedent xml = etree.XML("""<root></root>""") xsl = etree.XML(dedent( """<?xml version="1.0"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dp="http://www.datapower.com/extensions" extension-element-prefixes="dp"> <xsl:template match="node()|@*"> <xsl:copy> <xsl:apply-templates select="node()|@*"/> </xsl:copy> <xsl:variable name="destination_url"> <xsl:value-of select="'https://my.domain.example.com/'" /> </xsl:variable>  <dp:set-variable name="'var://context/mpgw/Destination'" value="$destination_url" /> </xsl:template> </xsl:stylesheet> """)) My goal is to use lxml to process this file and extract the contents of the DataPower variable 'var://context/mpgw/Destination' - i.e. the contents of the xsl:variable $destination_url - in the above case ' https://my.domain.example.com/'. I've created an XSLT extension for the "dp:set-variable" element to try to capture this information: class MyExtElement(etree.XSLTExtension): def execute(self, context, self_node, input_node, output_parent): if self_node.tag == '{ http://www.datapower.com/extensions}set-variable': print(f"Setting variable of name {self_node.get('name')} to value {self_node.get('value')}") extensions = { ('http://www.datapower.com/extensions', 'set-variable') : MyExtElement() } result = etree.XSLT(xsl, extensions=extensions)(xml) print(f"Result: {result}") When this is being called; instead of the value of destination_url I'm getting the static "$destination_url" string instead - with the output of the above script being: Setting variable of name 'var://context/mpgw/Destination' to value $destination_url Result: <?xml version="1.0"?> <root/> I am rather suprised that the $destination_url xsl:variable was not being evaluated - does anyone know why this is the case? Given the above, I've been trying to find a way to access the $destination_url xsl:variable from within my XSLTExtension - but unfortunately with no luck. Some things I've tried: * Calling self.process_children(context) within the extension - this just results in an empty list. * I've used dir() to look through the context, self_node and input_node args to try to find somewhere the variables may be accessible but with no luck. * I've used deepcopy to clone self_node to get an xpath function on this argument and made some attempts at using xpath to access the xsl variable - all unsuccessful: node = deepcopy(self_node) node.xpath("""<xsl:value-of select="$destination_url">""") *** lxml.etree.XPathEvalError: Invalid expression node.xpath("""$destination_url""") *** lxml.etree.XPathEvalError: Undefined variable node.xpath("""xsl:$destination_url""") *** lxml.etree.XPathEvalError: Invalid expression node.xpath("""xsl:$destination_url""", namespaces={'xsl':'http://www.w3.org/1999/XSL/Transform'}) *** lxml.etree.XPathEvalError: Invalid expressio node.xpath("""$destination_url""", namespaces={'xsl':'http://www.w3.org/1999/XSL/Transform'}) *** lxml.etree.XPathEvalError: Undefined variable node.xpath("""/xsl:$destination_url""", namespaces={'xsl':'http://www.w3.org/1999/XSL/Transform'}) *** lxml.etree.XPathEvalError: Invalid expression node.xpath("""<xsl:value-of select="$destination_url">""", namespaces={'xsl':'http://www.w3.org/1999/XSL/Transform'}) *** lxml.etree.XPathEvalError: Invalid expression I'm lost now! Any suggestions greatly appreciated! Cheers, aid

4 5

A "Memory allocation failed" problem
by Martin Mueller 28 Mar '21

28 Mar '21

I reported the following problem some months ago, but didn’t get (or missed) an answer. Here it is again. I’m not sure whether it is in fact an lxml problem, but it only occurs in one particular lxml script. That script ran without problems for about a year, but suddenly stopped working. It will now run properly through any individual file, but when run it on a sequence of files it will fail after a dozen or so files with a “memory allocation failed” message. If you start from the file on which it failed it will process that file properly, but fail after processing some files with the same error message. I run Python 3.7 in a conda environment in Pycharm. The failure is produced by a function that sorts attributes alphabetically and indents a TEI XML file in which every token is wrapped in a <w> element that contains between three and eight attributes. The files get edited a lot. We keep the attributes sorted to make it easier to recognize substantive changes or additions. def sort_and_indent(elem, level: int = 0): attrib = elem.attrib if len(attrib) > 1: attributes = sorted(attrib.items()) attrib.clear() attrib.update(attributes) i = "\n" + " " * level if len(elem): if not elem.text or not elem.text.strip(): elem.text = i + " " if not elem.tail or not elem.tail.strip(): elem.tail = i for elem in elem: sort_and_indent(elem, level + 1) if not elem.tail or not elem.tail.strip(): elem.tail = i else: if level and (not elem.tail or not elem.tail.strip()): elem.tail = i When the function fails it produces this error message: /Users/martinmueller/.conda/envs/earlyprintprocessing/bin/python /Users/martinmueller/Dropbox/earlyprintprocessing/rewriteree.py Traceback (most recent call last): File "/Users/martinmueller/Dropbox/earlyprintprocessing/rewriteree.py", line 71, in <module> do_etree(filename, item, counter) File "/Users/martinmueller/Dropbox/earlyprintprocessing/rewriteree.py", line 49, in do_etree tree = etree.parse(filename, parser) File "src/lxml/etree.pyx", line 3435, in lxml.etree.parse File "src/lxml/parser.pxi", line 1840, in lxml.etree._parseDocument File "src/lxml/parser.pxi", line 1866, in lxml.etree._parseDocumentFromURL File "src/lxml/parser.pxi", line 1770, in lxml.etree._parseDocFromFile File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult File "src/lxml/parser.pxi", line 640, in lxml.etree._raiseParseError File "/users/martinmueller/dropbox/eebochron/1470-1600/159/a/159-adp-A12229.xml", line 10137 lxml.etree.XMLSyntaxError: Memory allocation failed, line 10137, column 24 The error message appears to be generated by lxml, but it may not be an lxml problem. I checked memory usage on the Activity Monitor of my Mac, has 64GB of memory. Memory usage by Python goes beyond 2GB, but the point of failure doesn’t seem to be related to the memory usage that is reported: it keeps running at well over 2 GB in one batch of files, but in another run it fails at well below 2GB. I cannot associate the onset of this problem with any particular event. I thought it could have something to do with a Pycharm update, but I just ran the script outside of Pycharm with Python3.9 and lxml 4.6.2. I got the same error. In running the script twice from the first file in a batch, I noticed that it failed at exactly the same point in the same file. But it cannot be a function of that file because if you run the program starting with that file it will process it properly. It does appear, however, that something cumulative is going on: in moving from one file to the next, the script does not start from scratch, but keeps or fails to clear some memory that causes a failure when a trigger point is reached. I’d be very grateful for any help of advice on where to look for it. Martin Mueller Professor emeritus of English and Classics Northwestern University

2 2

How to best deal with HTML entities in an XML file?
by Jens Tröger 27 Mar '21

27 Mar '21

Hello, I received a bunch of XML files that contain HTML entities (so far, I’ve seen only   used). I can’t parse these files with an XML parser because of these HTML entities: >>> parser = lxml.etree.XMLParser(huge_tree=True, remove_comments=True, schema=None, dtd_validation=False) >>> xml = lxml.etree.parse("test.xml", parser) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "src/lxml/etree.pyx", line 3521, in lxml.etree.parse File "src/lxml/parser.pxi", line 1859, in lxml.etree._parseDocument File "src/lxml/parser.pxi", line 1885, in lxml.etree._parseDocumentFromURL File "src/lxml/parser.pxi", line 1789, in lxml.etree._parseDocFromFile File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError File "test.xml", line 66 lxml.etree.XMLSyntaxError: Entity 'nbsp' not defined, line 66, column 25 The `resolve_entities` parameter for XMLParser unfortunately doesn’t seem to resolve HTML entities. If I parse the file using an HTMLParser it works: >>> parser = lxml.etree.HTMLParser(huge_tree=True, remove_comments=True) >>> xml = lxml.etree.parse("test.xml", parser) >>> xml <lxml.etree._ElementTree object at 0x10589baa0> but then the upper/lower case of all tags is lost because HTML is case-insensitive (XML is not) and it seems that the HTML parser turns all tag names to lower case: >>> xml.getroot().find("body/*") <Element docxml at 0x1059cb730> This should be a `DocXML` tag name. Now my original XML file is broken and fails schema validation… So, what now? I feel very hesitant to treat the original XML file as a string and replace HTML entities (except & < >) on a string level. I think a better approach would be to make the XML parser aware of HTML entities but that may be a libxml2 issue rather than lxml? (Haven’t looked at the source yet.) Would you have any other recommendations? How else could I work with this issue? Much thanks! Jens -- Jens Tröger https://savage.light-speed.de/

2 2

Re: Using DTD ENTITY with lxml
by Holger.Joukl＠LBBW.de 26 Mar '21

26 Mar '21

Hi, I have created the following example files to test this 2-base.xsd <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE schema [ <!ENTITY lowalpha "a-z"> <!ENTITY hialpha "A-Z"> <!ENTITY alpha "&lowalpha;&hialpha;"> <!ENTITY digit "0-9"> <!ENTITY uword "([&digit;]{1,4}|[1-5][&digit;]{4}|6[0-4][&digit;]{3}|65[0-4][&digit;]{2}|655[0-2][&digit;]|6553[0-5])"> <!ENTITY Port ":&uword;"> ]> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="urn:paul:types2" targetNamespace="urn:paul:types2" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:simpleType name="PortType"> <xs:restriction base="xs:string"> <xs:pattern value="&Port;"/> </xs:restriction> </xs:simpleType> </xs:schema> 2-main.xsd <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="urn:paul:main2" xmlns:t2="urn:paul:types2" targetNamespace="urn:paul:main2" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:import namespace="urn:paul:types2" schemaLocation="2-base.xsd"/> <xs:element name="Port" type="t2:PortType"/> </xs:schema> 2.xml <?xml version="1.0" encoding="UTF-8"?> <Port xmlns="urn:paul:main2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:paul:main2 2-main.xsd">:10100</Port> 2.xml validates OK in XMLspy and if the value is changed to :101001 for example, a match error is given so we know the pattern is progressing through the xs:import etc 2.xml does not validate when used against lxml from lxml import etree with open('2-main.xsd', 'r') as schema_file: my_schema=etree.XMLSchema(etree.parse(schema_file)) filename='2.xml' with open(filename) as file: my_xml=etree.parse(file) my_schema.assertValid(my_xml) This gives the following error Traceback (most recent call last): File "G:\lxml-test\test1b.py", line 11, in <module> my_schema.assertValid(my_xml) File "src\lxml\etree.pyx", line 3623, in lxml.etree._Validator.assertValid lxml.etree.DocumentInvalid: Element '{urn:paul:main1}Port': [facet 'pattern'] The value ':10100' is not accepted by the pattern '[]+'., line 2 Is this something that 'should' be supported by lxml? If so, am I missing something in the validation? Should work and I can't reproduce your problem i.e. your given sample files work for me just fine (lxml 4.6.2, libxml2 2, 9, 7, libxslt 1, 1, 29). But I think you're actually using different data according to this output: Element '{urn:paul:main1}Port' Note the differing namespace as opposed to the 2.xml sample. Maybe some typo / difference in your runs with XMLSpy vs lxml is causing the headaches? Sidenote (but you probably know this): No need to open() files yourself, lxml.etree does it just fine: # validate.py from lxml import etree schema_tree = etree.parse('2-main.xsd') my_schema = etree.XMLSchema(schema_tree) my_xml = etree.parse('2.xml') result = my_schema.assertValid(my_xml) Best regards, Holger [cid:image001_e3261df1-efb3-4c64-a4de-9b63c5e52cfb.png]<https://www.lbbw.de/> [cid:image002_15f56f09-b8fe-4ddc-a247-e3cf3d25c71f.png]<https://twitter.com/lbbw> [cid:image003_e01dca75-944e-4ea3-8463-c9c061e5b36b.png] <https://www.linkedin.com/company/lbbw> [cid:image004_5642e2b4-e190-4f0c-a274-7ceb59fe913c.png] <https://www.xing.com/company/lbbw> [cid:image005_53d53024-7202-4d60-8d95-dc0f361239ca.png] <https://www.facebook.com/LBBW.Stuttgart/> [cid:image006_9c0b6462-8935-4b9c-9fe3-0caa90d8623b.png] <https://www.youtube.com/user/LBBWDirekt> [cid:image007_9a7ebd18-8e4e-4c3e-931a-ecc4bf575ace.png] <https://www.instagram.com/lbbw_karriere/> Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

2 1

Using DTD ENTITY with lxml
by Paul Higgs 24 Mar '21

24 Mar '21

Hello As we look to more and more ci verification of the XML Schemas that we develop for MPEG DASH, we have started to leverage the flexibility of DTD ENTITY definitions when creating XML schema pattern facets. Out Github uses lxml to verify pull requests, but it seems that the defined entities are not being used. I have created the following example files to test this 2-base.xsd <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE schema [ <!ENTITY lowalpha "a-z"> <!ENTITY hialpha "A-Z"> <!ENTITY alpha "&lowalpha;&hialpha;"> <!ENTITY digit "0-9"> <!ENTITY uword "([&digit;]{1,4}|[1-5][&digit;]{4}|6[0-4][&digit;]{3}|65[0-4][&digit;]{2}|655[0-2][&digit;]|6553[0-5])"> <!ENTITY Port ":&uword;"> ]> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="urn:paul:types2" targetNamespace="urn:paul:types2" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:simpleType name="PortType"> <xs:restriction base="xs:string"> <xs:pattern value="&Port;"/> </xs:restriction> </xs:simpleType> </xs:schema> 2-main.xsd <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="urn:paul:main2" xmlns:t2="urn:paul:types2" targetNamespace="urn:paul:main2" elementFormDefault="qualified" attributeFormDefault="unqualified"> <xs:import namespace="urn:paul:types2" schemaLocation="2-base.xsd"/> <xs:element name="Port" type="t2:PortType"/> </xs:schema> 2.xml <?xml version="1.0" encoding="UTF-8"?> <Port xmlns="urn:paul:main2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:paul:main2 2-main.xsd">:10100</Port> 2.xml validates OK in XMLspy and if the value is changed to :101001 for example, a match error is given so we know the pattern is progressing through the xs:import etc 2.xml does not validate when used against lxml from lxml import etree with open('2-main.xsd', 'r') as schema_file: my_schema=etree.XMLSchema(etree.parse(schema_file)) filename='2.xml' with open(filename) as file: my_xml=etree.parse(file) my_schema.assertValid(my_xml) This gives the following error Traceback (most recent call last): File "G:\lxml-test\test1b.py", line 11, in <module> my_schema.assertValid(my_xml) File "src\lxml\etree.pyx", line 3623, in lxml.etree._Validator.assertValid lxml.etree.DocumentInvalid: Element '{urn:paul:main1}Port': [facet 'pattern'] The value ':10100' is not accepted by the pattern '[]+'., line 2 Is this something that 'should' be supported by lxml? If so, am I missing something in the validation? Appreciate any hints, guidance etc Paul

1 0