Mailman 3 March 2025 - lxml - The Python XML Toolkit

a simple about tei.write(filename)
by Martin Mueller March 31, 2025

March 31, 2025

Following somebody’s helpful advice I have for years used this line of code to write out a file: tree.write(newname, encoding=tree.docinfo.encoding,, xml_declaration=True, pretty_print=True) When running a script in Pycharm, its AI suggested just “tree.write(newname)” . I used the simpler form for a document, and it seemed to make no difference to the structure or display of the document. It’s a lot less to type and certainly easier to remember. Does the specification of features ever make a difference? Martin Mueller Professor emeritus of English and Classics Northwestern University

1 0

Return type of text_content()
by tomi.belan＠gmail.com March 21, 2025

March 21, 2025

I noticed that the text_content() method of lxml.html elements returns a _ElementUnicodeResult, i.e. a 'smart' string. However, its getparent(), attrname are None, and is_tail, is_text, is_attribute are False. This is the case even if the element contains a single text node. The XPath "string()" used in text_content()'s implementation never returns an existing text node, but always a new string. Wouldn't it make more sense for text_content() to return a normal str? E.g. by adding smart_strings=False to _collect_string_content. I am not aware of any real issues caused by text_content() returning a 'smart' string -- for example, I don't think it can cause any memory leaks, because it doesn't seem to have a reference to the original document. But it still seems unexpected and perhaps unintentional. In theory this might be a breaking change, if anyone expects elem.text_content().getparent() to exist and return None. But https://lxml.de/lxmlhtml.html doesn't mention that text_content() returns a 'smart' string. 'Smart' strings are only documented at https://lxml.de/xpathxslt.html. Given lxml 6.0.0 is in the works, now seemed like a good time to suggest this change. Thanks for reading, and thank you for all your work on lxml. Tomi

3 3

XMLSyntaxError: Maximum entity amplification factor exceeded, see xmlCtxtSetMaxAmplification when parsing large DITA files with DTD validation enabled.
by 83lordko ___ March 20, 2025

March 20, 2025

I am getting following error when trying to parse large DITA/DITAMAP files with DTD validation enabled. Seems this is caused by xmlCtxtSetMaxAmplification setting of libxml2 (https://github.com/GNOME/libxml2/blob/master/NEWS) A new API function xmlCtxtSetMaxAmplification was added to allow parsing of files that would otherwise trigger the billion laughs protection. /** * xmlTextReaderSetMaxAmplification: * @reader: an XML reader * @maxAmpl: maximum amplification factor * * Set the maximum amplification factor. See xmlCtxtSetMaxAmplification. */ void xmlTextReaderSetMaxAmplification(xmlTextReaderPtr reader, unsigned maxAmpl) { if (reader == NULL) return; xmlCtxtSetMaxAmplification(reader->ctxt, maxAmpl); } Is it possible to avoid triggering the XMLSyntaxError in the LXML? The parser I am usinbg is constructed as below: dtd_xml_parser = et.XMLParser(dtd_validation=True, no_network=False, huge_tree=True)

1 0

Performance issues when using element.clear() in Python 3.x
by Noorulamry Daud March 4, 2025

March 4, 2025

Hello everyone, I've been cracking my head about this performance issue I'm having and I could use some help. At my work we have to parse extremely large XML files - 20GB and even larger. The basic algorithm is as follows: with open(file, "rb") as reader: context = etree.iterparse(reader, events=('start', 'end')) for ev, el in context: (processing) el.clear() In Python 2.7, the processing time for a 20GB XML file is approximately 40 minutes. In Python 3.13, it's 7 hours, more than ten times from Python 2. We went through a fine-toothed comb to find the reason why (there were minimal changes in the porting process), and out of desperation I commented out the el.clear() line, and apparently that is the reason - without it, performance in Python 3 matches with 2. Unfortunately when we tested this in a less well-endowed server, the program crashed due to running out of memory (it worked fine with Python 2). I tried substituting el.clear() with del el instead but it did not work - apparently there were still references somewhere, so the garbage collector didn't fire. Questions: 1. What is the difference between the Python 2 and Python 3's implementation of clear()? 2. Is there a way to solve this issue of performance penalty? I tried fast_iter, clearing the root element, re-assigning element to None, nothing works. Any help would be greatly appreciated. Regards,

4 9