Mailman 3 - lxml - The Python XML Toolkit

XML namespaces are not propagated over from the ancestor elements when using find* methods
by Matthew Ouyang Jan. 15, 2025

Jan. 15, 2025

I have a XML document that has namespaces in it. I want to use the find* methods to select elements but it's looks like it is not possible without specifying namespaces explicitly to every call. Is this true? Is this by design? This seems very burdensome to do so when namespaces are included in the XML document. It would be really nice if the namespaces in the XML document could be considered. xml = b"""<?xml version="1.0" encoding="UTF-8"?><root xmlns="urn:defaultnamespace"> <Timestamp>2025-01-09T17:46:08.766Z</Timestamp> <namespaced xmlns:ns="urn:subns"> <rootdefault>text</rootdefault> <ns:element>element</ns:element> </namespaced></root>"""from lxml import etreexml_doc = etree.fromstring(xml)print(xml_doc)print("namespaces: " + str(xml_doc.nsmap))print(f"find root without namespace {xml_doc.find(".")}")print(f"find root with namespace {xml_doc.find(".", namespaces = {**xml_doc.nsmap})}")print(f"find Timestamp without namespace {xml_doc.find("./Timestamp")}")print(f"find Timestamp with namespace {xml_doc.find("./Timestamp", namespaces = {**xml_doc.nsmap})}")print(f"find namespaced without namespace {xml_doc.find("./namespaced")}")print(f"find namespaced with namespace {xml_doc.find("./namespaced", namespaces = {**xml_doc.nsmap})}")print(f"find element with namespace {xml_doc.find("./namespaced/rootdefault")}")print(f"find rootdefault with namespace {xml_doc.find("./namespaced/rootdefault", namespaces = {**xml_doc.nsmap})}")print(f"find namespaced no namespace {xml_doc.find("./namespaced/element")}")try: print(f"find namespaced default namespace only {xml_doc.find("./namespaced/ns:element", namespaces = {None: "urn:defaultnamespace"})}")except SyntaxError as e: print(f"find namespaced default namespace only error {e}")print(f"find namespaced ns namespace only {xml_doc.find("./namespaced/ns:element", namespaces = {"ns": "urn:subns"})}")print(f"find namespaced with full namespace {xml_doc.find("./namespaced/ns:element", namespaces = {None: "urn:defaultnamespace", "ns": "urn:subns"})}")

3 3

Appending XSLT to XML
by Schimon Jehudah Jan. 15, 2025

Jan. 15, 2025

Greetings! I would want to know whether there is a function to append XSLT stylesheet to XML files. I have resorted to converting XML data to string, in order to append a reference to an XSLT styleshett. See https://git.xmpp-it.net/sch/Rivista/src/branch/main/rivista/xml/xslt.py Please advise. Kind regards, Schimon

3 4

Write etree to multiple files using xinclude
by Trevor Clarke Dec. 4, 2024

Dec. 4, 2024

I've got a fairly complex xml config setup with multiple files xincluded. I'm trying to write a tool that allows me to replace certain values programmatically and write the modified file. I lxml.etree.parse() the file then xinclude(). I replace the appropriate text values and want to write the tree back to the filesystem using the original files. I figured out that I can locate the file containing the modified node with something like: ``` cur, top = node, node while cur.base == node.base: top, cur = cur, cur.getparent() with open(top.base, "wb") as f: f.write(et.tostring(top)) ``` However, any elements that have a different base are still written to that same file. Is there a way I can either write a whole tree such that anything with a different base gets written to the appropriate file and an xinclude added for that file? The only other solution I can think of, is to find any node with a base attribute, replace with with an xinclude and add the node to a list of items to write.

1 0

Optimizing lxml for Handling Large XML Files: Tips and Experiences
by Lily Parker Oct. 30, 2024

Oct. 30, 2024

I've been using lxml to process large XML files recently, and I'm looking for ways to optimize performance. Specifically, I'm trying to filter specific nodes from large datasets and manage memory usage more effectively. I'd appreciate any tips or best practices from your experiences. Are there any techniques you use to enhance performance, or potential pitfalls I should watch out for? Thanks in advance for your insights!

2 1

GitHub lxml build scripts - helpful to others?
by abepolk＠gmail.com Oct. 8, 2024

Oct. 8, 2024

Hi all, I was interested in learning more about lxml. So, I wrote a couple of shell scripts that build lxml statically, and non-statically, and posted them on GitHub. Specifically, they are (I believe) the minimal code required to build lxml from source remotely on a freshly created Google Cloud Platform Cloud Compute VM, on a Debian Linux system. I used the documentation some. However, I had to add -fPIC in a static build command, and set the environmental variable LD_LIBRARY_PATH to use lxml when built non-statically. I also used -i and build_ext (instead of build) when building lxml statically, and this was not in the docs. My question is: would any of this help other lxml users and/or be worth putting in the documentation? Either way is fine - let me know! I'm happy to write up a PR for the docs if that's helpful too. See https://github.com/abepolk/lxml-scripts. Best, Abe Polk

1 0

Broken EXSLT link in docs
by Jens Tröger Sept. 29, 2024

Sept. 29, 2024

Hello, I think the EXSLT link here: https://lxml.de/xpathxslt.html#regular-expressions-in-xpath or source here: https://github.com/lxml/lxml/blob/9818374770aedc96f8f1e77943f45dea8e7fb4a8/… should change from http://www.exslt.org/ to https://exslt.github.io/ or some other valid URL. Happy to provide a PR but the change seems trivial enough… Cheers, Jens -- Jens Tröger https://savage.light-speed.de/

2 1

problem running self tests
by wiz＠gatalith.at Aug. 12, 2024

Aug. 12, 2024

Hi! I'm trying to run the self tests for the pkgsrc package for lxml, currently at 5.3.0. lxml itself seems to work fine AFAICT, but I can't get the self tests to runs. I tried following the documented method from https://lxml.de/build.html but I get: ``` # python3.12 test.py Traceback (most recent call last): File "/scratch/textproc/py-lxml/work/lxml-5.3.0/test.py", line 620, in <module> exitcode = main(sys.argv) ^^^^^^^^^^^^^^ File "/scratch/textproc/py-lxml/work/lxml-5.3.0/test.py", line 557, in main test_cases = get_test_cases(test_files, cfg, cov=cov) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/scratch/textproc/py-lxml/work/lxml-5.3.0/test.py", line 260, in get_test_cases module = import_module(file, cfg, cov=cov) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/scratch/textproc/py-lxml/work/lxml-5.3.0/test.py", line 201, in import_module mod = __import__(modname) ^^^^^^^^^^^^^^^^^^^ File "/scratch/textproc/py-lxml/work/lxml-5.3.0/src/lxml/html/__init__.py", line 49, in <module> from .. import etree ImportError: cannot import name 'etree' from 'lxml' (/scratch/textproc/py-lxml/work/lxml-5.3.0/src/lxml/__init__.py) ``` so I guessed the PYTHONPATH was not set correctly and tried that: ``` # PYTHONPATH=build/lib.netbsd-10.99.11-amd64-cpython-312/ python3.12 test.py ``` with the same result. I can import lxml.etree this way: ``` # PYTHONPATH=build/lib.netbsd-10.99.11-amd64-cpython-312/lxml/etree.so python3.12 -c "import lxml.etree" Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'lxml' # PYTHONPATH=build/lib.netbsd-10.99.11-amd64-cpython-312 python3.12 -c "import lxml.etree" # ``` so I guess the test script doesn't honor it. How can I run the tests? Thanks, Thomas

2 1

Assistance Needed: Python Code Crashes After Multiple Iterations When Extracting HTML Elements Caixa de entrada
by Carlos Eduardo de Schuller Banjar July 29, 2024

July 29, 2024

Hello, everyone! I am encountering an issue with my Python code that opens multiple HTML pages and extracts elements from a specific class, if they exist. The code runs fine for a few iterations but crashes after enough iterations, always on the same HTML page. Interestingly, if I process this page individually, the code works without any problems. This issue has been bothering me for some time, and I have tried many approaches. When debugging, I noticed that the code always crashes within the *lxml* package, which is used by *requests_html*. This could be a bug in the library, but what could explain the crash only after several iterations? <https://mail.google.com/mail/u/0?ui=2&ik=50de717578&attid=0.1&permmsgid=msg…> <https://mail.google.com/mail/u/0?ui=2&ik=50de717578&attid=0.1&permmsgid=msg…> A few more information: Python : sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) lxml.etree : (5, 2, 2, 0) libxml used : (2, 12, 6) libxml compiled : (2, 12, 6) libxslt used : (1, 1, 39) libxslt compiled : (1, 1, 39) I'm attaching the code I mentioned. You can see that the code is quite simple. import pandas as pd import requests_html def main(): df_links = pd.read_csv('./links.csv') session = requests_html.HTMLSession() for i in range(0, len(df_links.index)): url = df_links.iloc[i]['hyperlink'] print(f"[{i}/{len(df_links.index)}]: {url}", flush=True) try: response = session.get(url) if response.status_code == 200: response_html = response.html dateList = response_html.find('relative-time') except Exception as e: print(f"Something went wrong: {e}", flush=True) if __name__ == "__main__": main() The crash always happens at the following line: value = etree.fromstring(html, parser, **kw) in the function document_fromstring in *lxml/html/__init__.py*. Thank you for your time and help. Best regards, Kadu

1 0

Unable to obtain multiple validation errors against schema from a single xml file
by Claudio Pelazza July 5, 2024

July 5, 2024

Dear all, Hope this email finds you well!! I am trying to obtain multiple error messages from validating a test xml file, which I know in advance it has two or more different errors. However, when attempting to read lxml's log as suggested in the docs, I just get only one (the one which fails first). I'm a bit 'rusty' in python (no pun intended) but the few comments I find around the Web (e.g. Stack Overflow, etc.), even though not super explicit, do suggest it is possible.... am I doing anything wrong, or did I miss anything? (Chances are I did!!) Here is a code snippet (trying to achieve that in Databricks/Spark, lxml version is 5.2.2) from lxml import etree def validate_xml(xml_file, xsd_file): # Parse the XML and XSD files xml_doc = etree.parse(xml_file) with open(xsd_file, 'r') as f: xsd_doc = etree.XML(bytes(f.read(), 'utf-8')) # Create an XMLSchema object schema = etree.XMLSchema(xsd_doc) err_log = "" # Validate the XML document against the schema #is_valid = schema.validate(xml_doc) try: schema.assertValid(xml_doc) except etree.DocumentInvalid as err: err_log = str([error for error in schema.error_log]) #err_log = str(err) return err_log or "Valid" I know error_log is not iterable, but still, I was hoping I would find all the error messages by printing the entire log....what am I missing? isn't this possible?? (is this a SAX type parser only??) Also, looks like I cannot attach files in here?? Anyways, you can find example xml & xsd used at the end of this message. For the kind souls out there: Any hint/ suggestion/ example would be much appreciated!!! Thank you so much in advance!!! Claudio P. ------------------------------------------------- --- xml & xsd follows: ---Schema--- <?xml version="1.0" encoding="utf-8" ?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="BusinessCard"> <xsd:complexType> <xsd:sequence> <xsd:element name="Name" type="xsd:string"/> <xsd:element name="phone" maxOccurs="unbounded"> <xsd:complexType mixed="true"> <xsd:attribute name="type" use="required"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="mobile"/> <xsd:enumeration value="fax"/> <xsd:enumeration value="work"/> <xsd:enumeration value="home"/> </xsd:restriction> </xsd:simpleType> </xsd:attribute> </xsd:complexType> </xsd:element> <xsd:element name="email" type="xsd:string" minOccurs="0" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> ---------------------------------------------------------- ---xml--- <?xml version="1.0"?> <BusinessCard> <phone type="mobil">(415) 555-4567</phone> <phone type="work">(800) 555-9876</phone> <phone type="fax">(510) 555-1234</phone> <mail>joe(a)joe.com</mail> </BusinessCard>

1 0

Consider keeping manylinux1 wheels for Python 3.6
by Miro Hrončok May 16, 2024

May 16, 2024

Hello, I've noticed that lxml 5.1+ upgraded the manylinux wheels to a newer tag. The default ensurpip-bundled pip version in Python 3.6 does not support newer manylinuxes, hence it is likely that many CI systems that still test 3.6 now attempt to build lxml from sources. Since 5.2, this also fails with the old pip due to the old bundled pytoml, as indicated in a previous thread on this list. $ python3.6 -m venv venv3.6 $ venv3.6/bin/pip list Package Version ---------- ------- pip 18.1 setuptools 40.6.2 5.0.2 has a manylinux1 wheel: $ venv3.6/bin/pip install lxml==5.0.2 ... lxml-5.0.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl 5.1.0 builds from source but uses setup.py and works (with devel deps): $ venv3.6/bin/pip install lxml==5.1.0 ... lxml-5.1.0.tar.gz Running setup.py install for lxml ... 5.2.1 builds from source and will outright blow up when parsing pyproject.toml: $ venv3.6/bin/pip install lxml==5.2.1 ... lxml-5.1.0.tar.gz ... pip._vendor.pytoml.core.TomlError: /tmp/.../lxml/pyproject.toml(40, 1): msg If support for Python 3.6 is still desired, would it maybe make sense to keep building and uploading manylinux1 wheels to make it easier? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 5