Mailman 3 - lxml - The Python XML Toolkit

Unable to obtain multiple validation errors against schema from a single xml file
by Claudio Pelazza 05 Jul '24

05 Jul '24

Dear all, Hope this email finds you well!! I am trying to obtain multiple error messages from validating a test xml file, which I know in advance it has two or more different errors. However, when attempting to read lxml's log as suggested in the docs, I just get only one (the one which fails first). I'm a bit 'rusty' in python (no pun intended) but the few comments I find around the Web (e.g. Stack Overflow, etc.), even though not super explicit, do suggest it is possible.... am I doing anything wrong, or did I miss anything? (Chances are I did!!) Here is a code snippet (trying to achieve that in Databricks/Spark, lxml version is 5.2.2) from lxml import etree def validate_xml(xml_file, xsd_file): # Parse the XML and XSD files xml_doc = etree.parse(xml_file) with open(xsd_file, 'r') as f: xsd_doc = etree.XML(bytes(f.read(), 'utf-8')) # Create an XMLSchema object schema = etree.XMLSchema(xsd_doc) err_log = "" # Validate the XML document against the schema #is_valid = schema.validate(xml_doc) try: schema.assertValid(xml_doc) except etree.DocumentInvalid as err: err_log = str([error for error in schema.error_log]) #err_log = str(err) return err_log or "Valid" I know error_log is not iterable, but still, I was hoping I would find all the error messages by printing the entire log....what am I missing? isn't this possible?? (is this a SAX type parser only??) Also, looks like I cannot attach files in here?? Anyways, you can find example xml & xsd used at the end of this message. For the kind souls out there: Any hint/ suggestion/ example would be much appreciated!!! Thank you so much in advance!!! Claudio P. ------------------------------------------------- --- xml & xsd follows: ---Schema--- <?xml version="1.0" encoding="utf-8" ?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="BusinessCard"> <xsd:complexType> <xsd:sequence> <xsd:element name="Name" type="xsd:string"/> <xsd:element name="phone" maxOccurs="unbounded"> <xsd:complexType mixed="true"> <xsd:attribute name="type" use="required"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="mobile"/> <xsd:enumeration value="fax"/> <xsd:enumeration value="work"/> <xsd:enumeration value="home"/> </xsd:restriction> </xsd:simpleType> </xsd:attribute> </xsd:complexType> </xsd:element> <xsd:element name="email" type="xsd:string" minOccurs="0" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> ---------------------------------------------------------- ---xml--- <?xml version="1.0"?> <BusinessCard> <phone type="mobil">(415) 555-4567</phone> <phone type="work">(800) 555-9876</phone> <phone type="fax">(510) 555-1234</phone> <mail>joe(a)joe.com</mail> </BusinessCard>

1 0

Consider keeping manylinux1 wheels for Python 3.6
by Miro Hrončok 16 May '24

16 May '24

Hello, I've noticed that lxml 5.1+ upgraded the manylinux wheels to a newer tag. The default ensurpip-bundled pip version in Python 3.6 does not support newer manylinuxes, hence it is likely that many CI systems that still test 3.6 now attempt to build lxml from sources. Since 5.2, this also fails with the old pip due to the old bundled pytoml, as indicated in a previous thread on this list. $ python3.6 -m venv venv3.6 $ venv3.6/bin/pip list Package Version ---------- ------- pip 18.1 setuptools 40.6.2 5.0.2 has a manylinux1 wheel: $ venv3.6/bin/pip install lxml==5.0.2 ... lxml-5.0.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl 5.1.0 builds from source but uses setup.py and works (with devel deps): $ venv3.6/bin/pip install lxml==5.1.0 ... lxml-5.1.0.tar.gz Running setup.py install for lxml ... 5.2.1 builds from source and will outright blow up when parsing pyproject.toml: $ venv3.6/bin/pip install lxml==5.2.1 ... lxml-5.1.0.tar.gz ... pip._vendor.pytoml.core.TomlError: /tmp/.../lxml/pyproject.toml(40, 1): msg If support for Python 3.6 is still desired, would it maybe make sense to keep building and uploading manylinux1 wheels to make it easier? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 5

Building on Windows
by Gertjan Klein 06 May '24

06 May '24

Hi, I'd like to try a tiny change to the CDATA class. In order to try, I have to be able to build lxml. Unfortunately, on Windows. I've downloaded Visual Studio 2019 CE. I created a (Python 3.12) virtual environment, where I installed Cython (latest version). I cloned lxml sources from GitHub. I then opened a "Developer command prompt for VS 2019", activated the virtual environment, and typed: (.venv) C:\Temp\lxml\lxml>python setup.py build_ext -i --with-cython --static-deps This downloads the dependencies like libxml2 etc.; this goes without problems. Then compilation starts, and gives errors: [...] Creating library build\temp.win32-cpython-312\Release\src\lxml\etree.cp312-win_amd64.lib and object build\temp.win32-cpython-312\Release\src\lxml\etree.cp312-win_amd64.exp etree.obj : error LNK2001: unresolved external symbol _xmlStrchr etree.obj : error LNK2001: unresolved external symbol _xmlIOParseDTD etree.obj : error LNK2001: unresolved external symbol _xmlMemShow [...] There are in total 503 unresolved externals. I checked the first one, and find that is is present in the downloaded libxml2_a.lib, but without the underscore. The directories of the downloaded libraries are correctly added to the compiler command line. I am out of my depth here. I hope someone is willing to help me proceed. Kind regards, Gertjan.

2 7

5.2.0 doesn't build
by da.ve.k.gupta＠gmail.com 03 Apr '24

03 Apr '24

I'm attempting to develop a project that has been operational for a while. The project makes use of mixbox which references this library. It seems that version 5.2.0 of lxml was released yesterday. Strangely, pytoml is encountering an error related to the pyproject.toml file. Can someone investigate this issue? #21 9.785 Saved /wheels/tox-2.7.0-py2.py3-none-any.whl 12:18:26 #21 9.806 Collecting lxml (from mixbox==1.0.5->-r requirements.txt (line 7)) 12:18:26 #21 10.88 Downloading https://.../lxml-5.2.0.tar.gz (3.7MB) 12:18:26 #21 15.22 Exception: 12:18:26 #21 15.22 Traceback (most recent call last): 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 143, in main 12:18:26 #21 15.22 status = self.run(options, args) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/commands/wheel.py", line 161, in run 12:18:26 #21 15.22 resolver.resolve(requirement_set) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 102, in resolve 12:18:26 #21 15.22 self._resolve_one(requirement_set, req) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 256, in _resolve_one 12:18:26 #21 15.22 abstract_dist = self._get_abstract_dist_for(req_to_install) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for 12:18:26 #21 15.22 self.require_hashes 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement 12:18:26 #21 15.22 abstract_dist.prep_for_dist(finder, self.build_isolation) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 100, in prep_for_dist 12:18:26 #21 15.22 self.req.load_pyproject_toml() 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/req/req_install.py", line 428, in load_pyproject_toml 12:18:26 #21 15.22 str(self) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/pyproject.py", line 43, in load_pyproject_toml 12:18:26 #21 15.22 pp_toml = pytoml.load(f) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 303, in load 12:18:26 #21 15.22 filename=fin.name) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 370, in loads 12:18:26 #21 15.22 toks.expect('=', 'expected_equals') 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 250, in expect 12:18:26 #21 15.22 self.error(error_text) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 253, in error 12:18:26 #21 15.22 raise TomlError(message, self.pos[0][0], self.pos[0][1], self._filename) 12:18:26 #21 15.22 pytoml.core.TomlError: /tmp/pip-wheel-29w0tw8j/lxml/pyproject.toml(26, 8): expected_equals

4 3

Re: Calling XMLSchema with nonet
by Holger.Joukl＠LBBW.de 04 Mar '24

04 Mar '24

(cc-ing the mailing list) > Thanks for all the feedback :) For now, I will stick to just one part of your feedback. > > Consider your example: > > if not no_network: > parse_options = parse_options ^ xmlparser.XML_PARSE_NONET > > Won't that always "negate" the XML_PARSE_NONET bit? If 0, it will change to 1. If 1 it will change to 0. Right? So using --no_network will always pick the opposite of the default. Am I wrong? That's probably a bit unintuitive from the snippet I gave. It works like this: Since XML_PARSE_NONET is also part of _XML_DEFAULT_PARSE_OPTIONS, the XOR logic here switches it off again when no_network has been explicitly set to False. > And regarding the default behavior. When it comes to validating according to an xml schema then the default is to download the xsd's that are imported, at least on my system > (Ubuntu 22.04 with libxml 2.9.12 installed via APT). I tried with both xmllint, xmlstarlet and lxml. Perhaps the default is for > something other than downloading xsd's? I guess > there can be references / entities stuff in the target xml document, and those references will not be downloaded. Could that be it? > >Maybe the parser used for parsing XML Schemas is set up to ignore the "normal default" and in the case of "lxml" also set up to ignore the options set with no_network. It's been a long time since I experimented with includes/imports in XML Schemas in lxml. Can't really remember the workings but in my case it was rather the other way round, i.e. no (external/remote) network access and wanting to load included/imported schemas from a local catalog. Have you tried running with XML_CATALOG_FILES set to an empty value to suppress default catalog settings? E.g. XML_CATALOG_FILES= python myprog.py I found this: https://bugs.launchpad.net/lxml/+bug/1234114 So it seems like it is indeed not "simply" possible to suppress XMLSchema network access (but maybe through catalog setup: "[...] and external imports should always be covered by catalogues (otherwise, that's a configuration problem on the user side)[...]", see the issue conversation). Another thought might be custom URI resolvers but I don't know how they tie into XML Schema handling (https://lxml.de/resolvers.html#uri-resolvers). Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

2 1

Re: Calling XMLSchema with nonet
by Holger.Joukl＠LBBW.de 04 Mar '24

04 Mar '24

Hi, > I know how to set up the parser to not download entities. But I have not found a way to stop XMLCatalog from downloading other xsd's than the root xsd. > > from lxml import etree > parser = etree.XMLParser(no_network=True) > xsddoc = etree.parse('schemas/ler/2.0_ler.xsd',parser=parser) > xsd = etree.XMLSchema(xsddoc) > > The above code will recursively download the XSD's imported in 2.0_ler.xsd. > > I played around with xmllint and I believe that if XML_PARSE_NONET is True, if will not download those. But how do I set that option for the context in which XMLSchema runs? Hm, from a quick glance at the code XML_PARSE_NONET *is* set through the no_network parser __init__option: if not no_network: parse_options = parse_options ^ xmlparser.XML_PARSE_NONET (https://github.com/lxml/lxml/blob/3ccc7d583e325ceb0ebdf8fc295bbb7fc8cd404d/…) And it defaults to True, too. That said, maybe you could use custom XML Catalog setup (https://lxml.de/resolvers.html#xml-catalogs, see also the link to libxml2 catalog setup info there) to prevent unwanted network lookup? Might even be that some default catalog handling is taking place on your machine and causing the behavior you observe(?), see https://gitlab.gnome.org/GNOME/libxml2/-/wikis/Catalog-support#how-to-tune-…. E.g. could the imported documents have already been cached, i.e. they're not even loaded from remote? This is one thing XML catalogs can provide. Best regards, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

1 0

Calling XMLSchema with nonet
by Thomas Larsen Wessel 03 Mar '24

03 Mar '24

I know how to set up the parser to not download entities. But I have not found a way to stop XMLCatalog from downloading other xsd's than the root xsd. from lxml import etree parser = etree.XMLParser(no_network=True) xsddoc = etree.parse('schemas/ler/2.0_ler.xsd',parser=parser) xsd = etree.XMLSchema(xsddoc) The above code will recursively download the XSD's imported in 2.0_ler.xsd. I played around with xmllint and I believe that if XML_PARSE_NONET is True, if will not download those. But how do I set that option for the context in which XMLSchema runs? Sincerely, Thomas

1 0

Improve error message and docs saying C compiler required to build lxml
by abepolk＠gmail.com 26 Feb '24

26 Feb '24

Dear lxml developer community, Recently, I tried to build lxml from source because I wanted to better understand how the library worked. It took me a while, but I eventually was successful. In the process, I thought of ways it could be possible to contribute to error messages and documentation to make the process easier for future developers, and I was wondering if I or someone else could implement them. Specifically, as I am new to C compilation, I had forgotten to set up the C compiler. I ran `python3 setup.py build --with-cython --with-xml2-config=/usr/bin/xml2-config`. From my current understanding, the call to _build_ext.run(self) in line 235 of setupinfo.py (as of commit 97e6a1a) fails with a CompileError because the compiler executable didn't exist. Then, setupinfo.py tries to test if the user has libxml2 installed by a call to compiler.has_function with a function in libxml2, in lines 248-253. This fails too when the C compiler is not set up, leading to the hard-coded error printed in lines 257-261. This error message suggests that libxml2 is not installed correctly, whereas I had indeed installed libxml2 through Debian's apt package manager. I looked at the source code of distutils.ccompiler.has_function, and under the hood the function has_function writes a temporary simple C file with the function from the argument, and tries to compile it. If the compile fails for any reason, it returns False. This leads to the misleading False output from has_function, which implies that libxml2 is not installed when really the compiler executable just didn't exist in order to test it. I turn, this leads to that error message mentioned above being printed. I solved the problem eventually by adding `build-essential` and `gcc` to a `sudo apt install` command before I ran the build, thereby installing the needed C compiler and tools. I thought of two ways to improve error messages in the code, and also thought we might add something to the documentation. Regarding the error messages, we could explicitly run a check that the ccompiler executable actually exists and print out an error otherwise. Short of that, we could at least improve the hard-coded error message to mention the possibility that the compiler is not installed or the configured correctly. And, maybe add a comment to setupinfo.py saying that has_function returns False if the compiler isn't set up right. Regarding the documentation, the page https://lxml.de/build.html already makes it quite clear that the developer building lxml needs to install the development versions of libxml2 and libxslt first. Perhaps, to help those new to C compilation like me, there could also be a reminder that you need a working C compiler. I hope my message is helpful and can improve the error messages and documentation. Thanks, Abe Polk

1 0

What replaced xpath.evaluate() ?
by lpsmith＠uw.edu 16 Feb '24

16 Feb '24

I'm maintaining older code, which just broke because lxml took out xpath.evaluate(). The only note in the lxml changelog about it says it was 'redundant', meaning (I assume) that there's a better way to do the same thing, but there's no documentation about what that other way might be. Does anyone know what the new code should be? The code in question looks like: xpath = lxml.etree.XPath(target, namespaces=namespaces) root = lxml.etree.Element("root") try: xpath.evaluate(root) if model_etree and check_in_model_source: [etc. etc.] except lxml.etree.XPathEvalError as exception: if 'Undefined namespace prefix' in str(exception):

3 2

Streaming read/write
by Charlie Clark 29 Jan '24

29 Jan '24

Hiya, I was recently wondering about the best way to edit XML documents using both a streaming reader and writer. I'm sure this is possible using iterparse and xmlfile but I seem to remember that iterparse produces the full tree so that parent elements and their children are returned. Has anyone had any experience with this? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226

2 5