Mailman 3 - lxml - The Python XML Toolkit

Consider keeping manylinux1 wheels for Python 3.6
by Miro Hrončok 04 Apr '24

04 Apr '24

Hello, I've noticed that lxml 5.1+ upgraded the manylinux wheels to a newer tag. The default ensurpip-bundled pip version in Python 3.6 does not support newer manylinuxes, hence it is likely that many CI systems that still test 3.6 now attempt to build lxml from sources. Since 5.2, this also fails with the old pip due to the old bundled pytoml, as indicated in a previous thread on this list. $ python3.6 -m venv venv3.6 $ venv3.6/bin/pip list Package Version ---------- ------- pip 18.1 setuptools 40.6.2 5.0.2 has a manylinux1 wheel: $ venv3.6/bin/pip install lxml==5.0.2 ... lxml-5.0.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl 5.1.0 builds from source but uses setup.py and works (with devel deps): $ venv3.6/bin/pip install lxml==5.1.0 ... lxml-5.1.0.tar.gz Running setup.py install for lxml ... 5.2.1 builds from source and will outright blow up when parsing pyproject.toml: $ venv3.6/bin/pip install lxml==5.2.1 ... lxml-5.1.0.tar.gz ... pip._vendor.pytoml.core.TomlError: /tmp/.../lxml/pyproject.toml(40, 1): msg If support for Python 3.6 is still desired, would it maybe make sense to keep building and uploading manylinux1 wheels to make it easier? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

5.2.0 doesn't build
by da.ve.k.gupta＠gmail.com 03 Apr '24

03 Apr '24

I'm attempting to develop a project that has been operational for a while. The project makes use of mixbox which references this library. It seems that version 5.2.0 of lxml was released yesterday. Strangely, pytoml is encountering an error related to the pyproject.toml file. Can someone investigate this issue? #21 9.785 Saved /wheels/tox-2.7.0-py2.py3-none-any.whl 12:18:26 #21 9.806 Collecting lxml (from mixbox==1.0.5->-r requirements.txt (line 7)) 12:18:26 #21 10.88 Downloading https://.../lxml-5.2.0.tar.gz (3.7MB) 12:18:26 #21 15.22 Exception: 12:18:26 #21 15.22 Traceback (most recent call last): 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 143, in main 12:18:26 #21 15.22 status = self.run(options, args) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/commands/wheel.py", line 161, in run 12:18:26 #21 15.22 resolver.resolve(requirement_set) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 102, in resolve 12:18:26 #21 15.22 self._resolve_one(requirement_set, req) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 256, in _resolve_one 12:18:26 #21 15.22 abstract_dist = self._get_abstract_dist_for(req_to_install) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for 12:18:26 #21 15.22 self.require_hashes 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement 12:18:26 #21 15.22 abstract_dist.prep_for_dist(finder, self.build_isolation) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 100, in prep_for_dist 12:18:26 #21 15.22 self.req.load_pyproject_toml() 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/req/req_install.py", line 428, in load_pyproject_toml 12:18:26 #21 15.22 str(self) 12:18:26 #21 15.22 File "/usr/lib/python3/dist-packages/pip/_internal/pyproject.py", line 43, in load_pyproject_toml 12:18:26 #21 15.22 pp_toml = pytoml.load(f) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 303, in load 12:18:26 #21 15.22 filename=fin.name) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 370, in loads 12:18:26 #21 15.22 toks.expect('=', 'expected_equals') 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 250, in expect 12:18:26 #21 15.22 self.error(error_text) 12:18:26 #21 15.22 File "/usr/share/python-wheels/pytoml-0.1.2-py2.py3-none-any.whl/pytoml/parser.py", line 253, in error 12:18:26 #21 15.22 raise TomlError(message, self.pos[0][0], self.pos[0][1], self._filename) 12:18:26 #21 15.22 pytoml.core.TomlError: /tmp/pip-wheel-29w0tw8j/lxml/pyproject.toml(26, 8): expected_equals

4 3

Re: Calling XMLSchema with nonet
by Holger.Joukl＠LBBW.de 05 Mar '24

05 Mar '24

(cc-ing the mailing list) > Thanks for all the feedback :) For now, I will stick to just one part of your feedback. > > Consider your example: > > if not no_network: > parse_options = parse_options ^ xmlparser.XML_PARSE_NONET > > Won't that always "negate" the XML_PARSE_NONET bit? If 0, it will change to 1. If 1 it will change to 0. Right? So using --no_network will always pick the opposite of the default. Am I wrong? That's probably a bit unintuitive from the snippet I gave. It works like this: Since XML_PARSE_NONET is also part of _XML_DEFAULT_PARSE_OPTIONS, the XOR logic here switches it off again when no_network has been explicitly set to False. > And regarding the default behavior. When it comes to validating according to an xml schema then the default is to download the xsd's that are imported, at least on my system > (Ubuntu 22.04 with libxml 2.9.12 installed via APT). I tried with both xmllint, xmlstarlet and lxml. Perhaps the default is for > something other than downloading xsd's? I guess > there can be references / entities stuff in the target xml document, and those references will not be downloaded. Could that be it? > >Maybe the parser used for parsing XML Schemas is set up to ignore the "normal default" and in the case of "lxml" also set up to ignore the options set with no_network. It's been a long time since I experimented with includes/imports in XML Schemas in lxml. Can't really remember the workings but in my case it was rather the other way round, i.e. no (external/remote) network access and wanting to load included/imported schemas from a local catalog. Have you tried running with XML_CATALOG_FILES set to an empty value to suppress default catalog settings? E.g. XML_CATALOG_FILES= python myprog.py I found this: https://bugs.launchpad.net/lxml/+bug/1234114 So it seems like it is indeed not "simply" possible to suppress XMLSchema network access (but maybe through catalog setup: "[...] and external imports should always be covered by catalogues (otherwise, that's a configuration problem on the user side)[...]", see the issue conversation). Another thought might be custom URI resolvers but I don't know how they tie into XML Schema handling (https://lxml.de/resolvers.html#uri-resolvers). Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

2 1

Re: Calling XMLSchema with nonet
by Holger.Joukl＠LBBW.de 04 Mar '24

04 Mar '24

Hi, > I know how to set up the parser to not download entities. But I have not found a way to stop XMLCatalog from downloading other xsd's than the root xsd. > > from lxml import etree > parser = etree.XMLParser(no_network=True) > xsddoc = etree.parse('schemas/ler/2.0_ler.xsd',parser=parser) > xsd = etree.XMLSchema(xsddoc) > > The above code will recursively download the XSD's imported in 2.0_ler.xsd. > > I played around with xmllint and I believe that if XML_PARSE_NONET is True, if will not download those. But how do I set that option for the context in which XMLSchema runs? Hm, from a quick glance at the code XML_PARSE_NONET *is* set through the no_network parser __init__option: if not no_network: parse_options = parse_options ^ xmlparser.XML_PARSE_NONET (https://github.com/lxml/lxml/blob/3ccc7d583e325ceb0ebdf8fc295bbb7fc8cd404d/…) And it defaults to True, too. That said, maybe you could use custom XML Catalog setup (https://lxml.de/resolvers.html#xml-catalogs, see also the link to libxml2 catalog setup info there) to prevent unwanted network lookup? Might even be that some default catalog handling is taking place on your machine and causing the behavior you observe(?), see https://gitlab.gnome.org/GNOME/libxml2/-/wikis/Catalog-support#how-to-tune-…. E.g. could the imported documents have already been cached, i.e. they're not even loaded from remote? This is one thing XML catalogs can provide. Best regards, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

1 0

Calling XMLSchema with nonet
by Thomas Larsen Wessel 04 Mar '24

04 Mar '24

I know how to set up the parser to not download entities. But I have not found a way to stop XMLCatalog from downloading other xsd's than the root xsd. from lxml import etree parser = etree.XMLParser(no_network=True) xsddoc = etree.parse('schemas/ler/2.0_ler.xsd',parser=parser) xsd = etree.XMLSchema(xsddoc) The above code will recursively download the XSD's imported in 2.0_ler.xsd. I played around with xmllint and I believe that if XML_PARSE_NONET is True, if will not download those. But how do I set that option for the context in which XMLSchema runs? Sincerely, Thomas

1 0

Improve error message and docs saying C compiler required to build lxml
by abepolk＠gmail.com 26 Feb '24

26 Feb '24

Dear lxml developer community, Recently, I tried to build lxml from source because I wanted to better understand how the library worked. It took me a while, but I eventually was successful. In the process, I thought of ways it could be possible to contribute to error messages and documentation to make the process easier for future developers, and I was wondering if I or someone else could implement them. Specifically, as I am new to C compilation, I had forgotten to set up the C compiler. I ran `python3 setup.py build --with-cython --with-xml2-config=/usr/bin/xml2-config`. From my current understanding, the call to _build_ext.run(self) in line 235 of setupinfo.py (as of commit 97e6a1a) fails with a CompileError because the compiler executable didn't exist. Then, setupinfo.py tries to test if the user has libxml2 installed by a call to compiler.has_function with a function in libxml2, in lines 248-253. This fails too when the C compiler is not set up, leading to the hard-coded error printed in lines 257-261. This error message suggests that libxml2 is not installed correctly, whereas I had indeed installed libxml2 through Debian's apt package manager. I looked at the source code of distutils.ccompiler.has_function, and under the hood the function has_function writes a temporary simple C file with the function from the argument, and tries to compile it. If the compile fails for any reason, it returns False. This leads to the misleading False output from has_function, which implies that libxml2 is not installed when really the compiler executable just didn't exist in order to test it. I turn, this leads to that error message mentioned above being printed. I solved the problem eventually by adding `build-essential` and `gcc` to a `sudo apt install` command before I ran the build, thereby installing the needed C compiler and tools. I thought of two ways to improve error messages in the code, and also thought we might add something to the documentation. Regarding the error messages, we could explicitly run a check that the ccompiler executable actually exists and print out an error otherwise. Short of that, we could at least improve the hard-coded error message to mention the possibility that the compiler is not installed or the configured correctly. And, maybe add a comment to setupinfo.py saying that has_function returns False if the compiler isn't set up right. Regarding the documentation, the page https://lxml.de/build.html already makes it quite clear that the developer building lxml needs to install the development versions of libxml2 and libxslt first. Perhaps, to help those new to C compilation like me, there could also be a reminder that you need a working C compiler. I hope my message is helpful and can improve the error messages and documentation. Thanks, Abe Polk

1 0

What replaced xpath.evaluate() ?
by lpsmith＠uw.edu 16 Feb '24

16 Feb '24

I'm maintaining older code, which just broke because lxml took out xpath.evaluate(). The only note in the lxml changelog about it says it was 'redundant', meaning (I assume) that there's a better way to do the same thing, but there's no documentation about what that other way might be. Does anyone know what the new code should be? The code in question looks like: xpath = lxml.etree.XPath(target, namespaces=namespaces) root = lxml.etree.Element("root") try: xpath.evaluate(root) if model_etree and check_in_model_source: [etc. etc.] except lxml.etree.XPathEvalError as exception: if 'Undefined namespace prefix' in str(exception):

3 2

Streaming read/write
by Charlie Clark 29 Jan '24

29 Jan '24

Hiya, I was recently wondering about the best way to edit XML documents using both a streaming reader and writer. I'm sure this is possible using iterparse and xmlfile but I seem to remember that iterparse produces the full tree so that parent elements and their children are returned. Has anyone had any experience with this? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Sengelsweg 34 Düsseldorf D- 40489 Tel: +49-203-3925-0390 Mobile: +49-178-782-6226

2 5

Re: Processing as XML file.
by Holger.Joukl＠LBBW.de 30 Oct '23

30 Oct '23

Hi Keith, > I have a directory structure and a GDML file as follows > <!DOCTYPE xml [ > <!ENTITY define SYSTEM "DC1_define.xml"> > <!ENTITY materials SYSTEM "DC1_materials.xml"> > <!ENTITY solids SYSTEM "DC1_solids.xml"> > <!ENTITY setup SYSTEM "DC1_setup.xml"> > <!ENTITY struct SYSTEM "DC1_struct.xml"> >]> > > <xml xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xsi:noNamespaceSchemaLocation=http://service-spi.web.cern.ch/service-spi/app/releases/GDML/schema/gdml.xsd>&define;&materials;&solids;&setup;&struct;</xml> > Where the files labelled with .xml type are not strictlly xml files as they have no <xml> ...content ... </xml> just valid include contents > It is processed by > try: > from lxml import etree > parser = etree.XMLParser(resolve_entities=True) > root = etree.parse(filename, parser=parser) > # print('error log') > # print(parser.error_log) > > except ImportError: > > [...] > > This all works fine BUT > I now wish to implement other functions which process the files listed in the Entities as individual xml files, Is there a way to achieve this? > or alternatively is there a way to use an actual xml file as a include. (not having actually tried out any of this, but) what comes to mind is: - Parse with resolve_enttities=False and then handle the unresolved entities yourself. You should be able to get at the entity references in the tree e.g. using tree.iter(etree.Entity). The entity definitions can be accessed through a tree's docinfo attribute: tree.docinfo.internalDTD.entities() - If you have control over the XML file, maybe switch to XInclude instead of entities to include separate content? Again, instead of automatically processing inclusions (through tree.xinclude()) iterate the include nodes yourself for full control of what you want to achieve. - lxml supports custom resolvers. Both the DTD and XInclude approaches might be combinable with such custom resolvers to hook into the regular mechanics and get more control over the actual (see https://lxml.de/resolvers.html). Note: you're probably aware that resolve_entities=True can be a security risk if applied to untrusted XML input. Handle with care. Best regards, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

1 0

Processing as XML file.
by Keith Sloan 27 Oct '23

27 Oct '23

I have a directory structure and a GDML file as follows <!DOCTYPE xml [ <!ENTITY define SYSTEM "DC1_define.xml"> <!ENTITY materials SYSTEM "DC1_materials.xml"> <!ENTITY solids SYSTEM "DC1_solids.xml"> <!ENTITY setup SYSTEM "DC1_setup.xml"> <!ENTITY struct SYSTEM "DC1_struct.xml"> ]> <xml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://service-spi.web.cern.ch/service-spi/app/releases/GDML/schema/gdml.xsd">&define;&materials;&solids;&setup;&struct;</xml> Where the files labelled with .xml type are not strictlly xml files as they have no <xml> ...content ... </xml> just valid include contents It is processed by try: from lxml import etree parser = etree.XMLParser(resolve_entities=True) root = etree.parse(filename, parser=parser) # print('error log') # print(parser.error_log) except ImportError: try: import xml.etree.ElementTree as etree FreeCAD.Console.PrintMessage( "running with etree.ElementTree (import limitations)\n" ) FreeCAD.Console.PrintMessage( " for full import add lxml library \n" ) tree = etree.parse(filename) This all works fine BUT I now wish to implement other functions which process the files listed in the Entities as individual xml files, Is there a way to achieve this? or alternatively is there a way to use an actual xml file as a include. -- ========== Art & Ceramics =========== https://www.instagram.com/ksloan1952/

2 1