Mailman 3 March 2022 - lxml - The Python XML Toolkit

Is there an ElementTree class lookup hook?
by Xavier Morel 08 Mar '22

08 Mar '22

lxml provides support for custom Element classes (as well as element-ish e.g. Comment or PI) via the `ElementDefaultClassLookup` registry, and the ability to hook it into a parser. But that registry does not seem to have a slot for the root tree of the elements. Is there a hook somewhere to set *that*? I tried looking around the API docs but nothing really jumped out. PS: the documentation for `set_default_parser` explains that it sets the default parser *for the current thread* and that "You can create a separate parser for each thread explicitly or use a parser pool.", does it mean that in a "don't call any API which gets an implicit parser and manage your parsers by hand" sense or something else?

2 1

prefix_objectify - a better lxml.objectify : progress report
by Dr. Volker Jaenisch 05 Mar '22

05 Mar '22

Dear lxml people! You may have noticed that I am working on a better lxml.objectify. I have not decided yet on a final name for this so lets name it "prefix_objectify" in the meanwhile. This name reflects on its aim - to include the ns-prefix information into the Python property name. Motivation: If I have a ISO19139 snipplet |<gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gco="http://www.isotc211.org/2005/gco" > <gmd:fileIdentifier> <gco:CharacterString>2ce585df-df23-45f6-b8e1-184e64e7e3b5</gco:CharacterString> </gmd:fileIdentifier> <gmd:language> <gmd:LanguageCode codeList="https://www.loc.gov/standards/iso639-2/" codeListValue="ger">ger</gmd:LanguageCode> </gmd:language> </gmd:MD_Metadata> | lxml is not capable to expose the |gco:CharacterString as a Python property.| |https://stackoverflow.com/questions/71212909/python-lxml-objectify-gives-no-attribute-access-to-gcocharacterstring-node | So I decided to come up with a better solution including the namespace-prefix into the Python property name (as all other frameworks dealing with qualified names are doing). So |gco:CharacterString will be exposed as a Python property '|||gco_CharacterString'.|| Even the recent discussion on the flaws of lxml.objectify is rather controversical. So I decided to contribute "prefix_objectify" as different python package/project and not as a fork of LXML to avoid a clash. The usage of "prefix_objectify" should be as easy as : from lxml.objectify import ObjectifyElementClassLookup from lxml import etree from prefix_objectify import PrefixObjectifiedElement obj2_lookup = ObjectifyElementClassLookup(tree_class=PrefixObjectifiedElement) parser = etree.XMLParser() parser.set_element_class_lookup(obj2_lookup) node = etree.parse('4157d397-e2c3-4e6e-8a84-0712aa9c1162.xml', parser=parser).getroot() ATM while I do reverse engineering of the lxml.objectify code it is more convenient to make a fork first. This is e.g. due to the lack of pxd-file describing lxml.objectify. My fork of lxml lives here https://github.com/Inqbus/lxml/tree/objectify_prefix in the branch 'objectify_prefix'. Around half of the tests are running good. This is after hour work in progress and I welcome any feedback or contribution. Cheers, Volker -- ========================================================= inqbus Scientific Computing Dr. Volker Jaenisch Hungerbichlweg 3 +49 (8860) 9222 7 92 86977 Burggenhttps://inqbus.de =========================================================

1 0

Re: python lxml.objectify gives no attribute access to gco:CharacterString node
by Holger.Joukl＠LBBW.de 04 Mar '22

04 Mar '22

> Absolutely! To be 100% sure to access the child you need the {URI}Name notation and there will never be a shortcut to that. > If we can agree on this, it is obious that it is no good idea to rely your code on python properties exposed by lxml.objectify, at all: > * These properties are not representing all of the xml data. > * There is a magical assumption, that a property is only visible if its namespace matches that of its parent. Sorry, but where is this supported in the RFCs of XML? Where do the W3C XML Recommendations say anything about how an XML data handling library's API should behave? > * semantic changes can change the accessability visibility of properties. > * Property names does not even show their namespace or prefix. > I would never ever dare to base code on these brittle lxml.objectify python properties. Do you? I don't consider them brittle at all. How attribute lookup works is well-defined. Semantic changes (e.g. an element changes its namespace, not merely a prefix) means you deal with a whole other element (qualified name changed). This requires you to rewrite child-accessing code anyway, unless you want to access all elements, regardless of semantic meaning. But then, you could just as well user iterchildren() or iter(). > OK let us assume for a moment that there are silly people out there basing there code on a magical design decision from 2006 in >a particular python lib called lxml, that is based on no standard. So they are using lxml.objectify properties literal in their code aka > "process(node.image)". There was no "magical design decision". It was a deliberate design choice that elem.<attr> should lookup on ns-unqualified tag attr names from elem's namespace, not from other namespaces. But you can just as easily lookup qualified using getattr(elem, '{/other/ns'attr') or elem['{/other/ns'attr]. > Then we have the obligation to help these people. A week ago I was one of them. I was not basing my code but my debugging on lxml.objectify - but all the same. > I like to make the debugging of lxml less harmfull for people like me. With lxml.objectify2 the code of such poor people, only relying on a property name > (prone to semantic changes), can be supported with namespace prefixes, helping to gain more stability in closed contexts. But what's the use case here: Iterate over elem.__dict__ and then getattr(elem, key) for every key? Simply use elem.iterchildren() instead. I understand the debugging inconvenience (and proposed a possible mitigation for it, though I'm unsure if this should be actually done because of littering __dict__ and dir() with names that are not valid Python identifiers, which is uncommon). > The mapping of ns-prefix <-> ns-URI is already present in lxml.objectify in any node. > So in case of a xml file with > xmlns html : https://www.w3.org/1999/xhtml > or an xml file with > xmlns s : https://www.w3.org/1999/xhtml > there is in any case a stable (but for sure lokal/temporal) mapping between the prefix html|s and the html namespace URI) > So the only thing that lxml.objectify2 does is mediating between different representations (clarke, <prefix>_<name>) of the same property. > > So if the user gets a property from an lxml.objectify2 entity that is "s_image" lxml.objectify2 can map this (for this particulars xml-file) > to {https://www.w3.org/1999/xhtml}image when talking to the etree api. > If the user is of the overly optimistic kind they can use "s_image" literal in their code. This will fail in some cases (depending on the namepace > mapping and context). But it will also fail if they use "image" (standard lxml) in cases of changed semantics. > If we start a discussion here if changes in the namespace prefixes are more likely to happen than semantic changes, > the whole world will laught at us. So I think we can agree that LXML should be resilent against both types of change. So then what's the point of using elem.s_image in my code when this may suddenly cease to work if I could simply use getattr(elem, '{https://www.w3.org/1999/xhtml}attr') instead and be safe? IMHO You can't be resilient against those changes: If the meaning of a variable "color" (in XML terms: "{/my/colors}color") changes you'll have to change your code. Because color now suddenly contains sounds, not colors :-). And if its name changes to "my_shiny_color" (in XML terms "{/my/shiny/colors}color") you'll have to change your code, because now the name "color" is undefined. > To make lxml.objectify2 perfect I can add the option for an user to add a prefix-namespace mapping to lxml.objectify2. With this mapping any code can define stable prefixes to work with while being independent of the namespace prefixes of a given file. This is the same notion > as for instance for node.xpath(namespaces={}) in lxml. Which you'd have to hand in when parsing? Because you can't hand it in for elem.attr syntax or getattr(elem, 'attr'), unless you'd want to override Python's built-in getattr. > To conlude my proposal: > lxml.objectify2 is better: > * since it is an addition that changes nothing at the current lxml/objectify > * since it shows (__dict__) all sub_nodes (lxml.objectify does not) > * since it shows also the namespace prefixes (lxml.objectify does not) > * since it allows for more possibilities to access/display a node > unqualified property name -> 'image' [unstable] > prefix qualified property name -> 'html_image' [locally or in certain contexts stable] > full qualified property name -> '{https://www.w3.org/1999/xhtml}image' [globally stable] Do you propose another way to access with fully qualified property name apart from getattr() / indexed access? As it stands, this is already available and your other 2 methods are unstable (so unusable, in my book). And I fail to imagine s.th. like elem.https_www_w3_org_1999_xhtml_image as a desirable API. So to me, the proposal doesn’t add substantial gain apart from the debugging visibility but rather adds ambiguity. Maybe the debugging inconvenience could be addressed in lxml.objectify, as I mentioned. Maybe you could simply implement it in pure Python on top of objectify, as Stefan suggested. Or maybe teach your IDE to better support your debugging needs. One last thing I'd respectfully ask you to: > I would never ever dare to base code on these brittle lxml.objectify python properties. Do you? > [...] silly people out there basing there code on a magical design decision from 2006 in a particular python lib called lxml, > that is based on no standard. > Then we have the obligation to help these people > I like to make the debugging of lxml less harmfull for people like me. With lxml.objectify2 the code of such poor people, only relying on a property name > If we start a discussion here if changes in the namespace prefixes are more likely to happen than semantic changes, the whole world will laught at us Please tone down your wording ("brittle lxml.objectify properties", "silly people basing there code on a magical design decision from 2006", "based on no standard", "obligation to help these people", "make the debugging of lxml less harmful", "poor people", "the whole world will laught at us"). It doesn't help. Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

1 0

Re: python lxml.objectify gives no attribute access to gco:CharacterString node
by Holger.Joukl＠LBBW.de 04 Mar '22

04 Mar '22

Hi, Stefan wrote: > Note that the content of the XML file that your code is designed to process did not > change at all. It's just that some entirely unrelated content was added, in a > completely different and unrelated namespace. And it was just externally added > to the input data, or maybe just some tiny portion it, without telling you or your > code about it. Especially in places with optional content, where different > namespaces are already a little more common than elsewhere, this is fairly likely > to go unnoticed. > > I find this kind of behaviour dangerous enough to restrict the "magic" in the API to > what is easy to understand and predict. Any magic namespace prefix-based lookup scheme can be dangerous in a similar vein IMHO: E.g. >>> root = objectify.fromstring(""" ... <a:root xmlns:a="A" xmlns:b="B"> ... <a:x>1</a:x> ... <b:x>2</b:x> ... <x>3</x> ... </a:root>""") >>> root.b_x # fictitious ns-prefix-based lookup 2 If you now change one XML doc namespace prefix from xmls:b to xmlns:ns_b: >>> root = objectify.fromstring(""" ... <a:root xmlns:a="A" xmlns:ns_b="B"> ... <a:x>1</a:x> ... <ns_b:x>2</ns_b:x> ... <x>3</x> ... </a:root>""") >>> root.b_x # fictitious ns-prefix-based lookup Traceback (most recent call last): File "<stdin>", line 1, in <module> File "src/lxml/objectify.pyx", line 231, in lxml.objectify.ObjectifiedElement.__getattr__ File "src/lxml/objectify.pyx", line 450, in lxml.objectify._lookupChildOrRaise AttributeError: no such child: b_x Again, the very same code would suddenly cease to work, while the XML document remains semantically identical. You'd get an exception in the best case, or silently ignore data in the worst case. That aside: Volker wrote: > [...] > Debugging becomes a great hassle if you are not able e.g. in your > PyCharm IDE to navigate the XML tree your parser a currently processing. > Even worse if some nodes do not seem to even exist. > [...] > It is not that I like a more convenient way to address the data. To > address the data I use xpath. It is purely the fact that I cannot use > the objectified data in a debugger while debugging, that drives me mad. I admit I don’t fully understand the issue (I don't use PyCharm and don't know how it presents objects in debugging). To me, it seems easy enough to just do s.th. like >>> list(root.iterchildren()) [1, 2, 3] or >>> print(objectify.dump(root)) # see also objectify.enable_recursive_str() {A}root = None [ObjectifiedElement] {A}x = 1 [IntElement] {B}x = 2 [IntElement] x = 3 [IntElement] Does PyCharm use elem.__dict__ or dir(elem) to present an object's attributes in debugging? Then maybe a way to address OP's issue might be to populate elem.__dict__ not only with element children from the same namespace but with all children while *still* only attribute-lookup children from elem's namespace. I.e. instead of >>> root = objectify.fromstring(""" ... <a:root xmlns:a="A"> ... <a:x>1</a:x> ... <x>3</x> ... </a:root>""") >>> >>> root.__dict__ {'x': 1} __dict__ would yield >>> root.__dict__ # not how it works today! {'{A}x': 1, '{}x': 3} ...making all children appear in e.g. dir(), keeping existing getattr behavior: >>> root.a 1 Maybe this would lessen the "child visibility issue" in debugging? A breaking change of course, making __dict__ usage more surprising and arguably more "non-standard" compared to regular Python objects IMO, since they'd contain names that are not valid Python identifiers. A cursory glance over the implementation looks like this should be possible in theory. But I'm rather not convinced we should do this. Maybe the debugger/IDE can just be taught to give more helpful output? All the information is there in the first place... Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

3 6

Re: python lxml.objectify gives no attribute access to gco:CharacterString node
by Holger.Joukl＠LBBW.de 03 Mar '22

03 Mar '22

Hi Volker, > A week ago I formulated a quite simple lxml.objectify question on stackoverflow and got no answer, yet. > https://stackoverflow.com/questions/71212909/python-lxml-objectify-gives-no… > Since I don't get an easy answer I conclude that my feeling about something is wrong in lxml.objectfy is not a mare. > So I elevate this topic to the mailing list. > Any help appreciated > Volker Jaenisch This is expected and just how lxml.objectify works. Since {http://www.isotc211.org/2005/gco}CharacterString is from another namespace than its parent element ({"http://www.isotc211.org/2005/gmd}fileIdentifier) it cannot be retrieved from the parent with "simple (dot) attribute lookup". Instead you have to you use index access (just like you already did) or getattr: ``` >>> from lxml import etree, objectify >>> data = objectify.fromstring( ... """<gmd:MD_Metadata ... xmlns:gmd="http://www.isotc211.org/2005/gmd" ... xmlns:gco="http://www.isotc211.org/2005/gco" > ... <gmd:fileIdentifier> ... <gco:CharacterString>2ce585df-df23-45f6-b8e1-184e64e7e3b5</gco:CharacterString> ... </gmd:fileIdentifier> ... <gmd:language> ... <gmd:LanguageCode codeList="https://www.loc.gov/standards/iso639-2/" codeListValue="ger">ger</gmd:LanguageCode> ... </gmd:language> ... </gmd:MD_Metadata> ... """) >>> data.fileIdentifier["{http://www.isotc211.org/2005/gco}CharacterString"] '2ce585df-df23-45f6-b8e1-184e64e7e3b5' >>> getattr(data.fileIdentifier, "{http://www.isotc211.org/2005/gco}CharacterString") '2ce585df-df23-45f6-b8e1-184e64e7e3b5' >>> ``` The reason for this is that obviously {http://www.isotc211.org/2005/gco}CharacterString is not a valid Python identifier and it makes sense to restrict unqualified lookup to children from the same namespace. See also "Namespace handling" in the official docs: https://lxml.de/objectify.html#the-lxml-objectify-api Best, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

4 5

Re: python lxml.objectify gives no attribute access to gco:CharacterString node
by Holger.Joukl＠LBBW.de 01 Mar '22

01 Mar '22

Hi Volker, > > The reason for this is that obviously {http://www.isotc211.org/2005/gco}CharacterString is not a valid Python identifier and it makes sense > > to restrict unqualified lookup to children from the same namespace. > I like to disagree on > > and it makes sense > > to restrict unqualified lookup to children from the same namespace I beg to differ :-). > What does the namespace of a node has in common with the namespace of one of its subnodes? Nothing. It is quite common in XML that you borrow from other namespaces. I'd rather assume the vast majority of XML documents do not consist of many different namespaces and heavily "oscillate" between parents and children from different namespaces. But I've no data to back up that claim. For me, simply using parent['{/other/ns}child'] or getattr(parent, '{/other/ns}child') syntax just works. Not as beautiful as parent.child but not ugly, either. > Other namespace based python libs like for instance RDFlib solve this problem generically by adding the namespace to the python property. > {http://www.isotc211.org/2005/gco}CharacterString -> gco_CharacterString Well, but they're not adding the namespace but the ns-prefix. lxml uses Clarke-notation and qualified tag names everywhere, which is less error-prone and preferable in my experience (http://www.jclark.com/xml/xmlns.htm). No problems with namespace <--> prefix indirections, multiple prefixes for the same namespace, etc. Note that to the best of my knowledge not every allowed prefix is a valid python identifier (I think all XML name chars except ":" are allowed). So apart from tedious namespace <--> prefix indirection handling you'd also needed to cater for rules to replace characters (e.g. <my-nsprefix:root xmlns:my-nsprefix="/my/ns"/> --> my_ns_root?). If you wanted to lookup children using a <ns-prefix>_<unqualified name> attribute name syntax, that is. > The problem lies deeply burrowed in the nature of LXML objectify implementation. Objectify does not really transform the XML into a real python instance hierarchy (as RDFlib does), > but directs all attribute access via function calls to the C-libxml core. This is on one hand a desired behavior since one so can change XML on-the-fly and some of the changes > are visible as well in the XML as also in the objectified representation. > But on the other hand the information what namespace a node belongs to is not persistent in the node and therefore cannot be used for lookup. To me this is a feature rather than a problem and it's been a design decision from day one (https://mail.python.org/archives/list/lxml@python.org/message/GOTPAWC4MHI5L…). Some others seem to agree: https://zato.io/blog/posts/saving-time-with-a-pythonic-api-of-lxmlobjectify…. But I'm obviously and deeply biased ;-) Best regards, Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart HRA 4356, HRA 104 440 Amtsgericht Mannheim HRA 40687 Amtsgericht Mainz Die LBBW verarbeitet gemaess Erfordernissen der DSGVO Ihre personenbezogenen Daten. Informationen finden Sie unter https://www.lbbw.de/datenschutz.

2 1