Fwd: [lxml-dev] xpath extension functions
from lxml import etree doc = etree.parse('doc.xml') xpath = XPath(doc) Overall, it is very close to what I had in mind, except that I would not call that class XPath. I understand it wraps XPath functionality, but what you describe is very much what a xmlXPathContext does... i.e. bind namespaces and extension functions! So I suggest we call the class XPathContext.... I read your whole document with s/XPath/XPathContext/g; ;->
It's indeed probably just an xmlXPathContext wrapper.
I don't think we need to bother any developer with talk about a 'Context'; it's a superfluous term carrying over from libxml2 in my mind, just like the API doesn't mention RelaxNG contexts or XSLT contexts. Unless a class 'XPath' brings something else to mind and confuses you, I want to go with XPath. :)
Well, precisely. A XPath is a XPath, like "/p", this on the other hand is an object that will be used to interpret many XPaths. Frankly, even if this were not how it were called in the libxml library, XPathContext is probably how I would name it... I am not opposed to another name, but -1 on XPath, and frankly I cannot think of a name that describes it better than XPathContext... (DocumentContext, since it also contains namespace aspects? But again, the XPath may introduce namespaces that the document did not know about.)
xpath.registerNamespaces(namespace_dict) xpath.registerFunction('foo', f) I had already written something closer to xpath.registerFunction(f) which is the same as xpath.registerFunction(f, 'f') The name I see as optional, so it should be the second arg.
How would the name otherwise be deduced? If we make the name non-optional, they can be dictionary keys...
The name of the function, f.__name__. Or, even better, a translation of camel-caps to hyphenated conventions which are the norm in the xslt world.
results = xpath.evaluate('//p') OK, here you assume relative to the document. The context also allows the idea of current node, which is a good thing when you work within an extension function. So I would have a syntax results = xpath.evaluate('//p', node) which accepts a given node; by default the context's current location.
I though I described that later on in my document. :)
Oh, sorry. I misinterpreted the "context element" later in your document. I thought you meant the context of the application. It's true that the word context can be misleading.
xpathContext2 = xpathContext.cloneWithNode(node) (so the target node would be read-only) The second way could be to save the target node in a local variable while calling the extension function, in case it messes the target node. (Much less happy about that option.)
What is the target node?
This is how I call it ;-)
Finally, if we want to use a XPathContext with a different origin from within an extension function, I see two solutions:
What is a an XPathContext with a different origin and why would we want to do that? You mean to have an extension function do its own XPath evaluation?
Precisely. I have used this in some functions I wrote.
I can see here why cloning would be convenient, as you wouldn't need to figure out the extension functions anymore to set up; presumably you'd want to use the same set as before.
Correct. This was the recursion problem described in an earlier email.
On that note, if you want to reuse functions and namespaces, I would allow a clone function:
xpathContext2 = xpathContext.cloneWithDoc(doc) or something like that.
Yeah, a clone() could be doable, and is a reasonable idea, but I won't worry about it for now. If initializing the XPath object is as easy as putting in a document and two dictionaries (one for namespaces, one for functions), I think people can do that themselves. I'm looking to cut out the API we really need first.
Fair enough, though I think we do need some way to do the clone with a different [ origin | target node | context element ] for the reasons above. If you want to keep the API down, a way to do this would be to have getters on the namespace, functions _and_modules_ dictionaries, which we could use in the constructor: newContext = XPathContext(doc, oldContext.namespaces, oldContext.extensionFunctions, oldContext.extensionModules) (discussion later on why modules.) I think this is possible, but heavy; this is why I would prefer a convenience function newContext = oldContext.clone() or newContext = XPathContext(context=oldContext) But that is a matter of preference.
An upside would be the following: Assume that we store the (Pyrex) XPathContext object in the void* userData in the (C) xmlXPathContext. Recall that (Python) extension functions receive the (C) xmlXPathParserContext (as a Pyrex object of course), and hence can access the (C) xmlXPathContext, so we could give them access to the (Pyrex) XPathContext. That way, Kapil can store user data in some object variable in the (per-session) clone of the XPathContext; and make all the clones from a single one which is configured with namespaces and extensions.
I'm not sure I understand. Why not provide a new object altogether for UserData? I don't see why this need be the task of the XPath context. Could be a third argument to evaluate(), perhaps?
See it from the viewpoint of the extension function: All it receives (in C) is the xmlXPathContext. I assume that what the python extension function would receive would be a Pyrex object that wraps that C structure (however we call it!), or a Python object that we would have stored in that C structure (using the userData field). The latter choice actually risks losing information; the (wrapped) xmlXPathContext contains a lot of information that the extension function might need, like namespaces. Hence, I assume that further user data, for Kapil's need, would have to be accessible from this object that the Python function receives, which is still probably the xmlXPathContext wrapper. Hence the above paragraph.
Also, each document gets to reuse a XPathContext, which means that we do not have to set one up each time we evaluate a xpath.
This is a separate idea from the above user data story, right? I see this as an optimization of the .xpath method. It would require a way to see the XPath object of the document to be seeded with namespaces through a separate API on the document or something.. An alternative is just to do away with the .xpath method altogether and require people to use XPath() directly, which would make it harder for people to make performance mistakes. I'm not sure, the convenience it offers right now, especially for evaluating in element contexts, is pretty nice.
It is indeed... This is why I was toying with the idea of tying a default XPathContext to a document. Still ambivalent about it.
Finally, another thing I would add to the xpathContext API is the option to declare variables (again as a dictionary) that can be read (only) by extension functions. Another way for Kapil to do things... though there he only gets a literal or node, not a Python object. Still, a literal can also be a key into a thread-global dictionary of session objects.
Having some way to pass along Python objects to extension functions would be nice. This is separate from the XPath $variable concept, which we'll also need to support.
Yes, you are right, they are distinct. I was musing here.
Overall, I think this can fly.
Registration of functions could work as a dictionary too:
xpath.registerFunctions(function_dict) # function name : python func Yes... I do like this, but let us look at the XSLT philosophy, which assumes modules, before we do too much that is incompatible... So I am a tad less sure here.
I think worrying about XSLT when we get to it would be fine.
Sorry, but this is where we really disagree. I also started that way, and then realized that it was a way to get this nice API for extension functions, and then realize that I'd have to build a completely different one for XSLT... Which also involves extension functions, in a very different way. This is a way to seriously get into impedance mismatch with two incompatible APIs that attempt to do the same thing.
A dictionary of extension functions could be turned into some kind of module, right?
No. A module involves a) extension functions b) extension elements (each of which is strictly speaking a pair of functions, but I decided to sweep that under the carpet...) c) a namespace URI d) convenience methods for module initialization, for each document parsed More annoying, modules are registered globally, unlike XPath extension functions.
The way I see it, we should actually be able to associate extension functions and elements with a namespace. A
Extension elements are a separate story again, right? Can extension functions have a namespace?
Yes, very much so. (You will notice I have them in my toy implementation.)
module allows that in a neat fashion: a module encapsulates a namespace URI, extension functions and elements, and management function that are called at beginning/end of module setup and document transform respectively. (And yes, I have found both of those useful!) Also, modules are very much registered globally. (They are associated to a local name at the level of an individual transform, however.) SO.... One option is to actually reuse the existing libxslt.extensionModule class.
I won't allow anything like the libxml2-style APIs near the lxml API. :) I'm sure there are useful concepts in there, but I first want to tackle XPath. Then we'll look at XSLT. If we keep the XPath API as minimal and simple as we can (for the Python developer using lxml), we should be able to translate some of those concepts into the XSLT API. Anyway, I'll skip the XSLT part for now until we've implemented an XPath API.
Again, sorry to differ, and it is your project, but I hope you will reconsider. Note that I am very open to thinking of other ways to treat the modules, that is more pythonic and less close to the libxslt; I am however quite convinced that tackling the two projects as separate is a way to to wind up with a two-headed API, or a lot of glue to make it look like a single one. Also, to be honest, though I really admire what has happened so far in lxml to simplify the libxml API, I am not so convinced that the xslt extension API (i.e. using modules) so desperately needs fixing. But as I said, I would love to discuss alternatives if you care to discuss them. (I have not thought of any yet, myself, but I can try.) I hope you will not be offended by my strong opinions in the matter. Regards, Marc-Antoine Parent
participants (1)
-
Marc-Antoine Parent