Mailman 3 - lxml - The Python XML Toolkit

[lxml-dev] lxml 0.5 released
by Martijn Faassen April 8, 2005

April 8, 2005

Hi there, I've just released lxml version 0.5. A release at last! You can find more information about it, and download, here: http://codespeak.net/lxml/ Regards, Martijn

2 1

Re: [lxml-dev] using modules for XPath/XSLT extensions
by Marc-Antoine Parent Jan. 28, 2005

Jan. 28, 2005

> Even though I said I wouldn't consider XSLT extensions as I'm > overwhelmed, Marc-Antoine brought up the module concept. Thank you for looking at it ;-) > I haven't played a lot with extension functions at all with libxslt, > so I don't really know how it or the Python binding works now. > > However, do I understand you correctly, Marc-Antoine, that instead of > a dictionary of function_name->function, you would instead pass in a > Python module object? Or as … [View More]I see in your example, a class object. Sorry for the confusion, "module" is the libxslt terminology, which introduces confusion with the python modules. I was thinking of an instance object, with a class defining the xslt module. I am using an instance so as to allow users to add data to the instance. This is in line with libxslt-python usage, as I think it would be nice to allow people to reuse them with minimal changes. I am still making changes, however, like making the namespace URI accessible as a class-level method (it could be a read-only class attribute); this is because defining the URI exists at the same level as defining the extension functions that make up the module. > This module object (or class object) would contain some special > attributes: > > * the namespace the extension functions are in > > * the functions themselves > > * functions used to import these functions If you refer to the decorators, the point was mostly to allow them to identify which of the module functions are extension functions, and which are extension elements. Incidentally, I made the decorators take care of registration, but that is very much an implementation detail. In theory, registration should be handled by the styleInit method. > * a list of function names we want to register with XPath Conceptually, yes. Each extension function and element has a name visible to XSLT which may have nothing to do with the name of the function (or method) as Python sees it. Again, I put that functionality in an optional decorator arguments, with a default behaviour that would simply extract the method name and do some name conversion (camel caps to hyphenated lowercase) that happens to be sensible most of the time. Again, I wanted the default case to be simple. It is entirely possible to have (class or even static) methods in the extensionModule baseclass instead, such as registerExtensionFunction(bound_method, name, uri) registerExtensionElement(bound_method, name, uri) that would be called in styleInit. but I liked the idea of adding the URI as a mandatory class attribute, that and default name conversion would allow a (imho, simpler) decorator syntax: @xmlExtensionFunction def myFunctionName(...): would the be defined as my-function-name on the module's URI. > * possibly a mapping of python names to names as to be used in xpath, > for instance to handle - instead of _ or camelcase. I would prefer > having no automatic translation by default, as that's a bit too much > magic, but we could think about somekind of special function name > translation function that you could hook into. I agree that it would be even better if we could make the syntactic sugar easy to access, but not automatic. So let us assume that @xmlExtensionFunction def myFunctionName(...): defines myFunctionName , but we allow @xmlExtensionFunction(translateName=True) def myFunctionName(...): to define my-function-name and of course @xmlExtensionFunction(name="whatever-name") def myFunctionName(...): (In all cases I assume the URI is taken from the module; but I guess the registration function that gets called by the decorator would also have a namespaceURI argument ) > I haven't figured out yet where styleInit(self, stylesheet, URI) and > friends fit in... Am not too clear on the use cases yet. As I said, this is _normally_ where you register your module ext. ctxtInit, on the other hand, allows you to set up user data (like Kapil's) on your (xslt) module instance for a given application of the transformation. > Hm... Need to think more. Almost sorry I brought this up before I was ready with a fuller implementation... Marc-Antoine [View Less]

2 3

[lxml-dev] using modules for XPath/XSLT extensions
by Martijn Faassen Jan. 28, 2005

Jan. 28, 2005

Hi there, Even though I said I wouldn't consider XSLT extensions as I'm overwhelmed, Marc-Antoine brought up the module concept. I haven't played a lot with extension functions at all with libxslt, so I don't really know how it or the Python binding works now. However, do I understand you correctly, Marc-Antoine, that instead of a dictionary of function_name->function, you would instead pass in a Python module object? Or as I see in your example, a class object. This module object (… [View More]

1 0

Fwd: [lxml-dev] xpath extension functions
by Marc-Antoine Parent Jan. 28, 2005

Jan. 28, 2005

>>> >>> from lxml import etree >>> >>> doc = etree.parse('doc.xml') >>> >>> xpath = XPath(doc) >> Overall, it is very close to what I had in mind, except that I would >> not call that class XPath. I understand it wraps XPath functionality, >> but what you describe is very much what a xmlXPathContext does... >> i.e. bind namespaces and extension functions! So I suggest we call >> the class XPathContext.... I … [View More]read your whole document with >> s/XPath/XPathContext/g; ;-> > > It's indeed probably just an xmlXPathContext wrapper. > > I don't think we need to bother any developer with talk about a > 'Context'; it's a superfluous term carrying over from libxml2 in my > mind, just like the API doesn't mention RelaxNG contexts or XSLT > contexts. Unless a class 'XPath' brings something else to mind and > confuses you, I want to go with XPath. :) Well, precisely. A XPath is a XPath, like "/p", this on the other hand is an object that will be used to interpret many XPaths. Frankly, even if this were not how it were called in the libxml library, XPathContext is probably how I would name it... I am not opposed to another name, but -1 on XPath, and frankly I cannot think of a name that describes it better than XPathContext... (DocumentContext, since it also contains namespace aspects? But again, the XPath may introduce namespaces that the document did not know about.) >>> >>> xpath.registerNamespaces(namespace_dict) >>> >>> xpath.registerFunction('foo', f) >> I had already written something closer to >> >>> xpath.registerFunction(f) >> which is the same as >> >>> xpath.registerFunction(f, 'f') >> The name I see as optional, so it should be the second arg. > > How would the name otherwise be deduced? If we make the name > non-optional, they can be dictionary keys... The name of the function, f.__name__. Or, even better, a translation of camel-caps to hyphenated conventions which are the norm in the xslt world. >>> >>> results = xpath.evaluate('//p') >> OK, here you assume relative to the document. The context also >> allows the idea of current node, which is a good thing when you work >> within an extension function. >> So I would have a syntax >> >>> results = xpath.evaluate('//p', node) >> which accepts a given node; by default the context's current location. > > I though I described that later on in my document. :) Oh, sorry. I misinterpreted the "context element" later in your document. I thought you meant the context of the application. It's true that the word context can be misleading. >> xpathContext2 = xpathContext.cloneWithNode(node) >> (so the target node would be read-only) >> The second way could be to save the target node in a local variable >> while calling the extension function, in case it messes the target >> node. (Much less happy about that option.) > > What is the target node? This is how I call it ;-) >> Finally, if we want to use a XPathContext with a different origin >> from within an extension function, I see two solutions: > > What is a an XPathContext with a different origin and why would we > want to do that? You mean to have an extension function do its own > XPath evaluation? Precisely. I have used this in some functions I wrote. > I can see here why cloning would be convenient, as you wouldn't need > to figure out the extension functions anymore to set up; presumably > you'd want to use the same set as before. Correct. This was the recursion problem described in an earlier email. >> On that note, if you want to reuse functions and namespaces, I would >> allow a clone function: >> >>> xpathContext2 = xpathContext.cloneWithDoc(doc) >> or something like that. > > Yeah, a clone() could be doable, and is a reasonable idea, but I won't > worry about it for now. If initializing the XPath object is as easy as > putting in a document and two dictionaries (one for namespaces, one > for functions), I think people can do that themselves. I'm looking to > cut out the API we really need first. Fair enough, though I think we do need some way to do the clone with a different [ origin | target node | context element ] for the reasons above. If you want to keep the API down, a way to do this would be to have getters on the namespace, functions _and_modules_ dictionaries, which we could use in the constructor: newContext = XPathContext(doc, oldContext.namespaces, oldContext.extensionFunctions, oldContext.extensionModules) (discussion later on why modules.) I think this is possible, but heavy; this is why I would prefer a convenience function newContext = oldContext.clone() or newContext = XPathContext(context=oldContext) But that is a matter of preference. >> An upside would be the following: Assume that we store the (Pyrex) >> XPathContext object in the void* userData in the (C) xmlXPathContext. >> Recall that (Python) extension functions receive the (C) >> xmlXPathParserContext (as a Pyrex object of course), and hence can >> access the (C) xmlXPathContext, so we could give them access to the >> (Pyrex) XPathContext. >> That way, Kapil can store user data in some object variable in the >> (per-session) clone of the XPathContext; and make all the clones from >> a single one which is configured with namespaces and extensions. > > I'm not sure I understand. Why not provide a new object altogether for > UserData? I don't see why this need be the task of the XPath context. > Could be a third argument to evaluate(), perhaps? See it from the viewpoint of the extension function: All it receives (in C) is the xmlXPathContext. I assume that what the python extension function would receive would be a Pyrex object that wraps that C structure (however we call it!), or a Python object that we would have stored in that C structure (using the userData field). The latter choice actually risks losing information; the (wrapped) xmlXPathContext contains a lot of information that the extension function might need, like namespaces. Hence, I assume that further user data, for Kapil's need, would have to be accessible from this object that the Python function receives, which is still probably the xmlXPathContext wrapper. Hence the above paragraph. >> Also, each document gets to reuse a XPathContext, which means that we >> do not have to set one up each time we evaluate a xpath. > > This is a separate idea from the above user data story, right? I see > this as an optimization of the .xpath method. It would require a way > to see the XPath object of the document to be seeded with namespaces > through a separate API on the document or something.. An alternative > is just to do away with the .xpath method altogether and require > people to use XPath() directly, which would make it harder for people > to make performance mistakes. I'm not sure, the convenience it offers > right now, especially for evaluating in element contexts, is pretty > nice. It is indeed... This is why I was toying with the idea of tying a default XPathContext to a document. Still ambivalent about it. >> Finally, another thing I would add to the xpathContext API is the >> option to declare variables (again as a dictionary) that can be read >> (only) by extension functions. Another way for Kapil to do things... >> though there he only gets a literal or node, not a Python object. >> Still, a literal can also be a key into a thread-global dictionary of >> session objects. > > Having some way to pass along Python objects to extension functions > would be nice. This is separate from the XPath $variable concept, > which we'll also need to support. Yes, you are right, they are distinct. I was musing here. >> Overall, I think this can fly. >>> Registration of functions could work as a dictionary too: >>> >>> xpath.registerFunctions(function_dict) # function name : python >>> func >> Yes... I do like this, but let us look at the XSLT philosophy, which >> assumes modules, before we do too much that is incompatible... So I >> am a tad less sure here. > > I think worrying about XSLT when we get to it would be fine. Sorry, but this is where we really disagree. I also started that way, and then realized that it was a way to get this nice API for extension functions, and then realize that I'd have to build a completely different one for XSLT... Which also involves extension functions, in a very different way. This is a way to seriously get into impedance mismatch with two incompatible APIs that attempt to do the same thing. > A dictionary of extension functions could be turned into some kind of > module, right? No. A module involves a) extension functions b) extension elements (each of which is strictly speaking a pair of functions, but I decided to sweep that under the carpet...) c) a namespace URI d) convenience methods for module initialization, for each document parsed More annoying, modules are registered globally, unlike XPath extension functions. >> The way I see it, we should actually be able to associate extension >> functions and elements with a namespace. A > > Extension elements are a separate story again, right? Can extension > functions have a namespace? Yes, very much so. (You will notice I have them in my toy implementation.) >> module allows that in a neat fashion: >> a module encapsulates a namespace URI, extension functions and >> elements, and management function that are called at beginning/end of >> module setup and document transform respectively. (And yes, I have >> found both of those useful!) >> Also, modules are very much registered globally. (They are associated >> to a local name at the level of an individual transform, however.) >> SO.... >> One option is to actually reuse the existing libxslt.extensionModule >> class. > > I won't allow anything like the libxml2-style APIs near the lxml API. > :) I'm sure there are useful concepts in there, but I first want to > tackle XPath. Then we'll look at XSLT. If we keep the XPath API as > minimal and simple as we can (for the Python developer using lxml), we > should be able to translate some of those concepts into the XSLT API. > Anyway, I'll skip the XSLT part for now until we've implemented an > XPath API. Again, sorry to differ, and it is your project, but I hope you will reconsider. Note that I am very open to thinking of other ways to treat the modules, that is more pythonic and less close to the libxslt; I am however quite convinced that tackling the two projects as separate is a way to to wind up with a two-headed API, or a lot of glue to make it look like a single one. Also, to be honest, though I really admire what has happened so far in lxml to simplify the libxml API, I am not so convinced that the xslt extension API (i.e. using modules) so desperately needs fixing. But as I said, I would love to discuss alternatives if you care to discuss them. (I have not thought of any yet, myself, but I can try.) I hope you will not be offended by my strong opinions in the matter. Regards, Marc-Antoine Parent [View Less]

1 0

[lxml-dev] xpath extension functions
by Martijn Faassen Jan. 28, 2005

Jan. 28, 2005

Hi there, First of all, thanks Kapil and Marc-Antoine for the discussion about xpath extension functions. I hope you don't mind me admitting I'm somewhat overwhelmed. I thought a good way to look at it now would be from the perspective of the API developers see, and write down some concrete use cases. I understand that XPath extension functions are registered against an xmlXPathContext, which in turn is created from a document. Namespaces are potentially registered against this context, … [View More]and you can register extension functions. Currently the xpath API is a simple convenience function against Element and the ElementTree. That is pretty nice and easy, but once you're interested in more sophisticated use cases, it makes more sense to expose a separate XPath object, similar to the way we alread have an XSLT object and a RelaxNG object. I can imagine an interaction like this: >>> from lxml import etree >>> doc = etree.parse('doc.xml') >>> xpath = XPath(doc) >>> xpath.registerNamespaces(namespace_dict) >>> xpath.registerFunction('foo', f) >>> results = xpath.evaluate('//p') Registration of functions could work as a dictionary too: >>> xpath.registerFunctions(function_dict) # function name : python func Or an alternate, simpler, API with less methods, could be this: >>> xpath = XPath(doc, namespace_dict, extension_func_dict) To pass along a context element to an XPath evaluation, we could allow this: >>> results = xpath.evaluate('p', el) We can easily reimplement the existing xpath methods in terms of this functionality. They'd just be convenience functions which you shouldn't use if you're interested in calling a lot of xpath expressions against the same document. If you can have extension functions in dictionaries, registering them for more than one xpath context wouldn't be very hard to do at all. I think that's good enough for now and we don't need to worry hard about global registration. If it turns out we want it, it wouldn't be hard to implement an API to manipulate a global dictionary that gets put in whenever an XPath object is initialized (and then potentially overwritten by the functin dictionary argument). But all that is easy enough to do yourself, so I'd say let's skip it for now. I've skipped over the arguments such an extension function should be able to handle. What kind of a information does an extension function typically need? Let's give it the minimum amount of information needed, and not overwhelm it with libxml2 complexity. :) Exceptions raised in extension functions should definitely pop up to the Python level. I understand that this is complicated to do. Would it be doable to keep the information about the last exception raised on the XPath object? How this ties into XSLT processing I don't know yet. Perhaps we'll have a similar extension function registration API on the XSLT object and that could be enough... I'm not sure how all this ties into various threading issues Kapil brought up. Kapil, would this design work okay for you? Marc-Antoine, thanks again for all the work and please let me what you think about this design, and what I'm still missing. :) Regards, Martijn [View Less]

2 2

[lxml-dev] Some thoughts re XPath extension functions
by Marc-Antoine Parent Jan. 28, 2005

Jan. 28, 2005

OK, I have a first (barely) functional implementation of registerXPathExtensionFunc. (I sent it to Martijn, but it's not ready for checkin until some collective thinking happens.) For one thing, I currently collect functions globally, and register them with the new XPathContext that is created on demand every time the xpath function is called. That operation is unfortunately slow. I think that there are reasons that XPathContext are created on demand: If your extension function calls the … [View More]xpath method, I think it is necessary to use a new context. (Needs checking.) But then that means re-registering extensions, of course... I first thought of keeping a XPathContext with the xmlDoc. It would save creation time, for one thing. But it means that I have to guard against the recursion problems above. (Yes, I have done such things, it is a real situation.) Also, more care to guard against leaks. Do people here think it worth the trouble? And I also thought that the extension functions should be registered with the document, and not globally. Do people agree this is a good thing? More complicated in some ways, but it would allow different documents to have different extension functions registered. Is this useful in real life? I cannot think of a use case. I would definitely like feedback on this issue. Another alternative would be to make people manipulate XPathContext explicitely, and provide it (as an optional argument?) when calling XPaths functions. I think that is ugly, and again I cannot see use cases for using two distinct sets of functions on a single document. Anybody disagrees? On that note, extension functions receive a XPathParserContext, which indirectly gives access to a lot of relevant information that an extension function might need. For example, it gives access to the XPathContext, which can give access to the xsltTransformContext, which gives you access to the node being constructed as a result of a XSL transformation... I have built extension functions that use the latter; and of course the XPathContext tells you the node where the xpath is being evaluated. Is there anything else that people here know they might need from any of those contexts in XPath extension functions? (Not to mention XSL extension elements, when I get there...) Because we would have to put methods/accesors on appropriate Python wrappers around all those. And worry about wrapping and hence releasing these. Sigh. (I am thinking of putting a lot of accessors on the XPathContext only, so we do not have to wrap the other ones in Python and worry about memory management. Objections?) OK, I am through bombarding people for a while, my next step when I look at this next will be to build a test suite... MAP [View Less]

3 7

[lxml-dev] lxml XPath
by Marc-Antoine Parent Jan. 27, 2005

Jan. 27, 2005

>> I have started looking at what is missing for me to use this in my >> own projects, and the only big piece I could find missing for my >> purposes was XPathRegisterFunction. So I am starting to hack at it. >> (I know Python well, libxml2 somewhat though mostly as a user, and I >> am new to Pyrex. Great opportunity to learn, I love it so far.) > > Ah, great, a contributor! :) !!! I am sure those are always scarce. I cannot put tons of time either, but … [View More]

2 1

[lxml-dev] lxml progress
by Martijn Faassen Jan. 14, 2005

Jan. 14, 2005

Hi there, Since some people seem to be actually reading this and some progress has been made, I thought I'd give a report of what's been happening with lxml. * Since last week, I've added a lot more of the ElementTree API, such as the .find() function and friends, by directly using the code from ElementTree. * I actually am running the ElementTree and cElementTree test suites now. I still need to disable some tests, but a significant fraction is indeed running. * I've improved … [View More]the way libxml2's parser functionality gets used, in order to implement libxml2's top-level parse() function. * I've added XPath support to lxml.etree! An example of what you can do: >>> from lxml import etree >>> tree = etree.parse('ot.xml') >>> tree.xpath('(//v)[5]/text()') [u'And God called the light Day, and the darkness he called Night. And the evening and the morning were the first day.\n'] or, say, this, modifying the elements returned: >>> result = tree.xpath('(//v)[5]') >>> result[0].text = 'The day and night verse.' >>> tree.xpath('(//v)[5]/text()') [u'The day and night verse.'] * I've added the start of XSLT support to lxml.etree. An example: test.xslt <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="*" /> <xsl:template match="/"> <day><xsl:value-of select="(//v)[5]" /></day> </xsl:template> </xsl:stylesheet> >>> from lxml import etree >>> style_xml= etree.parse('test.xslt') >>> style = etree.XSLT(style_xml) >>> ot = etree.parse('ot.xml') >>> result = style.apply(ot) >>> style.tostring(result) u'<?xml version="1.0"?>\n<day>And God called the light Day, and the darkness he called Night. And the evening and the morning were the first day.\n</day>\n' >>> result.getroot().tag u'day' A note about performance. I've been in a mail discussion with Fredrik Lundh, the originator of ElementTree, over the past week, doing some performance comparisons. libxml2 is fast, but a bit of my thunder was stolen away by Fredrik when he released cElementTree. cElementTree is certainly no slacker either, and in some cases even beats the snot out of lxml. Fredrik deserves plenty of kudos for that. A bit of a bummer for me though. :) In my measurements (your mileage may vary), cElementTree's parsing is about as fast as lxml at parsing jobs. This on the same benchmark Fredrik's been using. cElementTree is more memory efficient though, though lxml.etree is still better than ElementTree and many other Python/XML tools. Somewhat to my disappointment, cElementTree and even ElementTree are right now a *lot* faster at .find() and friends than lxml.etree. Since they all use the same Python implementation, this means that lxml.etree's implementation of the ElementTree API is in some ways quite a bit slower than Fredrik's Python implementation! Thinking about it more, this is not a big surprise, as lxml.etree does a lot of heavy lifting to make sure the underlying libxml2 tree is exposed with an ElementTree API, and in addition has to worry about doing memory management with these structures. All is not lost however; lxml has xpath! libxml2's xpath is pretty fast; while more slow than (c)ElementTree's .findall() in some cases, it's a lot more powerful as well, being a full xpath implementation. Finally, XSLT seems pretty fast. In a simple test program, I can do a 1000 XSLT transformations in a few seconds, including a reparse of the XSLT stylesheet and document to transform, although granted this was done with a small document. Regards, Martijn [View Less]

1 0

[lxml-dev] lxml.etree (ElementTree reimplementation) is getting there
by Martijn Faassen Jan. 8, 2005

Jan. 8, 2005

Hi there, [resending this, as yesterday's mail never seems to have arrived] The lxml.etree implementation of ElementTree, on top of libxml2, is getting there now. It features automatic memory management and quite a bit of ElementTree compatibility. Not all of the ElementTree API has been implemented yet, but enough for many use cases. I did discover in the process of debugging that you need a recent version of libxml2 to make it all work without memory errors; apparently earlier ones, like … [View More]

1 0

[lxml-dev] test mail
by Martijn Faassen Jan. 8, 2005

Jan. 8, 2005

Hm, Yesterday's mail didn't seem to have arrived, trying a test email. Regards, Martijn

1 0