Mailman 3 August 2006 - lxml - The Python XML Toolkit

Re: [lxml-dev] An intriguing behaviour of xpath in lxml
by Stefan Behnel Aug. 2, 2006

Aug. 2, 2006

Hi Agustin. Agustin Villena wrote: > On 8/1/06, Stefan Behnel wrote: > >> Agustín Villena wrote: > >>> I already know that xpath(".") in the document node works, but is > >>> beyond my understanding why xpath("/") is not implemented. > >> > >> Well, what would you expect it to return? The XPath spec says: > >> > >> """ / selects the document root (which is always the parent of the > >> document element) """ > >> > >> The document element is returned by "/*", so it's the root element of > >> the document in ElementTree. The "document root" itself is not > >> available in the tree model provided by lxml. > > So, depending on the use case, we may consider > a) leaving it as is > b) raise a different exception to make the problem more understandable > c) return None to avoid the exception (not really a good idea, but > would match > the behaviour of the getparent() function) > d) return a node set with the root element (thus diverging from the > spec) > Well, the use case es really simple. I'm engaged in a internal course > teaching XML technologies to my co-workers, and I choose lxml as the > best trade-off between easy of use and power. The problem arises when my > "students" begun toying with xpath... Surprisingly the most common first > case that they tried is xpath("/"), and the Exception really confuses > them, and me. ;) Nice trap. Guess I'd try that first, too. > IMHO the is to paths: > - return a node set with the root element .PRO: is intuitive, CONS: > diverges from the spec It has the advantage of actually returning /something/ useful. It also allows users to access the root ElementTree if they like and thus more or less does what can be expected. I mean, this is a rare case anyway and it is actually well defined, so it would be wrong to raise an exception and thus tell the user "you did something wrong". It's a valid XPath expression and therefore perfectly reasonable to use it. I'll just document the difference and that's it. What I now implemented is: if the document root is returned, find its first child and return it as part of a node set instead. If it's not found, it returns None in the node set, but that shouldn't normally happen. > Thanks for your feedback and the great lxml :) Stefan

3 8

[lxml-dev] Inyecting a default XML namespace in an existing xml?
by Agustín Villena Aug. 2, 2006

Aug. 2, 2006

HI! I'm at the task of processing a bunch of digital signed XMLs. My problem is exemplified in this example: a) The original XMLs were enveloped in a container, that has a default namespace and a signature. The internal XMLs also have their own signature. I doesn't have access to this "envelopes" anymore <Envelope xmlns="http://www.example.org/example"> <Doc> <DocData ID="1"> Some Data </DocData> <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">  </Signature> </Doc> <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">  </Signature> </Envelope> b) Sadly, a 3rd party software "extracted" the internal documents, "forgetting" the envelope's default namespace, therefore inalidating the doc's signatures Example of invalid extracted documents <Doc> <DocData ID="1"> Some Data </DocData> <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">  </Signature> </Doc> What was needed (xml 1) -------------------------------------------------- <Doc xmlns="http://www.example.org/example"> <DocData ID="1"> Some Data </DocData> <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">  </Signature> </Doc> First question: * Is there any way with lxml to add a default namspace to an existing xml-tree Now, I'm trying to patch those messed xmls, injecting the namespace in the nodes that need to belong to the missing namespace, but the result is ugly: python code ------------------------------------------------------------- from lxml import etree NEW_NS = "http://www.example.org/example" doc = etree.parse("no_ns_doc.xml") #add namespace to the root node doc.getroot().tag="{%s}%s" %(NEW_NS,doc.getroot().tag) #add namespace to the first child of the root node, #since we don't want to touch de namespace of the #Signature Node for elem in doc.getroot()[0].getiterator(): elem.tag="{%s}%s" %(NEW_NS,elem.tag) doc.write("ns_patched_doc.xml") result (xml 2) ------------------------------------------------------------- <ns0:Doc xmlns:ns0="http://www.example.org/example"> <ns0:DocData ID="1"> Some Data </ns0:DocData>? <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">  </Signature> </ns0:Doc> I know that xml1 and xml2 are semantically the same, but the customer wants his XMLs as appear in the xml 1 example, or with a less ugly prefix. Is the anyway to force to use a more pretty prefix? Any ideas? Thanks Agustin from lxml import etree NEW_NS = "http://www.example.org/example" doc = etree.parse("no_ns_doc.xml") #add namespace to the root node doc.getroot().tag="{%s}%s" %(NEW_NS,doc.getroot().tag) #add namespace to the first child of the root node, #since we don't want to touch de namespace of the #Signature Node for elem in doc.getroot()[0].getiterator(): elem.tag="{%s}%s" %(NEW_NS,elem.tag) doc.write("ns_patched_doc.xml")

2 3

[lxml-dev] running the tests on the trunk
by Martijn Faassen Aug. 2, 2006

Aug. 2, 2006

Hi there, I have trouble running the tests on the current trunk of lxml: Ran 556 tests in 2.333s FAILED (failures=1, errors=7) A lot of this seems to have to do with this attribute error while running the tests: AttributeError: 'module' object has no attribute 'iterparse' What's going on? Regards, Martijn

2 3

[lxml-dev] An intriguing behaviour of xpath in lxml
by Agustín Villena Aug. 1, 2006

Aug. 1, 2006

Hi! Just a question. Assume the next code from lxml import etree from StringIO import StringIO xmlText = "<test>This is a test</test>" doc = etree.parse(StringIO(xmlText)) root = doc.xpath("/") The last line throws the next exception Not yet implemented result node type: 9 Traceback (most recent call last): File "C:\Archivos de programa\ActiveState Komodo 3.5\lib\support\dbgp\pythonlib\dbgp\client.py", line 1843, in runMain self.dbg.runfile(debug_args[0], debug_args) File "C:\Archivos de programa\ActiveState Komodo 3.5\lib\support\dbgp\pythonlib\dbgp\client.py", line 1538, in runfile h_execfile(file, args, module=main, tracer=self) File "C:\Archivos de programa\ActiveState Komodo 3.5\lib\support\dbgp\pythonlib\dbgp\client.py", line 596, in __init__ execfile(file, globals, locals) File "C:\dev\projects\python\python-xpath-example.py", line 7, in __main__ root = doc.xpath("/") File "c:\python24\lib\site-packages\lxml\etree.pyx", line 485, in etree._ElementTree.xpath File "c:\python24\lib\site-packages\lxml\xpath.pxi", line 75, in etree._XPathEvaluatorBase.evaluate File "c:\python24\lib\site-packages\lxml\xpath.pxi", line 212, in etree.XPathDocumentEvaluator.__call__ File "c:\python24\lib\site-packages\lxml\xpath.pxi", line 108, in etree._XPathEvaluatorBase._handle_result File "c:\python24\lib\site-packages\lxml\extensions.pxi", line 269, in etree._unwrapXPathObject File "c:\python24\lib\site-packages\lxml\extensions.pxi", line 317, in etree._createNodeSetResult NotImplementedError My question is: which is the reason behind this behaviour (if is there one)? (I already know that xpath(".") in the document node works, but is beyond my understanding why xpath("/") is not implemented. Cheers Agustin

3 3

Re: [lxml-dev] lxml - exslt - regexp:match()
by Stefan Behnel Aug. 1, 2006

Aug. 1, 2006

Michael Zeidler wrote: > regexp:match('123abc567','([0-9]+)([a-z]+)([0-9]+)') gibt kein Arry mit den gematchten > Gruppen zurück: > Wenn ich also mit > <xsl:variable name="test" select="regexp:match('123abc567','([0-9]+)([a-z]+)([0-9]+)')"/> > die variable $test setzte, müsste ich mit $test[0], $test[1], usw. auf die gematchten gruppen > zugreifen können. Siehe http://www.exslt.org/regexp/functions/match/index.html > > [translation]: > > regexp:match('123abc567','([0-9]+)([a-z]+)([0-9]+)') > > does not return an array containing the matched groups. Something like this > <xsl:variable name="test" select="regexp:match('123abc567','([0-9]+)([a-z]+)([0-9]+)')"/> > should allow me to ask for "$test[0]" etc. > > See http://www.exslt.org/regexp/functions/match/index.html Hmm, interesting. The page doesn't actually say that this is supposed to work. All they provide is an example with a /single/ group. The result of your test case is not defined. For comparison, I now implemented the examples from the page as unit tests, which sadly showed that Python's regexps are incompatible with what EXSLT requires. The Python RE "([a-z])+ " does not match "test " as in EXSLT, only the last "t" is returned for the group by re.findall(). So we can't claim compatibility with EXSLT at this point. -- Note, though, that I never really said it was compatible, it just builds on Python's re module. I still think that's enough for a Python XML library. That said, I fixed your use case in the current trunk, as I think it makes sense to expect the result above from such a call. Note, however, that EXSLT dictates that the first element in a non-global RE result (without 'g' flag) must be the entire string that matched, which even fits the semantics of the group() method in Python's MatchObjects. So your $test[0] will contain "123abc567", $test[1] is "123" etc. Stefan

2 2

Re: [lxml-dev] Segfault in lxml during element copy
by Stefan Behnel Aug. 1, 2006

Aug. 1, 2006

Hi John, John Krukoff wrote: > Thanks for the response. Yeah, I know just how vague an error report > this is. I was really hoping I was hitting something that someone else > had already encountered. I've already wasted a day trying to strip the > program down to just the lxml operations, and haven't been able to come > up with a reduced set of the program that still causes the crash. Try to think about the main treatments you apply to trees. Do you move elements between trees? What happens to the source tree? Does the crash go away if you keep a reference to it? (maybe in a set or list) Do you keep cyclic references between objects that reference elements, i.e. is the Python cyclic garbage collector involved in cleaning up XML trees? If you use XSLT, can you reproduce the crash if you build the result tree (or a simpler one) by hand? Do you use XPath calls or extension functions? Are they required to trigger the crash? These kinds of bugs are mostly related to garbage collection and Python reference counting, so try to concentrate on code that results in freeing references to elements and trees. There is also a tool we commonly use to debug memory handling in lxml.etree. It's called "valgrind". doc/valgrind.txt contains a command line that allows you to run lxml with it. This gives you a stack trace when problems occur or when the program crashes that *might* give us a hint on what happened. In case you want to try, you can send me the output in private e-mail (preferably bzip2-ed or gzipped) so that I can take a look at it. > I'll spend another day on this, and see if I can't do better. Thanks, we really appreciate this kind of help. Stefan

1 0

Re: [lxml-dev] Copying an ElementTree doesn't work.
by Stefan Behnel Aug. 1, 2006

Aug. 1, 2006

Hi John, John Krukoff wrote: > Quoting Stefan Behnel <behnel_ml(a)gkec.informatik.tu-darmstadt.de>: >> John Krukoff wrote: >>> Can someone explain to me why when an ElementTree is copied, it's root >>> element isn't copied? >>> >>>>>> import lxml.etree as etree >>>>>> import copy >>>>>> root = etree.XML( '<a/>' ) >>>>>> tree = copy.copy( etree.ElementTree( root ) ) >>>>>> tree.getroot( ) is None >>> True >> >> As ElementTrees are immutable, the above is not different from this: >> >> tree = etree.ElementTree(root) >> >> I'll add __copy__ and __deepcopy__, though, so that the above problem >> will disappear. So, thanks for reporting this. > > For what it's worth, the use case is that I have an element tree that I > want to copy multiple times, before performing destructive changes to > the copies. Currently, copying the contents of an element tree to > another element tree is kind of clunky: > >>>> original = etree.ElementTree( etree.XML( '<a/>' ) ) >>>> copied = etree.ElementTree( copy.copy( original.getroot( ) ) ) > > which is why I was asking if the expected use is to always pass around > elements and wrap them with element trees only when it was convient to > use the element tree methods (XSLT being what I'm interested in). > > So, thanks, the fix will make this look a little less ugly. Ok, sure. Just for code clarity, you might still want to use deepcopy() instead of copy(), not everybody is necessarily aware of the fact that lxml implements them the same way. Note also that copying an ElementTree actually now produces a shallow copy of the ElementTree. The XML tree is not touched in this case. Here is the patch, BTW, in case you want to apply it yourself. It will be in lxml 1.0.3 and 1.1, which are expected not too late this month. Stefan Index: src/lxml/etree.pyx =================================================================== --- src/lxml/etree.pyx (Revision 30633) +++ src/lxml/etree.pyx (Arbeitskopie) @@ -395,6 +395,15 @@ """ return self._context_node + def __copy__(self): + return ElementTree(self._context_node) + + def __deepcopy__(self, memo): + if self._context_node is None: + return ElementTree() + else: + return ElementTree( self._context_node.__copy__() ) + property docinfo: """Information about the document provided by parser and DTD. This value is only defined for ElementTree objects based on the root node

1 0

[lxml-dev] Segfault in lxml during element copy
by John Krukoff Aug. 1, 2006

Aug. 1, 2006

I've been working on an XML based middleware system written in python and lxml, and I've started experiencing a segfault problem with lxml just as it's being rolled out to the rest of the team. Embarrassing, you know? It looks like a double free problem, as the crash is always acompanied by a glibc message that looks like this: *** glibc detected *** free(): invalid pointer: 0x0813e1a4 *** I've tried to come up with a stripped down test case to repeat the problem, but have been unable to reproduce it except in the full application. It's not absolutely consistent, I'll have to run the same request 3 or 4 times before it crashes, but it always does, even while generating identical output from identical input for those 3 or 4 calls. I've tracked down the line it crashes at, and it's a simple copy called on an XML element: copied = copy.copy( element ) If I remove it, and operate on the source xml directly instead of copying it (it's really just a safety mechanism), it still crashes, just in more random locations. I'm running lxml 1.0.2, on Python 2.4.3, with libxml2 2.6.26 and libxslt 1.1.17 if it matters. The problem is reproducible on a coworkers machine, also running lxml 1.0.2 with slightly different minor revisions of the xml libraries.

2 1

[lxml-dev] Copying an ElementTree doesn't work.
by John Krukoff Aug. 1, 2006

Aug. 1, 2006

Can someone explain to me why when an ElementTree is copied, it's root element isn't copied? >>> import lxml.etree as etree >>> import copy >>> root = etree.XML( '<a/>' ) >>> tree = copy.copy( etree.ElementTree( root ) ) >>> tree.getroot( ) is None True I get the same behaviour with deepcopy as well. Am I just supposed to always be using Element s and not ElementTree s? I'm running lxml 1.0.2 on Python 2.4.3, if that matters.

2 1