Re: [lxml-dev] Segfault on XSLT/XPath undefined variable error.
Hi, John Krukoff wrote:
First, uninstalled cython and upgraded to 2.1.1, just to make sure this wasn't already fixed, since there were notes in the changelog about XSLT logging. With the same method, I can still get the same crash:
xmlXPathCompiledEval: evaluation failed Segmentation fault (core dumped)
With symbols, the backtrace has a bit more information:
but still no line numbers ... Could you compile with "-O -ggdb"?
#0 0xb76fb740 in __pyx_f_4lxml_5etree__forwardError () from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #1 0xb76fbb12 in __pyx_f_4lxml_5etree__receiveXSLTError () from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #2 0xb766aef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 #3 0xb766b091 in xsltTransformError () from /usr/lib/libxslt.so.1 #4 0xb768d434 in xsltValueOf () from /usr/lib/libxslt.so.1 #5 0xb768a5ba in ?? () from /usr/lib/libxslt.so.1 #6 0x0828d230 in ?? () #7 0x082bb618 in ?? () #8 0x0828d578 in ?? () #9 0x085db390 in ?? () #10 0x00000000 in ?? ()
Hmmm, I have no idea where in the (pretty short) _forwardError() function it could crash, especially after having run correctly before. Also, I'm wondering why the later stack frames do not show up. Are you sure libxslt is actually called from lxml.etree here? Given your valgrind log, you seem to be using other XML libraries as well. Does any of them use libxslt independently?
I've attached the valgrind log, hopefully it's not too large for the mailing list.
It was. :) I won't forward it, since it's not really helpful to other readers.
Do you want the valgrind core file too? It's 150MB. I'm trying to narrow down what I need to do first to get it to crash, I'll let you know if I come up with a simple test case as a result.
Please check any other libraries for libxslt usage first. The error log setup of lxml.etree is global for each thread, that might already be the problem here. If it is, it *might* be possible to remove the global setup at least for libxslt, but I will have to look into that. To figure out which library interferes here, you can install the debug packages for each library you use in the setup, until you get a complete stack trace above. Stefan
On Thu, 2008-07-31 at 07:41 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
First, uninstalled cython and upgraded to 2.1.1, just to make sure this wasn't already fixed, since there were notes in the changelog about XSLT logging. With the same method, I can still get the same crash:
xmlXPathCompiledEval: evaluation failed Segmentation fault (core dumped)
With symbols, the backtrace has a bit more information:
but still no line numbers ... Could you compile with "-O -ggdb"?
Do I need to alter the setup.py to do that, or is there a simpler way? I think the other library that links against libxml2 is libxmlsec, I'll see if I can get it to crash without calling that first. I suspect I can, as it was failing for my coworker with a different set of things called first, which didn't include the libxmlsec call. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways. I'll work around it by signing the document in a separate process, I think. Anyway, sorry to waste your time, since it looks like you've already fixed the lxml bug my coworker was seeing with the catching the XSLT STOP errors. He hasn't had any problems since upgrading to 2.1 yesterday. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
Hi, John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways.
I'll work around it by signing the document in a separate process, I think.
That should work, yes. However, I think in the specific case of libxslt, I might be able to make the error logging setup local to an XSLT call. It should not be needed anywhere else, so this would reduce the chance for collisions. Thanks for the report anyway, at least it made me aware of such problems. BTW, since you are using libxmlsec (which I never needed), how do you use it together with lxml? (Assuming that it doesn't crash, that is...) Do you think there is space left for a tighter integration, or does your current usage match your use case (well enough)? Stefan
On Thu, 2008-07-31 at 20:13 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways.
I'll work around it by signing the document in a separate process, I think.
That should work, yes. However, I think in the specific case of libxslt, I might be able to make the error logging setup local to an XSLT call. It should not be needed anywhere else, so this would reduce the chance for collisions.
Thanks for the report anyway, at least it made me aware of such problems.
BTW, since you are using libxmlsec (which I never needed), how do you use it together with lxml? (Assuming that it doesn't crash, that is...) Do you think there is space left for a tighter integration, or does your current usage match your use case (well enough)?
Stefan
I've been using this set of python bindings from verisign: http://xmlsig.sourceforge.net/testing.html Which required me to serialize my lxml tree, and then reparse, with the same for the output. I wouldn't recommend it, but then the verisign module itself is hard to recommend since it's both poorly documented and prone to crashing. If I had the time to start over, I'd give these bindings a look, since they at least look to be python specific: http://pyxmlsec.labs.libre-entreprise.org/ They don't look like much more than I'd get with just calling the thing directly with ctypes, and I chose the verisign module since the simpler interface was much easier to get up and running. So, yeah, things could work much better. I've no idea how commonly used the XML signature standard is, I'm implementing it for integration with a vendor that requires it. It seems to be much more of a Java/.NET enterprise kind of thing. From the sample code we've gotten, I am envious of how easy it is to use in .NET, given the standard library support for it. Since I've got some sample code from my vendor in C#, I'm thinking about just calling out to that since I need to move things to another process anyway. The .NET xmldsig stuff appears to work on mono, so I may go that route. Basically, I'm trying to say that as far as I've been able to tell, there's no good interface for doing xmldsig in python, which is actually the first time I've run into that with the language and an open standard. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
Hi, John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways.
could you check if this patch makes it work better for you? It basically restricts XSLT error logging to the lifetime of an XSL transformation. Stefan
On Fri, 2008-08-01 at 08:47 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways.
could you check if this patch makes it work better for you? It basically restricts XSLT error logging to the lifetime of an XSL transformation.
Stefan
I still need to compile lxml with -ggdb, where do I stick that in the setup.py/makefile? Interestingly, this is after I've switched to calling an external C# program to do my xml signing, and am no longer using libxmlsec. But, anyway, still crashed with the patch for me: Core was generated by `/usr/bin/python -tt ./Adapter.py'. Program terminated with signal 11, Segmentation fault. #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError () from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so (gdb) bt #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError () from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #1 0xb774bb12 in __pyx_f_4lxml_5etree__receiveXSLTError () from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #2 0xb76baef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 #3 0xb76bb091 in xsltTransformError () from /usr/lib/libxslt.so.1 #4 0xb76dd434 in xsltValueOf () from /usr/lib/libxslt.so.1 #5 0xb76da5ba in ?? () from /usr/lib/libxslt.so.1 #6 0x0878e718 in ?? () #7 0x08432c60 in ?? () #8 0x0878ec78 in ?? () #9 0x0878f6f8 in ?? () #10 0x00000000 in ?? () Fortunately, I've been able to simplify my crash conditions somewhat, so the valgrind log is significantly shorter. Looks like I'll need to find some time to work on that test case after all. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
On Wed, 2008-08-06 at 12:55 -0600, John Krukoff wrote:
Interestingly, this is after I've switched to calling an external C# program to do my xml signing, and am no longer using libxmlsec.
Whoops, I apologize for the misinformation. It does, in fact, only crash when using libxmlsec, as expected. I had a spurious import in my new code that was pulling in the xmlsig python bindings, even though I was no longer using them. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
Hi, John Krukoff wrote:
On Fri, 2008-08-01 at 08:47 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways. could you check if this patch makes it work better for you? It basically restricts XSLT error logging to the lifetime of an XSL transformation.
Stefan
I still need to compile lxml with -ggdb, where do I stick that in the setup.py/makefile?
Pass the "CFLAGS" env variable when calling setup.py, as in CFLAGS="-O -ggdb" make clean inplace
still crashed with the patch for me:
Core was generated by `/usr/bin/python -tt ./Adapter.py'. Program terminated with signal 11, Segmentation fault. #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so (gdb) bt #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #1 0xb774bb12 in __pyx_f_4lxml_5etree__receiveXSLTError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #2 0xb76baef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 #3 0xb76bb091 in xsltTransformError () from /usr/lib/libxslt.so.1 #4 0xb76dd434 in xsltValueOf () from /usr/lib/libxslt.so.1 #5 0xb76da5ba in ?? () from /usr/lib/libxslt.so.1 #6 0x0878e718 in ?? () #7 0x08432c60 in ?? () #8 0x0878ec78 in ?? () #9 0x0878f6f8 in ?? () #10 0x00000000 in ?? ()
Fortunately, I've been able to simplify my crash conditions somewhat, so the valgrind log is significantly shorter.
You can generally strip the "... within /lib/ld-2.6..." entries from it.
Looks like I'll need to find some time to work on that test case after all.
Please do, and also re-run your valgrind test with -ggdb. There seem to be some interesting problems in there (the "moveNodeToDocument()" sections), even if only crashes later. Seeing the line numbers here would really be helpful. Stefan
On Fri, 2008-08-08 at 08:25 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
On Fri, 2008-08-01 at 08:47 +0200, Stefan Behnel wrote:
Hi,
John Krukoff wrote:
Okay, can only get it to crash when first signing a document using libxmlsec, so I suppose I'll simply assume that the two libraries use the error log in incompatible ways. could you check if this patch makes it work better for you? It basically restricts XSLT error logging to the lifetime of an XSL transformation.
Stefan
I still need to compile lxml with -ggdb, where do I stick that in the setup.py/makefile?
Pass the "CFLAGS" env variable when calling setup.py, as in
CFLAGS="-O -ggdb" make clean inplace
still crashed with the patch for me:
Core was generated by `/usr/bin/python -tt ./Adapter.py'. Program terminated with signal 11, Segmentation fault. #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so (gdb) bt #0 0xb774b740 in __pyx_f_4lxml_5etree__forwardError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #1 0xb774bb12 in __pyx_f_4lxml_5etree__receiveXSLTError ()
from /usr/lib/python2.5/site-packages/lxml-2.1.1-py2.5-linux-i686.egg/lxml/etree.so #2 0xb76baef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 #3 0xb76bb091 in xsltTransformError () from /usr/lib/libxslt.so.1 #4 0xb76dd434 in xsltValueOf () from /usr/lib/libxslt.so.1 #5 0xb76da5ba in ?? () from /usr/lib/libxslt.so.1 #6 0x0878e718 in ?? () #7 0x08432c60 in ?? () #8 0x0878ec78 in ?? () #9 0x0878f6f8 in ?? () #10 0x00000000 in ?? ()
Fortunately, I've been able to simplify my crash conditions somewhat, so the valgrind log is significantly shorter.
You can generally strip the "... within /lib/ld-2.6..." entries from it.
Looks like I'll need to find some time to work on that test case after all.
Please do, and also re-run your valgrind test with -ggdb. There seem to be some interesting problems in there (the "moveNodeToDocument()" sections), even if only crashes later. Seeing the line numbers here would really be helpful.
Stefan
Okay, I can get a segfault with this minimal test case, where variable.xslt is the previously sent XSLT sheet which references an undefined parameter in an xpath expression: Python 2.5.2 (r252:60911, Jul 31 2008, 15:38:58) [GCC 4.1.2 (Gentoo 4.1.2 p1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from lxml import etree import xmlsig etree.XSLT( etree.parse( 'variable.xslt' ) )( etree.XML( '<a/>' ) ) xmlXPathCompiledEval: evaluation failed Segmentation fault
However, changing the import order causes the crash to go away, I assume due to order of initialization in the logging code: ~/Projects/Gizmo/www/Samples 17$ python Python 2.5.2 (r252:60911, Jul 31 2008, 15:38:58) [GCC 4.1.2 (Gentoo 4.1.2 p1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import xmlsig from lxml import etree etree.XSLT( etree.parse( 'variable.xslt' ) )( etree.XML( '<a/>' ) ) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "xslt.pxi", line 529, in lxml.etree.XSLT.__call__ (src/lxml/lxml.etree.c:91033) lxml.etree.XSLTApplyError: XPath evaluation returned no result.
Here is the backtrace from the crash with debugging info: Core was generated by `python'. Program terminated with signal 11, Segmentation fault. #0 0xb78ce53b in __pyx_f_4lxml_5etree__forwardError (__pyx_v_c_log_handler=<value optimized out>, __pyx_v_error=0xbfba01c8) at src/lxml/lxml.etree.c:40855 40855 ((struct __pyx_vtabstruct_4lxml_5etree__BaseErrorLog *)__pyx_v_log_handler->__pyx_vtab)->_receive(__pyx_v_log_handler, __pyx_v_error); (gdb) bt #0 0xb78ce53b in __pyx_f_4lxml_5etree__forwardError (__pyx_v_c_log_handler=<value optimized out>, __pyx_v_error=0xbfba01c8) at src/lxml/lxml.etree.c:40855 #1 0xb78ce79c in __pyx_f_4lxml_5etree__receiveXSLTError (__pyx_v_c_log_handler=0xb75fcf04, __pyx_v_msg=0xb7880450 "%s: file %s line %d element %s\n") at src/lxml/lxml.etree.c:41371 #2 0xb7859ef9 in xsltPrintErrorContext () from /usr/lib/libxslt.so.1 #3 0xb785a091 in xsltTransformError () from /usr/lib/libxslt.so.1 #4 0xb787c434 in xsltValueOf () from /usr/lib/libxslt.so.1 #5 0xb78795ba in ?? () from /usr/lib/libxslt.so.1 #6 0x0812bdb8 in ?? () #7 0x08127100 in ?? () #8 0x08124b60 in ?? () #9 0x08108eb0 in ?? () #10 0x00000000 in ?? () Attached is the valgrind log from the crash. Hope this helps. All of these are with the XSLT error logging patch you sent along previously. -- John Krukoff <jkrukoff@ltgc.com> Land Title Guarantee Company
participants (2)
-
John Krukoff -
Stefan Behnel