Hello, Some months ago, I migrated some part of our code from libxml2/libxslt modules to lxml module (which is much pythonic!). But my code never worked, because of a seg. fault :o( The crash occurs at: result = self.__xslFilter.apply(tree, {'filename':"'%s'" % baseName}) This code is used inside a Pyro server (Pyro is a client/server framework. see http://pyro.sourceforge.net), so in a multi-threaded environement. I ran the server using strace to see what happens: I got a ERESTARTNOHAND error. This error is very strange, and Google does not give usefull informations... I tested my code out of Pyro: it works. I made a simple Pyro serveur with just this part of the code: it works too. It only crashes when using the real server, which uses a lot of multi-threads calls. Another guy had a similar problem, but the error was in the python mysql client wrapper, only on stressed usage. It seems that there is a problem in either lxml and pymysql when used in a stressed multi-threaded way. I use locks in my server to avoid re-entrant calls to lxml, so I'm sure the problem is not there; I feel that it is related to the number of threads running... Does anyone has an idea of that problem? Thanks, PS: I also use lxml from mod_python, and all is working fine. -- Frédéric
Salut! Frédéric Mantegazza wrote:
Some months ago, I migrated some part of our code from libxml2/libxslt modules to lxml module (which is much pythonic!). But my code never worked, because of a seg. fault :o(
The crash occurs at:
result = self.__xslFilter.apply(tree, {'filename':"'%s'" % baseName})
This code is used inside a Pyro server (Pyro is a client/server framework. see http://pyro.sourceforge.net), so in a multi-threaded environement.
I ran the server using strace to see what happens: I got a ERESTARTNOHAND error. This error is very strange, and Google does not give usefull informations...
I tested my code out of Pyro: it works. I made a simple Pyro serveur with just this part of the code: it works too. It only crashes when using the real server, which uses a lot of multi-threads calls.
Another guy had a similar problem, but the error was in the python mysql client wrapper, only on stressed usage.
It seems that there is a problem in either lxml and pymysql when used in a stressed multi-threaded way. I use locks in my server to avoid re-entrant calls to lxml, so I'm sure the problem is not there; I feel that it is related to the number of threads running...
Does anyone has an idea of that problem?
Which version of lxml are you using? Could you try with 1.3.4? You can also build lxml from sources and pass the "--without-threading" option, which might work for you. Stefan
Le jeudi 30 août 2007 21:33, Stefan Behnel a écrit :
Which version of lxml are you using?
1.1.1 (debian etch package). Some weeks ago, I tried the 1.3.0, without success.
Could you try with 1.3.4?
Yes, sure, I'll do that.
You can also build lxml from sources and pass the "--without-threading" option, which might work for you.
Ok. What is the impact on the usage, without threading? Do I have to change something in my code? Thanks, -- Frédéric
Frédéric Mantegazza wrote:
Le jeudi 30 août 2007 21:33, Stefan Behnel a écrit :
Which version of lxml are you using?
1.1.1 (debian etch package). Some weeks ago, I tried the 1.3.0, without success.
Then 1.3.3 and later should work much better for you. You can even try 2.0alpha1, which will (hopefully) be released today. It has quite some robustness improvements in the XPath code.
You can also build lxml from sources and pass the "--without-threading" option, which might work for you.
Ok. What is the impact on the usage, without threading? Do I have to change something in my code?
No, it's just an internal switch that prevents lxml from freeing the GIL, so everything will run serialised (unless you call back into Python code). Stefan
Le jeudi 30 août 2007 21:33, Stefan Behnel a écrit :
Which version of lxml are you using? Could you try with 1.3.4? You can also build lxml from sources and pass the "--without-threading" option, which might work for you.
I tried 1.3.4: no more success, even with the "--without-threading" option... I'll give a try to the 2.0a1 when it is released. But I have to investigate a little deeper, because I'm wondering if the ERESTARTNOHAND is not a consequence of another problem... -- Frédéric
Le vendredi 31 août 2007 10:38, Frédéric Mantegazza a écrit :
But I have to investigate a little deeper, because I'm wondering if the ERESTARTNOHAND is not a consequence of another problem...
Ok, I would like to see exaclty when the error occurs inside the apply() method... I'm not familiar with pyrex... What can I use to print debug messages in lxml-1.3.4/src/lxml/xslt.pxi, in the __call__() methode of XLST class? PS: BTW, my call to XSLT.apply() is made from a thread which is itself launched by a thread. Could it be a problem for lxml? As I said, I was previously using python-libxslt1 and python-libxml2 without any problem, under the same conditions... -- Frédéric
Frédéric Mantegazza wrote:
Le vendredi 31 août 2007 10:38, Frédéric Mantegazza a écrit :
But I have to investigate a little deeper, because I'm wondering if the ERESTARTNOHAND is not a consequence of another problem...
Ok, I would like to see exaclty when the error occurs inside the apply() method...
I'm not familiar with pyrex... What can I use to print debug messages in lxml-1.3.4/src/lxml/xslt.pxi, in the __call__() methode of XLST class?
Most Python code is also valid Pyrex/Cython code. Just use "print".
PS: BTW, my call to XSLT.apply() is made from a thread which is itself launched by a thread. Could it be a problem for lxml? As I said, I was previously using python-libxslt1 and python-libxml2 without any problem, under the same conditions...
The problem is not how many threads there are but where you create an XSLT object and where you use it. Try creating it in the thread where you use it. Mostly for performance reasons, lxml behaves quite different from python-libxml2. Stefan
Le vendredi 31 août 2007 11:31, Stefan Behnel a écrit :
The problem is not how many threads there are but where you create an XSLT object and where you use it. Try creating it in the thread where you use it.
Ok, I got it! I was retreiving my filters from a FilterManager using a get() method, which was returning a deepcopy of the filters, which does not work. It is not surprising, as lxml is written in C (pyrex): I think pointers and other references where lost in the operation. BTW, my XSLT objects are created in a thread, and used in another, but it does not seems to be a problem. Stefan, thanks your very much for the usefull suggestions :o) -- Frédéric
Frédéric Mantegazza wrote:
Le vendredi 31 août 2007 11:31, Stefan Behnel a écrit :
The problem is not how many threads there are but where you create an XSLT object and where you use it. Try creating it in the thread where you use it.
Ok, I got it! I was retreiving my filters from a FilterManager using a get() method, which was returning a deepcopy of the filters, which does not work. It is not surprising, as lxml is written in C (pyrex): I think pointers and other references where lost in the operation.
Ah, right, deep copying XSLT objects doesn't work. I'd say it should work, though. I'll look into it.
BTW, my XSLT objects are created in a thread, and used in another, but it does not seems to be a problem.
It should work in most cases, but it can be a source of problems in others. If it works for you, you should be fine.
Stefan, thanks your very much for the usefull suggestions :o)
J't'en prie. Stefan
participants (2)
-
Frédéric Mantegazza -
Stefan Behnel