[lxml-dev] Crash on Win32 under heavy stress
Hi there, We're doing some stress testing before rolling some code based on lxml into production and we've been able to reproduce a crash when reusing the same XSLT object repeatedly. I will dump some information here in the hope that anyone can shed a light before I go trying to compile debug versions of everything and the kitchen sink. The test is composed of a rather small XML file, and a rather big and complex XSLT with several <xsl:import /> <xsl:include /> etc. We fire up 10 threads. Each one has it's own parsed XML file and XSLT, they are not being shared across threads. Each thread then goes on a loop, applying the XSLT to the XML file and serializing the result to a string. With a less than 1000 iterations the crash almost never happens. At about 50000 iterations, the crash is pretty much guaranteed to happen. There doesn't seem to be any memory leak or anything, memory usage is quite stable. This is using the 1.2.1 release. This is a static build on Win32, against: libxml2-2.6.26.win32 libxslt-1.1.17.win32 zlib-1.2.3.win32 iconv-1.9.2.win32 The crash information says its an 'access violation' and it happens somewhere in etree.pyd, when I bring up the debugger only etree.pyd and python24.dll are on the stack (which I guess is pretty useless information). I am hoping to make a debug build to get more information. I can try running the same code on Linux and seeing if it happens there too, or I can try a debug build, or hopefully someone will come out and say I'm doing something wrong here or that the version of libxml2 I'm using has known issues. I can also provide the test code + data upon request. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Hi Sidnei, I'll be away on vacation starting today, so I regret I can't provide too much critical help at the moment. So, just a quick note here. Sidnei da Silva wrote:
This is using the 1.2.1 release.
For a quick test, try with 1.3beta or the current trunk. No guarantee, though, especially the trunk is not necessarily in a perfect production state.
This is a static build on Win32, against:
libxml2-2.6.26.win32 libxslt-1.1.17.win32 zlib-1.2.3.win32 iconv-1.9.2.win32
Ok, first thing I'd personally try is the latest libxml2 and libxslt. The guys over there keep fixing bugs (even really old ones in recent versions), so things tend to get better over time. For example, I get a reproduceable XPath crasher in the HTML module Ian is working on with libxml2 2.6.27. It's gone with 2.6.28. libxslt is a good bet here, too. Sorry if this doesn't help, but trying other versions is as much as I can propose at the moment, especially if it's urgent. Ah, one last thing: I know, you're not testing threads for fun but for performance, but if that doesn't prove reliable for your application, you can still switch them off with --without-threading. Reliability is usually more important than performance for production. Stefan
Stefan Behnel wrote:
Ok, first thing I'd personally try is the latest libxml2 and libxslt. The guys over there keep fixing bugs (even really old ones in recent versions), so things tend to get better over time. For example, I get a reproduceable XPath crasher in the HTML module Ian is working on with libxml2 2.6.27. It's gone with 2.6.28. libxslt is a good bet here, too.
Which XPath is that? I'd rather avoid it if I can, for those that might have 2.6.27. -- Ian Bicking | ianb@colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers
Hi Ian, Ian Bicking wrote:
Stefan Behnel wrote:
I get a reproduceable XPath crasher in the HTML module Ian is working on with libxml2 2.6.27. It's gone with 2.6.28. libxslt is a good bet here, too.
Which XPath is that? I'd rather avoid it if I can, for those that might have 2.6.27.
It happens in the "clean" doctests, in one of the ".xpath()" method calls. One thing to try might be using XPath() instead. However, looking at the way clean() is implemented, I would rather rewrite a few places to use the already existing loop over getiterator() rather than XPath. Most likely, users will end up requiring the loop to run anyway, so it would be best to use it for more rather than additionally parsing and running a C loop on different XPath expressions. Stefan
Hi Stephan, On 6/1/07, Stefan Behnel <stefan_ml@behnel.de> wrote:
Sidnei da Silva wrote:
This is using the 1.2.1 release.
For a quick test, try with 1.3beta or the current trunk. No guarantee, though, especially the trunk is not necessarily in a perfect production state.
I wish I could compile trunk, but I haven't figured out a resolution to that issue with MSVC. :(
This is a static build on Win32, against:
libxml2-2.6.26.win32 libxslt-1.1.17.win32 zlib-1.2.3.win32 iconv-1.9.2.win32
Ok, first thing I'd personally try is the latest libxml2 and libxslt. The guys over there keep fixing bugs (even really old ones in recent versions), so things tend to get better over time. For example, I get a reproduceable XPath crasher in the HTML module Ian is working on with libxml2 2.6.27. It's gone with 2.6.28. libxslt is a good bet here, too.
I've updated to libxml2-2.6.27 and libxslt-1.1.19 and it hasn't crashed so far. I've mailed Igor Zlatkovic to see if he will be building binaries for the latest libxml2/libxslt anytime soon.
Sorry if this doesn't help, but trying other versions is as much as I can propose at the moment, especially if it's urgent.
That was good advice, and it seems to have solved the immediate issue. Thank you a lot!
Ah, one last thing: I know, you're not testing threads for fun but for performance, but if that doesn't prove reliable for your application, you can still switch them off with --without-threading. Reliability is usually more important than performance for production.
Is there any document describing the effects of --without-threading? I guess it's safe to enable that flag if you are sure you won't share objects between threads? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Hi Sidnei, Sidnei da Silva wrote:
On 6/1/07, Stefan Behnel <stefan_ml@behnel.de> wrote:
Sidnei da Silva wrote:
This is using the 1.2.1 release.
For a quick test, try with 1.3beta or the current trunk. No guarantee, though, especially the trunk is not necessarily in a perfect production state.
I wish I could compile trunk, but I haven't figured out a resolution to that issue with MSVC. :(
Ever tried MinGW? I read in a couple of posts that it works pretty well with Python modules - not sure about Pyrex and setuptools, though.
I've updated to libxml2-2.6.27 and libxslt-1.1.19 and it hasn't crashed so far. I've mailed Igor Zlatkovic to see if he will be building binaries for the latest libxml2/libxslt anytime soon.
Well, if 2.6.27 works for your code, then that's fine. I just said there seem to be certain XPath expressions that make it crash. If you don't hit that problem, you should be on the safe side.
Sorry if this doesn't help, but trying other versions is as much as I can propose at the moment, especially if it's urgent.
That was good advice, and it seems to have solved the immediate issue. Thank you a lot!
It's actually really hard to keep lxml working across various different libxml2 and libxslt versions. We can only try to keep stuff that crashes libxml2 out of lxml itself, but we can't always keep users from writing stuff that crashes certain versions.
Ah, one last thing: I know, you're not testing threads for fun but for performance, but if that doesn't prove reliable for your application, you can still switch them off with --without-threading. Reliability is usually more important than performance for production.
Is there any document describing the effects of --without-threading? I guess it's safe to enable that flag if you are sure you won't share objects between threads?
That's not documented. It just disables all threading code in lxml, meaning, it never frees the GIL and thus prevents libxml2/libxslt code from running concurrently. Stefan
participants (3)
-
Ian Bicking -
Sidnei da Silva -
Stefan Behnel