Hi Martijn, Martijn Faassen wrote:

Attached is a small tarball that demonstrates code that crashes when the code is run in a thread but doesn't crash when it is run stand-alone. I isolated the specific XSLT + XML combination that seems to trigger this crash. I suspect it has to do with passing an XSLT object to a thread.

I've seen enough of these all over the place to consider this possible. ;) I'll look into this as soon as I get to it. I was about to release another beta anyway - the latest changelog has gotten longer than I expected, and I really love being able to say that lxml is now fully Py3 compatible. So I'll see if I can get this to work before putting out a 2.2beta4. The still-future Cython 0.11 has also matured a lot by now, so it's worth another release.

I run this with lxml 2.1.5 in Python 2.4, libxml2 2.6.32 and libxslt 1.1.24

Just in case, if the crash is related to transformation errors, you might want to try with 2.2beta3, or even with the trunk, if you also install the latest trunk Cython (sorry for that).

By the way, the FAQ implies that passing an XSLT object into a thread will slow things down (probably as the XSLT would be re-interpreted). Is that still true in the current codebase? I had the impression from previous discussions that this would change.

Yes, the (ugly) code section that this statement was referring to was killed somewhere in 2.1.x. I removed the paragraph from the FAQ and also clarified a couple of other things while at it. lxml now even has a working test case for passing trees along a thread pipeline, so the safety of threading really has improved a lot lately. It's impressively hard to get these things right. Threads are just plain evil. Their only excuse in lxml is that XML handling is often I/O expensive and can involve major time consuming operations inside libxml2 and libxslt (XSLT is really a great candidate for that). So freeing the GIL when we know we are about to do most of our work outside of the Python interpreter gets you pretty far. Stefan

Hi, Martijn Faassen wrote:

Attached is a small tarball that demonstrates code that crashes when the code is run in a thread but doesn't crash when it is run stand-alone. I isolated the specific XSLT + XML combination that seems to trigger this crash. I suspect it has to do with passing an XSLT object to a thread.

Ok, this is plain evil. What you do here is this: ... <tr class="odd"> <xsl:attribute name="class">top-row</xsl:attribute> ... Note how the attribute value is changed after being set. In libxslt, this leads to a result tree update that removes the old attribute and replaces it by the new one. In your case, the stylesheet that was parsed outside the thread inherits the name dict from the main thread, while the input document inherits the one from the worker thread that executes this function: def render(id, xml, stylesheet): doc = etree.parse(StringIO(xml)) result_tree = stylesheet(doc) So the first "class" attribute name comes from the stylesheet dict and gets stored in the result document that inherits the thread dict of the input document. When it is overwritten and deleted, it is looked up in the thread dict, is not found there, and thus free()-ed, although it continues to 'live' in the stylesheet dict. This must really be the only place in XSLT where the result document is not only created incrementally but where its existing content gets overwritten. For now, I really do not know how to work around this. There can only be one dict for the result document, but the original attribute can come from the stylesheet or the input document (or even the current thread dict where the XSLT is executed), and the dict lookup happens from deep inside libxslt. I'm very open to ideas. Stefan