Mailman 3 [lxml-dev] Crash with lxml on Mac OS X - lxml - The Python XML Toolkit

[lxml-dev] Crash with lxml on Mac OS X

Piet van Oostrum

10 Jun 2006 10 Jun '06

4:14 p.m.

I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: from lxml import etree root = etree.Element("html") head = etree.SubElement(root, "head") title = etree.SubElement(head, "title") title.text = "Page Title" body = etree.SubElement(root, "body") body.set("bgcolor", "#ffffff") body.text = "Hello, World!" tree = etree.ElementTree(root) tree.write("page.xhtml") tree = etree.parse("page.xhtml") print tree.findtext("head/title") The error messages are: TEMP> python testlxml.py python(3059) malloc: *** Deallocation of a pointer not malloced: 0x2; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0xa86c1dbc; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0x97e90; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug python(3059) malloc: *** Deallocation of a pointer not malloced: 0x80; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Page Title Modules/gcmodule.c:215: failed assertion `gc->gc.gc_refs == GC_REACHABLE' Abort trap The file is created correctly, however. When I insert some print messages, some or all of the errors disappear. When I split the program in two, one writing and another reading the file they run without problems. I have installed through darwinports: libxml2 @2.6.23_0 libxslt @1.1.15_0 I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors. -- Piet van Oostrum URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet@vanoostrum.org

Show replies by date

Stefan Behnel

10 Jun 10 Jun

8:53 p.m.

Hi Piet, Piet van Oostrum wrote:

...

I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: [snip]

I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors.

Hmm, I can't reproduce that. I ran your program with both 1.0.1 and the current trunk and neither of them show any errors (not even under valgrind). Could you install the official 1.0.1 egg and try again? There were a number of memory bug fixes since the 1.0 release and I don't know which SVN version of lxml you tried (it tells you when you print lxml.etree.LXML_VERSION). Stefan

Piet van Oostrum

10:47 p.m.

...

...
...
...
...
Stefan Behnel (SB) wrote:

...

SB> Hi Piet, SB> Piet van Oostrum wrote:

...
...
I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: SB> [snip]

I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors.

...

SB> Hmm, I can't reproduce that. I ran your program with both 1.0.1 SB> and the current trunk and neither of them show any errors (not SB> even under valgrind).

...

SB> Could you install the official 1.0.1 egg and try again? There SB> were a number of memory bug fixes since the 1.0 release and I SB> don't know which SVN version of lxml you tried (it tells you when SB> you print lxml.etree.LXML_VERSION).

The SVN version:

...

...
...
import lxml.etree lxml.etree.LXML_VERSION (1, 0, 1, 28613)

There is no official egg for macosx. I removed it and installed it again with easy_install. And now it only gives 'Bus Error'. I did it with both gcc 4.0 and 3.3

...

...
...
import lxml.etree lxml.etree.LXML_VERSION (1, 0, 1, 0)

-- Piet van Oostrum URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet@vanoostrum.org

Stefan Behnel

11:25 p.m.

Hi Piet, Piet van Oostrum wrote:

...

...
SB> Hi Piet, SB> Piet van Oostrum wrote:

...
...
I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: SB> [snip] I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors.

The SVN version:

...
...
...
import lxml.etree lxml.etree.LXML_VERSION (1, 0, 1, 28613)

That's new enough, thanks. The last number is the SVN revision.

...

There is no official egg for macosx.

Sorry, my fault. We don't have one for 1.0.1, only for 1.0 so far.

...

I removed it and installed it again with easy_install. And now it only gives 'Bus Error'. I did it with both gcc 4.0 and 3.3

That's most likely just the same error. Since I can't reproduce this here (tried Python 2.4 with libxml2 2.6.23 and 2.6.24), could anyone of the other MacOS users please check that this is reproducible on other machines? Or someone with Python 2.3? Piet, this looks like a garbage collection bug. Normally, these things happen when object go out of scope, like the 'tree', when you replace it with the newly parsed one. I assume this does not happen when you parse the tree into a variable with a different name like 'parsed_tree'. Does this only happen for findtext or also when you access the text (or an element) directly? Stefan

Piet van Oostrum

11 Jun 11 Jun

3:48 a.m.

...

...
...
...
...
Stefan Behnel (SB) wrote:

...

SB> Piet, this looks like a garbage collection bug. Normally, these SB> things happen when object go out of scope, like the 'tree', when SB> you replace it with the newly parsed one. I assume this does not SB> happen when you parse the tree into a variable with a different SB> name like 'parsed_tree'. Does this only happen for findtext or SB> also when you access the text (or an element) directly?

Stefan, I think it is more elementary than garbage collection. When I use a different variable for the parsed tree, it still crashes. In fact this program works: from lxml import etree tree1 = etree.parse("page.xhtml") print tree1.findtext("head/title") but this crashes: from lxml import etree root = etree.Element("html") tree1 = etree.parse("page.xhtml") I am not familiar with Pyrex, so looking in the source doesn't help me very much. -- Piet van Oostrum URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet@vanoostrum.org

Stefan Behnel

11:37 a.m.

Hi Piet, Piet van Oostrum wrote:

...

I think it is more elementary than garbage collection. When I use a different variable for the parsed tree, it still crashes. In fact this program works:

from lxml import etree tree1 = etree.parse("page.xhtml") print tree1.findtext("head/title")

but this crashes:

from lxml import etree root = etree.Element("html") tree1 = etree.parse("page.xhtml")

That is very weird behaviour. I can see the difference between the two, the first one calls the parser first and the second one builds a document in memory before calling the parser. So there are differences in setting up the string dictionary that is used to share tag names and things like that. However, I still cannot reproduce this and I have no idea why this should fail. It works just fine on my machine. I attached a debug patch against the SVN branch of 1.0.1 (branch/lxml-1.0) that prints traces from the code path. Could you check that it outputs this for the second example: CREATE2 STORING2 DICT FREE1 DICT REF1 DEALLOC 5848768 DEALLOC 5346432 The exact numbers behind DEALLOC do not matter. I called it like this from the lxml directory: make clean inplace && PYTHONPATH=src python -c \ 'from lxml import etree; root = etree.Element("html"); \ tree1 = etree.parse("page.xhtml")' We also have a Python 2.4 egg of 1.0.1 for MacOS now, so if you want to try that (in case you have Py2.4 available)...

Piet van Oostrum

7:37 p.m.

...

...
...
...
...
Stefan Behnel (SB) wrote:

...

SB> Hi Piet, SB> Piet van Oostrum wrote:

...
...
I think it is more elementary than garbage collection. When I use a different variable for the parsed tree, it still crashes. In fact this program works:

from lxml import etree tree1 = etree.parse("page.xhtml") print tree1.findtext("head/title")

but this crashes:

from lxml import etree root = etree.Element("html") tree1 = etree.parse("page.xhtml")

...

SB> That is very weird behaviour. I can see the difference between SB> the two, the first one calls the parser first and the second one SB> builds a document in memory before calling the parser. So there SB> are differences in setting up the string dictionary that is used SB> to share tag names and things like that. However, I still cannot SB> reproduce this and I have no idea why this should fail. It works SB> just fine on my machine.

Hi Stefan, I think I have solved the mystery.

...

SB> I attached a debug patch against the SVN branch of 1.0.1 (branch/lxml-1.0) SB> that prints traces from the code path. Could you check that it outputs this SB> for the second example:

When applying your patch to the SVN version I found out that your parser.pxi is quite different from mine (the patch failed). You seem to have an older version. I manually applied the patch, but then something odd struck me. It has this definition: cdef void _initDocDict(self, xmlDoc* result): "Store dict of last object parsed if no shared dict yet" cdef _ParserContext context if result is NULL: return context = self._findThreadParserContext() if context._c_dict is NULL: #print "storing shared dict" if result.dict is NULL: if self._c_dict is NULL: result.dict = xmlparser.xmlDictCreate() else: result.dict = xmlparser.xmlDictCreateSub(self._c_dict) context._c_dict = result.dict xmlparser.xmlDictReference(context._c_dict) elif result.dict != context._c_dict: if result.dict is not NULL: xmlparser.xmlDictFree(result.dict) result.dict = context._c_dict xmlparser.xmlDictReference(result.dict) Notice that most of the _c_dict references are context._c_dict but there are two self._c_dict. Probably forgotten to be changed. I changed them to context._c_dict and with that the crash disappears. Then I compiled the version from the 1.0.1 source tarball and now that also works. I now remove the existing version before installing a new one. I noticed that otherwise the install process might not overwrite the existing version. I think this is a result of the egg-based install. Apparently 1.0.1 has solved some memory problems, but the SVN version introduced some new ones :=(

...

SB> We also have a Python 2.4 egg of 1.0.1 for MacOS now, so if you want to try SB> that (in case you have Py2.4 available)...

No I don't have 2.4 installed. I can live with 2.3.5 up to now, and might directly jump to 2.5. -- Piet van Oostrum URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet@vanoostrum.org

Stefan Behnel

9:29 p.m.

Hi Piet, Piet van Oostrum wrote:

...

When applying your patch to the SVN version I found out that your parser.pxi is quite different from mine (the patch failed). You seem to have an older version.

Not older, just different. You used the trunk, not the 1.0 branch. The trunk is currently in a somewhat experimental state and will eventually become 1.1 (though maybe "1.1 alpha" first).

...

I manually applied the patch, but then something odd struck me. It has this definition:

cdef void _initDocDict(self, xmlDoc* result): "Store dict of last object parsed if no shared dict yet" cdef _ParserContext context if result is NULL: return context = self._findThreadParserContext()

Yup, the trunk now has threading support, which required some changes under the hood. The current trunk is actually quite different from what was released as 1.0.1.

...

if context._c_dict is NULL: #print "storing shared dict" if result.dict is NULL: if self._c_dict is NULL: result.dict = xmlparser.xmlDictCreate() else: result.dict = xmlparser.xmlDictCreateSub(self._c_dict) context._c_dict = result.dict

[snip]

...

Notice that most of the _c_dict references are context._c_dict but there are two self._c_dict. Probably forgotten to be changed. I changed them to context._c_dict and with that the crash disappears.

That's not quite the right patch, but thanks for pointing me to that. There actually is a bug hidden in this code that can appear under the circumstances you described. If this is called from the main thread before the parser dictionary is initialised, "context" will equal "self", meaning that the dict is set twice and the first reference will disappear. I'll fix that.

...

Then I compiled the version from the 1.0.1 source tarball and now that also works.

That's good news.

...

I now remove the existing version before installing a new one. I noticed that otherwise the install process might not overwrite the existing version. I think this is a result of the egg-based install.

I think so, too. I guess you accidentally installed the trunk version (which also currently calls itself 1.0.1, guess I'll change that to avoid confusion). This is normally not a problem, unless you install from different sources that use the same version number - like the current trunk, the branch and the 1.0.1 release...

...

Apparently 1.0.1 has solved some memory problems, but the SVN version introduced some new ones :=(

That's ok, 1.1 is not even close to being released, so these bugs will hopefully be found before we get there (just like the one you found above). Sorry for the confusion. I'll change the trunk version to 1.1.alpha to prevent similar problems in the future. Stefan

Stefan Behnel

12 Jun 12 Jun

12:25 a.m.

Hi Piet, Piet van Oostrum wrote:

...

Apparently 1.0.1 has solved some memory problems, but the SVN version introduced some new ones :=(

if you want, you can test with the current trunk version. It will compile into 1.1alpha-SVNREVISION. It should no longer have the problem you found - otherwise I'd really have to dig back into the code... Stefan

Paul Everitt

11 Jun 11 Jun

12:40 p.m.

Stefan Behnel wrote:

...

Hi Piet,

Piet van Oostrum wrote:

...
...
SB> Hi Piet, SB> Piet van Oostrum wrote:

...
...
I installed lxml on Mac OSX Tiger (10.4.7) with the standard Python 2.3.5 and the following simple program crashes: SB> [snip] I have tried it with the 1.0 egg from cheeseshop, source install from the 1.0 source, and the 1.0.1 svn source, with Pyrex 0.9.4.1. They all give the same errors. The SVN version: import lxml.etree lxml.etree.LXML_VERSION (1, 0, 1, 28613)

That's new enough, thanks. The last number is the SVN revision.

...
There is no official egg for macosx.

Sorry, my fault. We don't have one for 1.0.1, only for 1.0 so far.

...
I removed it and installed it again with easy_install. And now it only gives 'Bus Error'. I did it with both gcc 4.0 and 3.3

That's most likely just the same error.

Since I can't reproduce this here (tried Python 2.4 with libxml2 2.6.23 and 2.6.24), could anyone of the other MacOS users please check that this is reproducible on other machines? Or someone with Python 2.3?

I tried it with my own Python 2.4.x build on OS X. No crash. Piet, should I try with the built-in Python? --Paul

Stefan Behnel

1:14 p.m.

Hi Piet, Piet van Oostrum schrieb:

...

I have installed through darwinports: libxml2 @2.6.23_0 libxslt @1.1.15_0

You can verify that these are correctly picked up by lxml with the attributes lxml.etree.LIBXML_VERSION and lxml.etree.LIBXML_COMPILED_VERSION. Given the fact that it works for Paul, it might be a problem with the compiler setup and/or the search path for the system libraries. Please make sure that both attributes show the same version. Stefan

6528

Age (days ago)

6529

Last active (days ago)

List overview

Download

10 comments

3 participants

participants (3)

Paul Everitt
Piet van Oostrum
Stefan Behnel

[lxml-dev] Crash with lxml on Mac OS X

Piet van Oostrum

Stefan Behnel

Piet van Oostrum

Stefan Behnel

Piet van Oostrum

Stefan Behnel

Piet van Oostrum

Stefan Behnel

Stefan Behnel

Paul Everitt

Stefan Behnel

tags

participants (3)