PyMem_Malloc() vs PyObject_Malloc()
I noticed something (in 2.5) yesterday, which may be a feature, but is more likely a bug. In tokenizer.c, tok->encoding is allocated using PyMem_MALLOC(). However, this then gets handed to a node->r_str in parsetok.c, and then released in node.c using PyObject_Free(). Now, by coincidence, PyObject_Free() will default to free() for objects that it doesn't recognize, so this works. But is this documented behavior? The reason I ran into this was that I had redirect the PyMem_* API to a different allocator, but left the PyObject_* one alone. My feeling Is that these two APIs shouldn't be interchangeable. Especially since you can't hand a PyObject_Malloc'd object to PyMem_Free() so the inverse shouldn't be expected to work. Any thoughts? Kristján
Kristján Valur Jónsson wrote:
I noticed something (in 2.5) yesterday, which may be a feature, but is more likely a bug. In tokenizer.c, tok->encoding is allocated using PyMem_MALLOC(). However, this then gets handed to a node->r_str in parsetok.c, and then released in node.c using PyObject_Free().
Now, by coincidence, PyObject_Free() will default to free() for objects that it doesn't recognize, so this works. But is this documented behavior? The reason I ran into this was that I had redirect the PyMem_* API to a different allocator, but left the PyObject_* one alone.
My feeling Is that these two APIs shouldn't be interchangeable. Especially since you can't hand a PyObject_Malloc'd object to PyMem_Free() so the inverse shouldn't be expected to work.
Any thoughts?
This is a bug. Please file a bug report for this. In general, either PyObject_* xor PyMem_* memory API should used. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 04 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Kristján Valur Jónsson wrote:
My feeling Is that these two APIs shouldn’t be interchangeable. Especially since you can’t hand a PyObject_Malloc’d object to PyMem_Free() so the inverse shouldn’t be expected to work.
I thought this had officially been deemed illegal for a while, and Google found the reference I was looking for in the What's New for 2.5: """Previously these different families all reduced to the platform's malloc() and free() functions. This meant it didn't matter if you got things wrong and allocated memory with the PyMem function but freed it with the PyObject function. With 2.5's changes to obmalloc, these families now do different things and mismatches will probably result in a segfault. You should carefully test your C extension modules with Python 2.5.""" So either the allocation or the release needs to change here. The behaviour of PyObject_Del when handed a pointer it doesn't recognise is currently undocumented. It may be best to make it officially undefined in order to further discourage people from relying on the implicit delegation to PyMem_Free. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (3)
-
Kristján Valur Jónsson
-
M.-A. Lemburg
-
Nick Coghlan