[Python-Dev] RE: Program very slow to finish
Tim Peters
tim.one@home.com
Tue, 6 Nov 2001 17:30:57 -0500
[Neil Schemenauer]
> The problem is with extension modules. We can make sure code in CVS
> always has the big lock held when calling PyMalloc. We can't be sure
> that extension modules are safe.
So can that *ever* be solved? Some part of the puzzle went unaddressed
here.
> The other, more serious problem is that many extension modules allocate
> memory with PyObject_New() and free it with free() or PyMem_DEL()
> instead of PyObject_Del() or PyObject_DEL().
Umm, OK, PyObject_DEL() ... is a macro ... which redirects to
PyObject_FREE() ... which is a macro ... which redirects to
PyCore_OBJECT_FREE() ... which is a macro ... which redirects to
PyCore_OBJECT_FREE_FUNC() ... which is a macro ... which redirects to
_PyCore_ObjectFree (WITH_PYMALLOC) or PyCore_FREE_FUNC (without).
PyCore_FREE_FUNC is a macro ... which redirects to free(). And
_PyCore_ObjectFree ... doesn't exist. I must have missed an #undef in there
somewhere ... ah, OK, in the WITH_PYMALLOC case, PyCore_OBJECT_FREE_FUNC(==
_PyCore_ObjectFree) appears in obmalloc.c's
#define _THIS_FREE PyCore_OBJECT_FREE_FUNC
so that _THIS_FREE is actually _PyCore_ObjectFree; then obmalloc.c's
void
_THIS_FREE(void *p)
{
expands to
void
PyCore_OBJECT_FREE_FUNC(void *p)
{
which expands again to
void
_PyCore_ObjectFree(void *p)
{
and we're done. Couldn't be simpler <wink>. Then PyMem_DEL ... oh, forget
it. Now I remember why I gave up on this last time I looked at it -- it's
Preprocessor Hell.
> If pymalloc is enabled then memory allocated by a PyObject_* function
> must be freed by a PyObject_* funciton.
Damn, you're good.
> mxDateTime is the first module I ran into that does this but there are
> many others I'm sure. I think almost all of the C modules I have
> written do it. The code in xxmodule.c used to do it as well.
>
> no-solutions-only-more-problems-hey-i'm-sick-today-too-ly y'rs Neil
Big Hammer? Change every one of the existing guys to resolve to malloc()
and free(). Then declare them all obsolete, define a much smaller new set
of names, edit the core to use the new guys, and non-core modules get stuck
with malloc/free until they're rewritten too. It sucks, but we'd be in
better shape today if that *had* been done the last time this got reworked.