[Python-Dev] Object creation hook

Kristján V. Jónsson kristjan at ccpgames.com
Wed Jan 24 12:27:49 CET 2007


Thanks, but the question is really, "how do I build a better debug hook than sys.getobjects?" so I argue this is a valid python-dev question.

We have been using gc.get_objects() but it has several problems:
1) It returns all objects in the system.  This results in a list so long that it often kills the system.  Our system is of a scale that makes this very risky.
2) There is no way to frame certain operations and get just those objects that were created during their execution.  In our case, we would like to get the server cluster running, then frame a certain operation to get a callback for all created objects, so that we could track that they were later destroyed correctly.  I have done this previously by storing the id()s of all objects returned from gc.get_objects() and comparing them "before" and "after" but this suffers from 1) above, and the ambiguity of id() in the face of objects being created and destroyed.


Working with the metaclasses sounds reasonable if one has the luxury of changing the entire codebase to use a different metaclass.  It also doesn't work with old style classes (plenty of those), c extensions, and builtins.

(I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from a python script, I'm fairly sure that is possible in Ruby :)  but I digress...)

My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding this final line:
if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\

The function then looks like this:

void PyCCP_CreationHookFunc(PyObject * obj)
{
        if (PyCCP_CreationHook) {
                PyObject *result, *tmp = PyCCP_CreationHook;
                PyCCP_CreationHook = 0; //guard against recursion
                result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook, obj, 0);
                Py_XDECREF(result);
                if (!result)
                        PyErr_Clear();
                PyCCP_CreationHook = tmp;
        }
}

Doing this, and setting a no-op function as as the PyCCP_CreationHook, does seem to work for a while in the interactive python shell, but it then crashes and I haven't been able to work out the crash.  In any case, doing stuff at the point of GC list insertion is very hairy, especially in the face of __del__ methods and such (of which we don't have many)

I am waiting to get Rational Purify set up on my machine and then I'll maybe be able to understand the crash case better.

Cheers,
Kristján


-----Original Message-----
From: "Martin v. Löwis" [mailto:martin at v.loewis.de]
Sent: 23. janúar 2007 23:32
To: Kristján V. Jónsson
Cc: 'python-dev at python.org'
Subject: Re: [Python-Dev] Object creation hook

Kristján V. Jónsson schrieb:
> I am trying to insert a hook into python enabling a callback for all
> just-created objects.  The intention is to debug and find memory leaks,
> e.g. by having the hook function insert the object into a WeakKeyDictionary.

I'd like to point out that this isn't a python-dev question, but more
appropriate for comp.lang.python (as it is of the "how do I x with
Python?" kind).

I would use a debug build, and use sys.getobjects to determine all
objects and find leaks.

Regards,
Martin


More information about the Python-Dev mailing list