[Python-Dev] Object creation hook

Neal Norwitz nnorwitz at gmail.com
Sat Jan 27 04:24:37 CET 2007


Short of using a memory debugger such as Valgrind or Purify, have you
considered looking for reference leaks?  These may be the cause and
can be checked with pure python code.  See how Lib/test/regrtest.py
handles the -R option.

n
--

On 1/24/07, Kristján V. Jónsson <kristjan at ccpgames.com> wrote:
> Thanks, but the question is really, "how do I build a better debug hook than sys.getobjects?" so I argue this is a valid python-dev question.
>
> We have been using gc.get_objects() but it has several problems:
> 1) It returns all objects in the system.  This results in a list so long that it often kills the system.  Our system is of a scale that makes this very risky.
> 2) There is no way to frame certain operations and get just those objects that were created during their execution.  In our case, we would like to get the server cluster running, then frame a certain operation to get a callback for all created objects, so that we could track that they were later destroyed correctly.  I have done this previously by storing the id()s of all objects returned from gc.get_objects() and comparing them "before" and "after" but this suffers from 1) above, and the ambiguity of id() in the face of objects being created and destroyed.
>
>
> Working with the metaclasses sounds reasonable if one has the luxury of changing the entire codebase to use a different metaclass.  It also doesn't work with old style classes (plenty of those), c extensions, and builtins.
>
> (I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from a python script, I'm fairly sure that is possible in Ruby :)  but I digress...)
>
> My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding this final line:
> if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\
>
> The function then looks like this:
>
> void PyCCP_CreationHookFunc(PyObject * obj)
> {
>         if (PyCCP_CreationHook) {
>                 PyObject *result, *tmp = PyCCP_CreationHook;
>                 PyCCP_CreationHook = 0; //guard against recursion
>                 result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook, obj, 0);
>                 Py_XDECREF(result);
>                 if (!result)
>                         PyErr_Clear();
>                 PyCCP_CreationHook = tmp;
>         }
> }
>
> Doing this, and setting a no-op function as as the PyCCP_CreationHook, does seem to work for a while in the interactive python shell, but it then crashes and I haven't been able to work out the crash.  In any case, doing stuff at the point of GC list insertion is very hairy, especially in the face of __del__ methods and such (of which we don't have many)
>
> I am waiting to get Rational Purify set up on my machine and then I'll maybe be able to understand the crash case better.
>
> Cheers,
> Kristján
>
>
> -----Original Message-----
> From: "Martin v. Löwis" [mailto:martin at v.loewis.de]
> Sent: 23. janúar 2007 23:32
> To: Kristján V. Jónsson
> Cc: 'python-dev at python.org'
> Subject: Re: [Python-Dev] Object creation hook
>
> Kristján V. Jónsson schrieb:
> > I am trying to insert a hook into python enabling a callback for all
> > just-created objects.  The intention is to debug and find memory leaks,
> > e.g. by having the hook function insert the object into a WeakKeyDictionary.
>
> I'd like to point out that this isn't a python-dev question, but more
> appropriate for comp.lang.python (as it is of the "how do I x with
> Python?" kind).
>
> I would use a debug build, and use sys.getobjects to determine all
> objects and find leaks.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>


More information about the Python-Dev mailing list