Inspired by discussions with Vic and through browsing the vlibxml2 code,
I've implemented bit of memory management functionality which, after a
lot of manual debugging, seems to be doing the right thing. So far...
A new addition to the lxml trunk is nodereg, and associated testing
stuff (noderegtest.pyx and test_nodereg.py). nodereg is a system for
registering Python-level node proxies, plus some base classes for the
document and node objects in a typical libxml2 tree wrapper.
The nodereg module functionality can be used to make sure that memory
(in particular libxml2 tree nodes) gets collected when it is possible,
and not before. :) This sounds easy, but it is surprisingly tricky.
* look into hooking in libxml2's memory debugging functionality for
testing. Investigate Vic's code in that area/get Vic's advice.
* start rewriting etree, dom, or vlibxml2 to use nodereg. This will
likely further evolve nodereg.
* Add more functionality to nodereg. One thing that currently is not
handled is attribute nodes, for instance.
* Optimize nodereg. The strategy currently employed requires, in the
worst case, a lot of full-tree walks to determine whether a node in the
tree can be successfully garbage collected. We need to come up with some
smart algorithm/datastructure to avoid this having to happen to often.
Another thing I would really like to do is investigate adding weakref
support to Pyrex. Right now I had to first jump through a bit of a hoop
to make it work. Then later on I took a long time debugging an obscure
case where there would be a remaining refcount on an object even if the
only object still pointing to the object was a WeakValueDictionary. I
finally traced it down to Pyrex introducing this. I'm not clear why, but
somehow the base class got involved (which was not weakreferenceable as
defined by Pyrex). This somehow managed to trick the object into keeping
a reference while it shouldn't, causing it never to be deallocated.
Being able to just say 'this class can be weakreferenced' in Pyrex
should make this go away.