[Python-Dev] Cyclic GC issues

[Nick Bastin]

On Oct 11, 2004, at 2:59 AM, Jason Evans wrote:

>>>   Working around this in Crux requires a lot of extra reference 
>>> counting
>>>   complexity, because there are three different cases for reference
>>>   counts, depending on how many members there are in a ring (1, 2, 
>>> or 3+
>>>   members).
>>
>> Why is that? Just DECREF every PyObject* in "self", then set the
>> PyObject* to NULL. No need for any additional reference counts.
>
> The low level tree code is implemented completely separately from the
> Python object wrapper code.  This means that, for example, the Python
> "Node" object does not actually store PyObject* pointers to Ring 
> objects;
> instead it uses the low level tree code to find out what Ring objects 
> are
> attached to it.  Crux was designed this way in order to be able to
> implement various low level algorithms such that they never have to 
> call
> interpreter-related code.  As such, reference breaking code must 
> actually
> tear down the low level tree in such a way that it is always 
> "consistent"
> (simply setting pointers to NULL doesn't fit well with this way of 
> doing
> things).

You've now reached the point where the Python-GC-ease-of-implementation 
breaks down.  We have encountered the same problem - the GC is fine as 
long as you're solely extending Python, but if you're embedding it, 
chances are you'll encounter some interesting issues like this along 
the way.  It's not that you can't make it work - you can, but need to 
do a lot more work yourself.  Basically, if I understand you correctly 
(and I may not be), there are times when you do not want tp_dealloc to 
actually destroy the low-level data structure, because somebody else 
(not living in the Python reference-counted world) isn't done with it 
yet.  The problem is in knowing when those times are, and when they 
aren't.  In this case, you may need to implement basic reference 
counting in your own low-level data structures, so you know when you're 
really done with a ring and ready to destroy it.  The other option is 
to attempt to make sure that you're not creating cycles (in python) and 
avoid interfacing with GC entirely, but that may not be  possible in 
your case.

--
Nick