[Python-Dev] Cyclic GC issues
jasone at canonware.com
Tue Oct 12 04:25:47 CEST 2004
On Mon, Oct 11, 2004 at 09:21:57PM -0400, Tim Peters wrote:
> [Jason Evans]
> I think you're in for a rude awakening if you expect any language's gc
> support to help you manage arbitrary blobs of memory obtained from the
> system malloc().
I don't expect that. What I do expect is for there to be clean and
efficient mechanisms for integrating custom Python object types into
Python. Python more or less meets this expectation.
> If all the nodes in your object graph derived from PyObject, Python's gc
> would do it all for you. If you want to plant pointers in and among
> arbitrary C structs, then you'll have to implement your own gc, or hope
> that you can bash a general gc thingie (like BDW -- and I'm not sure
> there is anything else of that sort usefully available, so "like" is
> probably optimistic) into guessing the layout of your structs.
No, this is not necessary. What is necessary is for my code to accurately
maintain and report all references to Python objects during traversal. The
Crux code does so, and works as designed.
> > Crux was designed this way in order to be able to implement various low level
> > algorithms such that they never have to call interpreter-related code.
> I don't know why that was thought to be desirable, or even exactly
> what it means. One plausible meaning is that you don't want to tie
> the "various low level algorithms" to Python, because you hope to
> reuse them in other, non-Python contexts. And, in addition to that,
> you don't want, or don't know how, to abstract the necessary
> operations into macros or functions that are pluggable for the
> different contexts you have in mind.
First, "hoping" to be able use my code in other contexts is what allowed me
to transition from using a different embeddable interpreter to using Python
inside of two days. Had I melded all of the scripting glue code with the
low level tree code, it would not have been nearly as easy to transition to
Second, I have the scripting language glue and low level tree code
separated for two reasons: 1) performance, and 2) modularization. There's
no question of whether I want to use cpp magic to be able to munge two
large piles of code into one huge pile of code, nor whether I'm capable of
doing so. To do so would simply take the code in a bad direction.
> >> Hmmm. Even if a ring member holds only a single reference to another
> >> ring member, would not calling visit() on this member in turn invoke
> >> visit() on the other one, which would, in turn, invoke visit() on
> >> the next one, and so on?
> > Yes, this is true.
> It's false, but I covered that one yesterday.
It is true, given my interpretation of the original context of the
dialogue. We were discussing whether it would work for a node object to
hold a reference to a single ring object, and rely on the ring objects to
report their neighbors. Indeed this would work.
> > The problem I ran into had to do with the node logically holding references to all
> > the ring objects, but only reporting one of them.
> If a ring object doesn't derive from PyObject, then visit() must not
> be passed a pointer to a ring object. Doing so would account for many
> catastrophic meanings of "doesn't work" <wink>.
If you want to see the Crux source code, I'll be happy to provide you with
a source archive. In the meanwhile, please rest assured that the code
doesn't suck nearly as badly as you are insinuating here.
> We can help you use Python's gc, but not if you're unwilling or unable
> to change anything about what you do.
Prior to switching to using Python, Crux integrated with a language that
implemented atomic mark and sweep garbage collection. I have had to change
a tremendous amount about Crux in order to use a reference counted
interpreter. In fact, I made some changes to Crux this morning that were a
direct result of your original response to my email (ring objects now
report references to themselves, which makes reference counting of ring
objects much simpler).
Many of the issues I ran into had to do with the impedence mismatch between
my previous experience and the way Python works. I appreciate you taking
the time to respond to me, especially since you've helped me gain a deeper
understanding of how Python's GC works. However, please get it out of your
mind that I am incompetent and/or set in my ways.
> > I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(),
> > the clear() call does not pay attention to the return value.
> Heh. So true! See
> > It would be possible to check the return value, and defer deallocation if the
> > return value were non-zero, and ob_refcnt were 1. (However, it would probably
> > have to be a separate list of objects, contrary to what I claimed before.)
> I'm not sure, but I think your desire for this was still based on
> misunderstanding of what Python's gc does.
Of the original issues I brought up, I think this one potentially still has
merit, though I'm not sure that it's of sufficiently general utility. The
idea is to allow extension modules some control over the order in which
objects are deallocated, so that it is possible to tear down
interconnections in a particular order.
However, it occurs to me that since tp_clear can leave references in place,
it might be possible to control the object destruction order by leaving a
DAG of references in place, and letting Python deallocate objects in
dependency order. Does this work?
More information about the Python-Dev