Jim Jewett wrote:

The checkins list has been struggling with generator reference leaks; the latest conclusion was that some are unavoidable because of __del__ cycles. That sort of defeats the purpose of resource managers.

Seems to me we need a whole new approach to finalization that's friendly to cyclic gc, such as a way of registering a finalizer that doesn't depend on the original object. If such a mechanism were available, could it be used instead of a __del__ method to clean up after a generator? (I'm asking because I'm not sure exactly what a generator __del__ needs to do.)

As a strawman proposal:

deletes = [(obj.__del__.cycle, obj) for obj in cycle if hasattr(obj, "__del__") and hasattr(obj.__del__, "cycle")] deletes.sort() for (cycle, obj) in deletes: obj.__del__()

I think we need to be very careful about doing anything like this. From what Tim said recently, the consequences of an object getting its __del__ annotation wrong could be as bad as crashing the interpreter. -- Greg

Duncan Booth wrote:

Surely if you have a cycle what you want to do is to pick just *one* of the objects in the cycle and break the link which makes it participate in the cycle. That should be sufficient to cause the rest of the cycle to collapse with __del__ methods being called from the normal refcounting mechanism.

So something like this:

for obj in cycle: if hasattr(obj, "__breakcycle__"): obj.__breakcycle__() break

Every object which knows it can participate in a cycle then has the option of defining a method which it can use to tear down the cycle. e.g. by releasing the resource and then deleting all of its attributes, but no guarantees are made over which obj has this method called. An object with a __breakcycle__ method would have to be extra careful as its methods could still be called after it has broken the cycle, but it does mean that the responsibilities are in the right place (i.e. defining the method implies taking that into account).

Unfortunately, there's two problems with that idea: a. it's broken, since we now have a partially torn down object at the tail end of our former cycle. What happens if the penultimate object's finaliser tries to access that broken one? b.it doesn't actually help in the case of generators (which are the ones causing all the grief). The generator object itself (which implements the __del__ method) knows nothing about what caused the cycle (the cycle is going to be due to the Python code in the body of the generator). As PJE posted the other day, the problem is that the GC assumes that because the *type* has a __del__ method, the *instance* needs finalisation. And for objects with an explicit close method (like generators), context management semantics (like generator-based context managers), or the ability to be finalised in the normal course of events (like generator-iterators), most instances *don't* need finalisation, as they'll have already been finalised in the normal course of events. Generators are even more special, in that they only require finalisation in the first place if they're stopped on a yield statement inside a try-finally block. A simple Boolean attribute (e.g. __finalized__) should be enough. If the type has a __del__ method, then the GC would check the __finalized__ attribute. If it's both present and true, the GC can ignore the finaliser on that instance (i.e. never invokes it, and doesn't treat cycles as uncollectable because of it) I don't know the GC well enough to know how hard that would be to implement, but I suspect we need to do it (or something like it) if PEP 342 isn't going to cause annoying memory leaks in real applications. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

[Phillip J. Eby]

... As Tim suggested, it'd be better to have the code be generator-specific, at least for now. That had actually been my original plan, to make it generator-specific, but I was afraid of breaking encapsulation in the garbage collector by having it know about generators.

It sucks in a way, but so would adding yet another new slot just for (at present, and possibly forever) making gc and generators play nicer together. "Practicality beats purity" here.

But now that Uncle Timmy has blessed the approach, I'll go back and add it in. (On Monday, unless somebody gets to it before me.)

It won't be me: I wasn't even able to make enough time to understand the new generator features at the Python level, let alone the implementation. At PyCon, when Guido showed his slide with a new yield-as-expression example, for the rest of his talk I was wondering what the heck the example meant <0.3 wink>.

Greg Ewing wrote:

Nick Coghlan wrote:

...

Generators are even more special, in that they only require finalisation in the first place if they're stopped on a yield statement inside a try-finally block.

I find it rather worrying that there could be a few rare cases in which my generators cause memory leaks, through no fault of my own and without my being able to do anything about it.

The GC changes PJE is looking at are to make sure you *can* do something about it. If the generator hasn't been started, or has already finished, then the GC won't consider it as needing finalisation.

Will there be a coding practice one can follow to ensure that this doesn't happen?

I believe PJE's fix should take care of most cases (depending on how aggressive we can safely be, it may even take care of all of them). If there are any remaining cases, I think the main thing is to avoid keeping half-finished generators around: from contextlib import closing with closing(itr): # Use the iterator in here as you wish # secure in the knowledge it will be # cleaned up promptly when you are done # whether it is a file, a generator or # something with a database connection for item in itr: print item Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 4/1/06, Greg Ewing wrote:

I'm becoming more and more convinced that we desperately need something better than __del__ methods to do finalisation. A garbage collector that can't be relied upon to collect garbage is simply not acceptable.

Sure. I don't believe it's too hard, it just means violating some of the premises people have been writing __del__ methods under. For instance, to clean up cycles nicely we might have to set some attributes to None before calling __del__, so you can't rely on attributes being meaningful anymore. However, this is already the case for global names; I've seen many people wonder about their __del__ method raising warnings (since exceptions are ignored) going, say, 'NoneType has no attribute 'registry'' when they try to un-register their class but the global registry has been cleaned up already. While we're at it, I would like for the new __del__ (which would probably have to be a new method) to disallow reviving self, just because it makes it unnecessarily complicated and it's rarely needed. Allowing a partially deleted object (or an object part of a partially deleted reference-cycle) to revive itself is not terribly useful, and there's no way to restore the rest of the cycle. I suggested a __dealloc__ method earlier in the thread to do this. I didn't think of allowing attributes to be cleared before calling the method, but I do believe that is necessary to allow, now that I've thought more about it. An alternative would be to make GC check for a 'cleanup-cycle' method on any of the objects in the cycle, and just feed it the complete cycle of objects, asking it to clean it up itself (or maybe reconnect one of the objects itself.) That would also make debugging uncollectable cycles a lot easier ;-) But I'm not sure whether that will improve things. The generator example, the trigger for this discussion, could solve its cycle by just closing itself, after which the cycle is either broken or reconnected, but I don't know if other typical cycles could be resolved that easily. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On 4/3/06, Michael Hudson wrote:

I'm not sure the problem is so much that anyone _wants_ to support resurrection in __del__, it's just that it can't be prevented.

Well, Java has an answer to that (at least I believe Tim Peters told me so years ago): it allows resurrection, but will only call the finalizer once. IOW if the resurrected object is GC'ed a second time, its finalizer won't be called. This would require a bit "__del__ already called" on an object, but don't we have a whole word of GC-related flags? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Apr 3, 2006, at 3:12 PM, Neil Schemenauer wrote:

Guido van Rossum wrote:

...

This would require a bit "__del__ already called" on an object, but don't we have a whole word of GC-related flags?

No.

Actually there is. Kinda. Currently python's refcounting scheme uses 4 words per object (gc_next, gc_prev, gc_refs, ob_refcnt), and has one spare word in the padding of PyGC_Head that's just sitting there wasting memory. So really it's using up 5 words per object, and that 5th word could actually be used for flags... /* GC information is stored BEFORE the object structure. */ typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; int gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head; #define PyObject_HEAD \ _PyObject_HEAD_EXTRA \ int ob_refcnt; \ struct _typeobject *ob_type; typedef struct _object { PyObject_HEAD } PyObject; James

On Apr 3, 2006, at 4:02 PM, Tim Peters wrote:

Using which compiler? This varies across boxes. Most obviously, on a 64-bit box all these members are 8 bytes (note that ob_refcnt is Py_ssize_t in 2.5, not int anymore), but even on some 32-bit boxes the "long double" trick only forces 4-byte alignment.

Hm, yes, my mistake. I see that on linux/x86, long double only takes 12 bytes and is 4-byte aligned. Even though the actual CPU really wants it 8-byte aligned, the ABI has not been changed to allow that. On OSX/ppc32, OSX/ppc64 and linux/x86-64, doubles are 16 bytes, and 8- byte aligned. So the struct uses 16 bytes or 32 bytes, and has the extra word of unused space. All right, then, you could use the top bit of the ob_refcnt field. There's no way to possibly have 2**32 objects on a 32bit system anyhow. James

Martin v. Löwis wrote:

Of this kind, we have several spare bits: there are atleast two bits in the ob_type field,

And that would require changing all code that dereferenced the ob_type field! This would be much worse -- at least Py_INCREF and Py_DECREF are macros... -- Greg

[various people debating how to steal a bit from an existing PyObject member] Let's stop this. It's a "bike shed" argument: even if we had 1000 spare bits, it wouldn't do any good. Having a bit to record whether an object has been finalized doesn't solve any problem that's been raised recently (it doesn't cure any problems with resurrection, for example). Even if we wanted it just to support a slow, dubious way to collect trash cycles containing objects with __del__ methods, gcmodule.c would need major rewriting to make that happen. If someone is interested enough to do that, fine, but it sure ain't me ;-)

Michael Hudson wrote:

And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks.

Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well. Has that been resolved? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

On 4/3/06, Michael Hudson wrote:

Greg Ewing writes:

...

Michael Hudson wrote:

...

And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks.

Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well. Has that been resolved?

Talk about FUD. Yes, it works, as far as I know.

Not sure if everyone is talking about the same thing. This is still a problem (at least for me): http://svn.python.org/projects/python/trunk/Lib/test/crashers/weakref_in_del... It creates a weakref to self in __del__. There are 7 crashers, plus 5 more due to infinite recursion. :-( That doesn't include the parts of test_trace that are commented out. At least test_trace needs to be fixed prior to 2.5. n

[Michael Hudson]

...

...

And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks.

[Greg Ewing]

...

Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well. Has that been resolved?

[Michael]

Talk about FUD. Yes, it works, as far as I know.

I'm sure Greg has in mind this thread (which was in fact also the thread that floated the idea of getting rid of __del__ in P3K): http://mail.python.org/pipermail/python-dev/2004-November/049744.html As that said, some weakref gc semantics are pretty arbitrary now, and it gave two patches that implemented distinct semantic variants. A problem is that the variant semantics also seem pretty arbitrary ;-), and there's a dearth of compelling use cases to guide a decision. If someone devoted enough time to seriously trying to get rid of __del__, I suspect compelling use cases would arise. I never use __del__ anyway, so my motivation to spend time on it is hard to detect.

[Tim Peters]

...

A problem is that the variant semantics also seem pretty arbitrary ;-), and there's a dearth of compelling use cases to guide a decision.

[Greg Ewing] |> At the time I think I suggested that it would be

reasonable if weakref callbacks were guaranteed to be called as long as they weren't trash themselves and they didn't reference any trash. From what you said in a recent message, it sounds like that's the way it currently works.

Nope, and that was the point of the first message in the thread I referenced. The issues were explained well in that thread, so I don't want to repeat it all here again.

These semantics would be sufficient to be able to use weakrefs to register finalisers, I think. You keep a global list of weakrefs with callbacks that call the finalizer and then remove the weakref from the global list.

I'll put together a prototype some time and see if it works. I actually have a use case at the moment -- a couple of types that need to release external resources, and since they're part of a library I'm distributing, I can't be sure people won't put them in cycles.

Note that it's very easy to do this with __del__. The trick is for your type not to have a __del__ method itself, but to point to a simple "cleanup object" with a __del__ method. Give that "contained" object references to the resources you want to close, and you're done. Because your "real" objects don't have __del__ methods, cycles involving them can't inhibit gc. The cleanup object's only purpose in life is to close resources. Like: class _RealTypeResourceCleaner: def __init__(self, *resources): self.resources = resources def __del__(self): if self.resources is not None: for r in self.resources: r.close() self.resources = None # and typically no other methods are needed, or desirable, in # this helper class class RealType: def __init__(*args): ... # and then, e.g., self.cleaner = _ResourceCleaner(resource1, resource2) ... tons of stuff, but no __del__ .... That's the simple, general way to mix cycles and __del__ without problems.

[Michael Hudson]

... What happened to the 'get rid of __del__ in py3k' idea?

Apart from its initial mention, every now & again someone asks what happened to it :-).

On 4/1/06, Nick Coghlan wrote:

Greg Ewing wrote:

...

I find it rather worrying that there could be a few rare cases in which my generators cause memory leaks, through no fault of my own and without my being able to do anything about it.

The GC changes PJE is looking at are to make sure you *can* do something about it. If the generator hasn't been started, or has already finished, then the GC won't consider it as needing finalisation.

Actually, if a generator has already finished, it no longer holds a suspended frame alive, and there is no cycle (at least not through the generator.) That's why test_generators no longer leaks; explicitly closing the generator breaks the cycle. So the only thing fixing GC would add is cleaning up cycles where a created but not started generator is the only thing keeping the cycle alive. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!