[Python-3000] Removing __del__

[Giovanni Bajo]

Tim Peters wrote:

> [Giovanni Bajo]
>> Raymond, there is one thing I don't understand in your line of
>> reasoning. You say that you prefer explicit finalization, but that
>> implicit finalization still needs to be supported. And for that,
>> you'd rather drop __del__ and use weakrefs. But why? You say that
>> __del__ is harardous, but I can't see how weakrefs are less
>> hazardous. As an implicit finalization method, they live on the
>> fragile assumption that the callback won't hold a reference to the
>> object: an assumption which cannot be enforced in any way but
>> cautious programming and scrupolous auditing of the code.
>
> Nope, not so.  Read Modules/gc_weakref.txt for the gory details.
> [...]
> The dodge
> Python currently takes is that, when a WR is part of CT, and the WR's
> referent is also part of CT, the WR's CB (if any) is never invoked.
> This is defensible since the order in which trash objects are
> finalized isn't defined, so it's legitimate to kill the WR first.
> It's unclear whether that's entirely desirable behavior, though.
> There were excruciating discussions about this earlier, but nobody had
> a concrete use case favoring a specific position.

Thanks for the explanation, and I believe you are confirming my position.
You are saying that the CB of a WR which is part of CT is never invoked. In
the above quote, I'm saying that if the user does a mistake and writes a CB
(as an implicit finalizer) which holds a reference to the WO, it is creating
a CT, so the CB will never be invoked. For instance:

class Wrapper:
     def __init__(self, *args):
           self.handle = CAPI.init(*args)
           self._wr = weakref.ref(self, lambda: CAPI.close(self.handle))   #
BUG HERE!

In this case, we have a CT: a Wrapper instance is the WO, which holds a
strong reference to the WR (self._wr), which holds a strong reference to the
CB (the lambda), which holds a strong reference to the WO again (through the
implicit usage of nested scopes). Thus, in this case, the CB will never be
called. Is that right? I have tried this variant to verify myself:

>>> import weakref
>>> class Wrapper:
...     def __init__(self):
...             def test():
...                     print "finalizer called", self.a
...             self.a = 1234
...             self._wr = weakref.ref(self, test)
...
>>> w = Wrapper()
>>> del w
>>> import gc
>>> gc.collect()
6
>>> gc.collect()
0
>>> gc.garbage
[]


Given these examples, I still can't see why weakrefs are thought to be a
preferrable solution for implicit finalization, when compared to __del__.
They mostly share the same problems when it comes to cyclic trash, but
__del__ is far more easier to teach, explain and understand. I can teach not
to use cycles with __del__ quickly and I can verify if there's a mistake by
looking at gc.garbage; teaching how to properly use weakrefs, callbacks, and
how to avoid reference loops with nested scopes is much harder to grasp in
the first place, and does not seem to provide any advantage.

===============================

Tim, I sort of hoped you jumped into this discussion. I had this link around
I wanted to show you:
http://mail.python.org/pipermail/python-dev/2000-March/002526.html

I re-read most threads in those weeks about finalization issues with cyclic
trash. Guido was proposing a solution with __del__ and CT, which
approximately worked this way:

- When a CT is detected, any __del__ method is invoked once per instance, in
random order.
- We make sure that each __del__ method is called once and only once per
instance (by using some sort of flag; Guido was proposing to set
self.__dict__["__del__"] = None, but that predates new-style classes as far
as I can tell).
- After all __del__ methods in the CT have been called exactly once, we
collect the trash as usual (break links by reclaiming the __dict__ of the
instances, or whatever).

Since we are discussing Py3k here, I believe it is the right time to revive
this discussion. The __close__ proposal I'm backing (sumed up in this mail:
http://mail.python.org/pipermail/python-3000/2006-September/003892.html) is
pretty similar to how Guido was proposing to modify __del__. If there are
technical grounds for this (and my opinion does not matter much, but Guido
was proposing the same thing, which kinds of gives me hope in this regard),
I believe it would be a far superior solution for the problem of implicit
finalization in the presence of CT in Py3k.

I think the idea is that, if you make sure that a __close__ method is called
exactly once (and before __dict__ is reclaimed), it really does not matter
much in which order you call __close__ methods within the CT. I mean, it
*might* matter for already-written in-the-wild __del__ methods of course,
but it sounds a *very* reasonable constraint for Py3k's __close__ methods. I
would like to see real-world examples where calling __close__ in random
order break things.

In the message linked above, you reply with:

[Tim]
> I would have no objection to "__del__ called only once" if it weren't
> for that Python currently does something different.  I don't know
> whether people rely on that now; if they do, it's a much more
> dangerous thing to change than adding a new keyword.

Would you still have this same position? Do you consider this "only once"
rule as a possible way to solve implicit finalization in GC?
-- 
Giovanni Bajo