[Python-3000] Removing __del__

[Giovanni Bajo]

Nick Coghlan wrote:

>> Raymond, there is one thing I don't understand in your line of
>> reasoning. You say that you prefer explicit finalization, but that
>> implicit finalization still needs to be supported. And for that,
>> you'd rather drop __del__ and use weakrefs. But why? You say that
>> __del__ is harardous, but I can't see how weakrefs are less
>> hazardous.
>
> As I see it, __del__ is more hazardous because it's an attractive
> nuisance - it *looks* like it should be easy to use, but I'm willing
> to bet that a lot of the __del__ methods implemented in the wild are
> either actual or potential bugs. For example, it would be easy for a
> maintenance programmer to make a change to include a reference in a
> data structure from a child node back to its parent node to address a
> problem, and suddenly the application's memory usage goes through the
> roof due to uncollectable cycles.

Is that easier or harder to detect such a cycle, compared to accidentally
adding a reference to self (through implicit nested scopes, or bound
methods) in the finalizer callback? You have to admit that, at best, they
are equally hazardous.

As things stand *now* (in Python 2.5 I mean),  __del__ is easier to
understand/teach, easier to debug (gc.garbage vs finalizers silently
ignored), and easier to use (no boilerplate in user's code, no additional
finalization API which does not even exist). I saw numerous proposal to
address these weakref "defects" by adding some kind of finalizer API, by
modifying the GC to put uncollectable loops with weakref finalizers in
gc.garbage, and so on. Most finalization APIs (including yours) create
cycles just by using them, which also mean that you *must* wait for the GC
to kick in before the object is finalized, making it useless for several
situations where you want implicit finalizations to happen immediately
(file.close() just to name one). [and we are speaking of implicit
finalization now, I know of 'with'].

It would require some effort to make weakref finalizers *barely* as usable
as __del__, and will absolutely not solve the problem per-se: the user will
still have to pay attention and understand the hoops (different kind of
hoops, but still hoops). So, why do we not spend this same time trying to
*fix* __del__ instead? If somebody comes up with a sane way to define the
semantic for a new finalizer method (like the __close__ proposal), which can
be invoked *even* in the case of cycles, would you still prefer to go the
weakref way?


> Even the initial implementation of
> the generator __del__ slot in the *Python 2.5 core* was buggy,
> leading to such cycles - if the developers of the Python interpreter
> find it hard to get __del__ right, then there's something seriously
> wrong with it in its current form.

I don't think it's a fair comparison: generator is a pretty complex class,
compared to an average class developed in Python which might need a __del__
method. I would also bet that you would get your first attempt of
finalization of generators through weakrefs wrong.


> By explicitly stating that __del__ will go away in Py3k, with the
> current intent being to replace it with explicit finalization (via
> with statements) and the implicit finalization offered by weakref
> callbacks, it encourages people to look for ways to make the API for
> the latter easier to use.
>
> For example, a "finalizer" factory function could be added to weakref:
>
> _finalizer_refs = set()
> def finalizer(*args, **kwds):
>      """Create a finalizer from an object, callback and keyword
>      dictionary""" # Use positional args and a closure to avoid
>      namespace collisions obj, callback = args
>      def _finalizer(_ref=None):
>          """Callable that invokes the finalization callback"""
>          # Use closure to get at weakref to allow direct invocation
>          # This creates a cycle, so this approach relies on cyclic GC
>          # to clean up the finalizer objects!
>          try:
>              _finalizer_refs.remove(ref)
>          except KeyError:
>              pass
>          else:
>              callback(_finalizer)
>      # Give callback access to keyword arguments
>      _finalizer.__dict__ = kwds
>      ref = weakref.ref(obj, _finalizer)
>      _finalizer_refs.add(ref)
>      return _finalizer

So uhm, am I reading it bad or your implementation (like any other similar
API I have seen till now) create a cycle *just* by using it? This finalizer
API ofhuscates user code by forcing to use a separate _data object to hold
(part of) the context for apparently no good reason, and make the object
collectable *only* through the cyclic GC (while __del__ would happily be
invoked in simple cases when the object goes out of context).

> P.S. the central finalizers list also works a treat for debugging why
> objects aren't getting finalized as expected - a simple loop like
> "for wr in weakref.finalizers: print gc.get_referrers(wr)" after a
> gc.collect() call works pretty well.

Yes, this is indeed interesting. One step closer to get at the __del__
feature set :)
-- 
Giovanni Bajo