[Python-Dev] PEP 442 clarification for type hierarchies

[Stefan Behnel]

Hi,

I was just continuing in my monologue when you replied, so I'll just drop
my response below.

Antoine Pitrou, 05.08.2013 20:51:
> On Sun, 04 Aug 2013 09:23:41 +0200
> Stefan Behnel wrote:
>> I'm currently catching up on PEP 442, which managed to fly completely below
>> my radar so far. It's a really helpful change that could end up fixing a
>> major usability problem that Cython was suffering from: user provided
>> deallocation code now has a safe execution environment (well, at least in
>> Py3.4+). That makes Cython a prime candidate for testing this, and I've
>> just started to migrate the implementation.
> 
> That's great to hear. "Safe execution environment" for finalization
> code is exactly what the PEP is about.
> 
>> One thing that I found to be missing from the PEP is inheritance handling.
>> The current implementation doesn't seem to care about base types at all, so
>> it appears to be the responsibility of the type to call its super type
>> finalisation function. Is that really intended?
> 
> Yes, it is intended that users have to call super().__del__() in their
> __del__ implementation, if they want to call the upper-level finalizer.
> This is exactly the same as in __init__() and (most?) other special
> functions.

That's the Python side of things. However, if a subtype overwrites
tp_finalize(), then there should be a protocol for making sure the super
type's tp_finalize() is called, and that it's being called in the right
kind of execution environment.


>> Another bit is the exception handling. According to the documentation,
>> tp_finalize() is supposed to first save the current exception state, then
>> do the cleanup, then call WriteUnraisable() if necessary, then restore the
>> exception state.
>>
>> http://docs.python.org/3.4/c-api/typeobj.html#PyTypeObject.tp_finalize
>>
>> Is there a reason why this is left to the user implementation, rather than
>> doing it generically right in PyObject_CallFinalizer() ? That would also
>> make it more efficient to call through the super type hierarchy, I guess. I
>> don't see a need to repeat this exception state swapping at each level.
> 
> I didn't give much thought to this detail. Originally I was simply
> copying this bit of semantics from tp_dealloc and tp_del, but indeed we
> could do better. Do you want to open an issue about it?


I think the main problem I have with the PEP is this part:

"""
The PEP doesn't change the semantics of:
 * C extension types with a custom tp_dealloc function.
"""

Meaning, it was designed to explicitly ignore this use case. That's a
mistake, IMHO. If we are to add a new finalisation protocol, why not make
it work in the general case (i.e. fix the problem once and for all),
instead of restricting it to a special case and leaving the rest to each
user to figure out again?

Separating the finalisation from the deallocation is IMO a good idea. It
fixes cyclic garbage collection, that's excellent. And it removes the
differences between GC and normal refcounting cleanup by clarifying in what
states finalisation and deallocation are executed (one safe, one not).

I think what's missing is the following.

* split deallocation into a distinct finalisation and deallocation phase

* make finalisation run recursively (or iteratively) through all
inheritance levels, in a well defined execution environment (e.g. after
saving away the exception state)

* after successful finalisation, run the deallocation, as before.

My guess is that a recursive finalisation phase where subtype code calls
into the supertype is generally more efficient, so I think I'd prefer that.

I think it's a mistake that the current implementation calls the
finalisation from tp_dealloc(). Instead, both the finalisation and the
deallocation should be called externally and independently from the cleanup
mechanism behind Py_DECREF(). (There is no problem in CS that can't be
solved by adding another level of indirection...)

An obvious open question is how to deal with exceptions during
finalisation. Any break in the execution chain would mean that a part of
the type wouldn't be finalised. One way to handle this could be to simply
assume that the deallocation phase would still clean up anything that's
left over. Or the protocol could dictate that each level must swallow its
own exceptions and call the super type finaliser with a clean exception
state. This might suggest that an external iterative call loop through the
finaliser hierarchy has a usability advantage over recursive calls.

Just dropping my idea here.

Stefan