[Python-Dev] Object finalization for local (ie function) scopes

[Oliver Schoenborn]

This is a module I made to test a concept of finalization *not* based on
__del__. I'm curious to find out if the techniques look like they could be
implemented directly in the Python interpreter to lessen the burden on the
user, which is why I post this here.

Summary:
- Seems like can't rely on __del__ being called (standard says no guarantee)
- Means a coder of a class for which instances need some sort of
finalization (e.g. file objects need close()) must document textually the
need for finalization
- User of class must remember to use try/finally blocks and position them
judiciously since several objects may require "finalization"; this is easy
to forget if there are multiple (normal or exceptional) exit points.

The technique is based on inheritance: a class that needs finalization is
derived from scope.NeedsFinalization. This class adds info to a special
function, called scope.ScopeGuardian, that wraps a try/finally around the
function that you are interested in making "finalization safe" (ie the
function in which you would normally have to worry about manually putting a
try/finally). This seems to work well and has the advantage that:

- Coder of class makes explicit by base class that finalizaiton is
important, no need to remember to document
- User of class knows just by looking at class def, can't overlook  a
sentence saying "call this-that when done with object"
- Some error checking can be done to decrease likelyhood of user forgetting
the try/finallies (which are no longer needed) or of wanting finalization
before ref count goes to 0.

The info added to the function, seems to me, could be done at the
interpreter implementation level, e.g. by adding a new field to function
objects (f_name, etc), such as f_scopedobjs, so it would be almost trivial
to use even for beginners. There are a couple of limitations in the current
implementation that are easy to fix (e.g. make it work for recursive
functions, nested scope guarding, and methods), don't know yet about thread
safety.

Anyways, from having looked at the archives, doesn't seem like this
technique has been used, and would't require addition of a new keyword (only
a new class, .e.g as opposed to pep310 approach).

Thanks for any feedback,

Oliver

-------------- next part --------------
"""
This module provides a class, NeedsFinalization, that you derive from
to indicate that instances of it need finalization upon local
(function) scope. This module also provides a ScopeGuarded() function,
that takes as argument a function and returns a scope-guarded
function.

To use this module in a function that instantiates objects that must
be finalized on exit of function, derive the class of those objects
from NeedsFinalization. Don't forget to call
NeedsFinalization.__init__ in the constructors and to define
_finalize(self) for each class. Then, in the module where your
function is defined, do yourFn = ScopeGuarded(yourFn). This rebinds
your function to be scope-guarded. Now when you call yourFn(), a
special function actually gets called that makes sure all instances of
NeedsFinalization in your original yourFn get deleted. Note that some
runtime error checking is done for you: if you forget to call
NeedsFinalization.__init__(self), or you forget to wrap func but
func contains instances of NeedsFinalization, or if you 

Ex::

    class MyClass(NeedsFinalization):
        def __init__(self):
            NeedsFinalization.__init__(self)

        def _finalize(self):
            # do finalization stuff

    def yourFunc(name, greeting):
        obj1 = MyClass() # needs finalization
        obj2 = 'asdf'    # doesn't need it
        # do stuff with name, greeting, obj1 and obj2
        return (0,3)

    yourFunc = ScopeGuarded(yourFunc)
    yourFunc(1,'hello')


This module demonstrates that scope exit for local scopes can have
the equivalent of automatic, guaranteed finalizers, using techniques
that could most likely be implemented inside Python interpreter.
This would bypass the need for re-assigned function names and avoid
the two extra levels of indirection needed in the current
implementation. This would require a built-in class
NeedsFinalization but ScopeGuarded would become hidden from user:
instead, every NeedsFinalization can add itself to a special
built-in field of function objects, and the function would go over
that list upon return and do the finally clause.
  
Notes:

- Current implementation doesn't work for recursive functions, or
  multiple levels of scope guarding, or multiple threads: this could
  probably work if a stack of lists is used for func.f_scopedObjs
  instead of a plain list. However, this module is merely a
  demonstration of the concept for study.

- Probably doesn't work for diamond inheritence diagrams: the
  _finalize() method of root of diamond will get called twice. How
  should this situation be trapped? Perhaps instead of asking user to
  call base class _finalize(), the user would call
  self.finalize_bases(), and that function, implemented in
  NeedsFinalization, would make sure things only get called once (if
  possible).

- Could it be made to work for unbound methods? Docs of inspect module
  seem to say no.
  
- There may be other issues to be resolved, other than the two
  above (recursion etc).
  
:Author:     Oliver Schoenborn
:Since:      Jun 9, 2004
:Version:    $Id$
:Copyright:  \(c) 2004 Oliver
:License:    Same as for Python

"""

import inspect, sys

def ScopeGuarded(func):
    """Return func wrapped such that if you call func(...) you
    get the _finalize() method of all instances that derive
    from NeedsFinalization, instantiate in func, to be called
    upon return, normal or via exception. """
    return lambda *args, **kwargs: ScopeGuardian(func, *args, **kwargs)
    
class NeedsFinalization:
    def __init__(self):
        """Find ScopeGuardian function that called function
        that instantiated us."""
        self.__finalized = False
        
        stack = inspect.stack()
        try:
            lframe = stack[0][0]
            while (lframe is not None) and (lframe.f_code.co_name != 'ScopeGuardian'):
                lframe = lframe.f_back
                
        finally:
            del stack

        if lframe is None:
            msg = '%s: Forgot to scope-guard your function???' % repr(self)
            raise RuntimeError, msg

        # ok, we're ready
        fname = lframe.f_code.co_name
        guardedFunc = lframe.f_locals['func']
        
        print '%s: instantiated by scope-guarded %s' % (repr(self), guardedFunc)
        guardedFunc.f_scopedObjs.append(self)
        
    def finalizeMaster(self):
        """Derived classes MUST define a self._finalize() method,
        where they do their finalization for scope exit."""
        print '%s: Finalize() being called' % repr(self)
        assert sys.getrefcount(self) == 5
        self._finalize()
        self.__finalized = True

    def __del__(self):
        """This just does some error checking, probably want to remove
        in production in case derived objects involved in cycles."""
        try:
            problem = not self.__finalized
        except AttributeError:
            print '%s: NeedsFinalization.__init__ not called for %s' % (repr(self), self.__class__)
            raise RuntimeError, msg

        if not problem:
            print '%s: Finalized properly' % repr(self)
        else:
            # this is being called because exception was thrown
            pass
            
def ScopeGuardian(func, *args, **kwargs):
    """Make func finalize its locals that said neede finalization.
    This is done by wrapping func function in a try/finally block.
    When the finally clause is executed, all instances of classes
    derived from NeedsFinalization, and instantiated in func, will be
    in a special attribute func.f_scopedObjs. We go over that list and
    call finalizeMaster() for each item in list."""
    try:
        func.f_scopedObjs = []
        func(*args, **kwargs)
        print 'Scoped variables created during call to %s: %s\n\n' \
              % (func, func.f_scopedObjs)
        
    finally:
        if func.f_scopedObjs != []:
            for obj in func.f_scopedObjs:
                obj.finalizeMaster()
            func.f_scopedObjs = []


def test():
    
    class TestFree:
        def _finalize():
            raise RuntimeError, 'finalize() not supposed to be called'

    class TestDanger(NeedsFinalization):
        def __init__(self):
            NeedsFinalization.__init__(self)
        
        def _finalize(self):
            """Override this. If you class inherits from a
            class derived from NeedsFinalization, make sure to
            call parent.finalize()."""
            pass
    
    class TestDangerNoInit(NeedsFinalization):
        def _finalize(self):
            pass

    def func1():
        ok = TestFree()
        danger = TestDanger()

    # test when forget to use ScopeGuardian
    try:
        hello = func1()
        assert False, 'Expected exception not thrown!'
    except RuntimeError, msg:
        print 'Expected exception caught: ', msg
        
    func1 = ScopeGuarded(func1)
    func1()

    def func2(objType):
        dontKnow = objType()

    func2 = ScopeGuarded(func2)
    #try:
    func2(TestDangerNoInit)
    func2(TestDanger)
        #assert False, 'Expected exception not thrown!'
    #except RuntimeError, msg:
        #print 'Expected exception caught: ', msg
        

if __name__ == '__main__':
    test()