Required to call superclass __init__

I'm not talking about having the runtime call the superclass __init__ for you, as I am aware of the arguments over it and I am against it myself. I'm talking about checking whether it's been called within a subclass's own __init__. There are many kinds of objects with such complex underpinnings or initialization that leaving out the call to superclass __init__ would be disastrous. There are two situations I can think of where enforcing its invocation could be useful: a corporate environment and a teaching environment. (I've done the former and I'm working in the latter.) If someone forgets to call a superclass __init__, problems may not show up until much later. Even if they do show up immediately, it's almost never obvious what the real problem is, especially to someone who is new to programming or is working on someone else's code. I've got a working prototype metaclass and class instance (require_super) and decorator (super_required). Decorating a require_super method with @super_required will require any subclass override to call its superclass method, or it throws a TypeError upon exiting the subclass method. Here's how it works on the __init__ problem: class A(require_super): @super_required def __init__(self): pass a = A() # No problem class B(A): def __init__(self): super(B, self).__init__() b = B() # No problem class C(B): def __init__(self): pass # this could be a problem c = C() # TypeError: C.__init__: no super call class D(C): def __init__(self): super(D, self).__init__() d = D() # TypeError: C.__init__: no super call As long as A.__init__ is eventually called, it doesn't raise a TypeError. There's not much magic involved (as metaclasses go), just explicit and implicit method wrappers, and no crufty-looking magic words in the subclasses. Not calling the superclass method results in immediate runtime feedback. I've tested this on a medium-small, real-life single-inheritance hierarchy and it seems to work just fine. (I *think* it should work with multiple inheritance.) Two questions: 1. Is the original problem (missed superclass method calls) big enough to warrant language, runtime, or library support for a similar solution? 2. Does anybody but me think this is a great idea? Neil

On Tue, Nov 13, 2007 at 12:23:40AM -0700, Neil Toronto wrote:
I've got a working prototype metaclass and class instance (require_super) and decorator (super_required).
Chicken and egg problem, in my eyes. If the user is clever enough to use the class and the decorator isn't she clever enough to call inherited __init__? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

On 11/13/07, Oleg Broytmann <phd@phd.pp.ru> wrote:
Is this restricted to __init__ (and __new__?) or could it be used on any method? Is there (and should there be?) a way around it, by catching the TypeError? By creating a decoy object to call super on?
It may not be the same user. A library or framework writer would create the base class and use the decorator to (somewhat) ensure that subclasses meet the full interface requirements. A subclass writer should call the super.__init__ because it is in the API, but Neil's metaclass makes it easier to debug if they forget. -jJ

Jim Jewett wrote:
It can be used on any method.
Is there (and should there be?) a way around it, by catching the TypeError? By creating a decoy object to call super on?
Definitely should be, and I made one because I plan on using this myself. :) Currently, you can set self.<method>_super = True (or self.__<method>__super = True) instead of doing the superclass method call. (Yes, it currently litters the class instance with flags, but that's an implementation detail.) If you're not going to call the superclass method, you need to state that explicitly. class C(B): def __init__(self): self.__init__super = True c = C() # No problem I've fiddled with the idea of having a redecoration with @super_required remove the requirement from the current method but place it back on future overrides. Maybe a @super_not_required could remove it completely.
Exactly so. Neil

Since I know you're all dying to see the code... ;) This works for instance methods, classmethods, staticmethods (if cls or self is the first parameter, as in __new__), and probably most decorated methods. Current Issues I Can Think Of: - @classmethod overrides that don't redecorate with @classmethod always raise TypeError (maybe not such a bad thing) - exceptions can cause entries in the __supercalls__ set to accumulate unboundedly (can be fixed) Anyway, here's a concrete implementation. Is the "missed super call" problem big or annoying enough to warrant having language, runtime, or library support for a similar solution? import threading import types class _supercall_set(object): def __init__(self, *args, **kwargs): # Removing threading.local() should make this faster # but not thread-safe self.loc = threading.local() self.loc.s = set(*args, **kwargs) def add(self, key): self.loc.s.add(key) def discard(self, key): self.loc.s.discard(key) def __contains__(self, key): return self.loc.s.__contains__(key) def __repr__(self): return self.loc.s.__repr__() def _unwrap_rewrap(func): '''For supported types (classmethod, staticmethod, function), returns the actual function and a function to re-wrap it, if necessary. Raises TypeError if func's type isn't supported.''' if isinstance(func, classmethod): return func.__get__(func).im_func, type(func) elif isinstance(func, staticmethod): return func.__get__(func), type(func) elif isinstance(func, types.FunctionType): return func, lambda func: func raise TypeError('unsupported type %s' % type(func)) def super_required(func): ''' Marks a method as requiring subclass overrides to call it, either directly or via a super() call. Works with all undecorated methods, classmethods, staticmethods (fragile: only if 'cls' or 'self' is the first parameter) including __new__, and probably most other decorated methods. Correct operation is guaranteed only when the method is in a subclass of require_super. If a super_required override has a superclass method that is also super_required, the override will not be required to call the superclass method, either directly or via a super() call. The superclass call requirement can be cancelled for a method and methods of the same name in all future subclasses using the super_not_required decorator. The implementation should be as thread-safe as the classes it's used in. Recursion should work as long as the last, innermost call calls the superclass method. (It's usually best to avoid it.) This is not robust to method injection, but then again, what is? Examples: class A(require_super): @super_required def __init__(self): pass class B(A): def __init__(self): pass b = B() # TypeError: B.__init__: no super call # B.__init__ needs a super(B, self).__init__() class C(require_super): @super_required @classmethod # order of decorators doesn't matter def clsmeth(cls): pass class D(C): @classmethod def clsmeth(cls): pass d = D() d.clsmeth() # TypeError: D.clsmeth: no super call # C.clsmeth needs a super(C, cls).clsmeth() ''' func, rewrap = _unwrap_rewrap(func) name = func.func_name def super_wrapper(self_or_cls, *args, **kwargs): retval = func(self_or_cls, *args, **kwargs) # Flag that the super call happened self_or_cls.__supercalls__.discard((id(self_or_cls), name)) return retval super_wrapper.func_name = func.func_name super_wrapper.func_doc = func.func_doc super_wrapper.__super_required__ = True # Pass it down return rewrap(super_wrapper) def super_not_required(func): '''Marks a method as no longer requiring subclass overrides to call it. This is only meaningful for methods in subclasses of require_super.''' func.__super_required__ = False return func def _get_sub_wrapper(func, class_name, method_name): '''Returns a wrapper function that: 1. Adds key to __supercalls__ 2. Calls the wrapped function 3. Checks for key in __supercalls__ - if there, raises TypeError''' def sub_wrapper(self_or_cls, *args, **kwargs): key = (id(self_or_cls), method_name) self_or_cls.__supercalls__.add(key) retval = func(self_or_cls, *args, **kwargs) if key not in self_or_cls.__supercalls__: return retval self_or_cls.__supercalls__.discard(key) raise TypeError("%s.%s: no super call" % (class_name, method_name)) sub_wrapper.func_name = func.func_name sub_wrapper.func_doc = func.func_doc sub_wrapper.__super_required__ = True # Pass it down return sub_wrapper class _require_super_meta(type): def __new__(typ, cls_name, bases, dct): # Search through all attributes for method_name, func in dct.items(): try: func, rewrap = _unwrap_rewrap(func) except TypeError: continue # unsupported type if hasattr(func, '__super_required__'): continue # decorated - don't wrap it again # See if a base class's method is __super_required__ for base in bases: try: if getattr(base, method_name).__super_required__: break except AttributeError: pass # not there or no __super_required__ else: continue # outer loop # Wrap up the function newfunc = _get_sub_wrapper(func, cls_name, method_name) dct[method_name] = rewrap(newfunc) return type.__new__(typ, cls_name, bases, dct) class require_super(object): '''Inheriting from require_super makes super_required and super_not_required decorators work.''' __metaclass__ = _require_super_meta # This will be visible to classes and instances __supercalls__ = _supercall_set()

On 11/14/07, Neil Toronto <ntoronto@cs.byu.edu> wrote:
Current Issues I Can Think Of:
recipe, yes. Library or more? I'm not sure -- and I don't think this is ready yet. It feels too complicated, as though there may still be plenty of simplifications that should happen before it gets frozen. I don't yet see what those simplifications should actually be, but maybe someone else will if you publish and wait long enough. -jJ

Jim Jewett wrote:
The first thing I noticed was that the naming scheme is confusing. Between required_super and super_required, neither of them indicate to me which is the function decorator and which is the base class. Furthermore, I don't see why required_super (the base class) needs a distinct name. Perhaps I am being a bit to clever, but couldn't we just overload the __new__ method of the base class. def _super_required(func): ... class super_required(object): ... def __new__(cls, *func): if len(func) > 0: return _super_required(*func) return object.__new__(cls) Leaving your example now being spelled as: class A(super_required): @super_required def __init__(self): pass I can't think of a case that the the base class would ever be passed arguments, so this seems ok and rids us of the naming oddities. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On Tue, Nov 13, 2007 at 12:23:40AM -0700, Neil Toronto wrote:
I've got a working prototype metaclass and class instance (require_super) and decorator (super_required).
Chicken and egg problem, in my eyes. If the user is clever enough to use the class and the decorator isn't she clever enough to call inherited __init__? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

On 11/13/07, Oleg Broytmann <phd@phd.pp.ru> wrote:
Is this restricted to __init__ (and __new__?) or could it be used on any method? Is there (and should there be?) a way around it, by catching the TypeError? By creating a decoy object to call super on?
It may not be the same user. A library or framework writer would create the base class and use the decorator to (somewhat) ensure that subclasses meet the full interface requirements. A subclass writer should call the super.__init__ because it is in the API, but Neil's metaclass makes it easier to debug if they forget. -jJ

Jim Jewett wrote:
It can be used on any method.
Is there (and should there be?) a way around it, by catching the TypeError? By creating a decoy object to call super on?
Definitely should be, and I made one because I plan on using this myself. :) Currently, you can set self.<method>_super = True (or self.__<method>__super = True) instead of doing the superclass method call. (Yes, it currently litters the class instance with flags, but that's an implementation detail.) If you're not going to call the superclass method, you need to state that explicitly. class C(B): def __init__(self): self.__init__super = True c = C() # No problem I've fiddled with the idea of having a redecoration with @super_required remove the requirement from the current method but place it back on future overrides. Maybe a @super_not_required could remove it completely.
Exactly so. Neil

Since I know you're all dying to see the code... ;) This works for instance methods, classmethods, staticmethods (if cls or self is the first parameter, as in __new__), and probably most decorated methods. Current Issues I Can Think Of: - @classmethod overrides that don't redecorate with @classmethod always raise TypeError (maybe not such a bad thing) - exceptions can cause entries in the __supercalls__ set to accumulate unboundedly (can be fixed) Anyway, here's a concrete implementation. Is the "missed super call" problem big or annoying enough to warrant having language, runtime, or library support for a similar solution? import threading import types class _supercall_set(object): def __init__(self, *args, **kwargs): # Removing threading.local() should make this faster # but not thread-safe self.loc = threading.local() self.loc.s = set(*args, **kwargs) def add(self, key): self.loc.s.add(key) def discard(self, key): self.loc.s.discard(key) def __contains__(self, key): return self.loc.s.__contains__(key) def __repr__(self): return self.loc.s.__repr__() def _unwrap_rewrap(func): '''For supported types (classmethod, staticmethod, function), returns the actual function and a function to re-wrap it, if necessary. Raises TypeError if func's type isn't supported.''' if isinstance(func, classmethod): return func.__get__(func).im_func, type(func) elif isinstance(func, staticmethod): return func.__get__(func), type(func) elif isinstance(func, types.FunctionType): return func, lambda func: func raise TypeError('unsupported type %s' % type(func)) def super_required(func): ''' Marks a method as requiring subclass overrides to call it, either directly or via a super() call. Works with all undecorated methods, classmethods, staticmethods (fragile: only if 'cls' or 'self' is the first parameter) including __new__, and probably most other decorated methods. Correct operation is guaranteed only when the method is in a subclass of require_super. If a super_required override has a superclass method that is also super_required, the override will not be required to call the superclass method, either directly or via a super() call. The superclass call requirement can be cancelled for a method and methods of the same name in all future subclasses using the super_not_required decorator. The implementation should be as thread-safe as the classes it's used in. Recursion should work as long as the last, innermost call calls the superclass method. (It's usually best to avoid it.) This is not robust to method injection, but then again, what is? Examples: class A(require_super): @super_required def __init__(self): pass class B(A): def __init__(self): pass b = B() # TypeError: B.__init__: no super call # B.__init__ needs a super(B, self).__init__() class C(require_super): @super_required @classmethod # order of decorators doesn't matter def clsmeth(cls): pass class D(C): @classmethod def clsmeth(cls): pass d = D() d.clsmeth() # TypeError: D.clsmeth: no super call # C.clsmeth needs a super(C, cls).clsmeth() ''' func, rewrap = _unwrap_rewrap(func) name = func.func_name def super_wrapper(self_or_cls, *args, **kwargs): retval = func(self_or_cls, *args, **kwargs) # Flag that the super call happened self_or_cls.__supercalls__.discard((id(self_or_cls), name)) return retval super_wrapper.func_name = func.func_name super_wrapper.func_doc = func.func_doc super_wrapper.__super_required__ = True # Pass it down return rewrap(super_wrapper) def super_not_required(func): '''Marks a method as no longer requiring subclass overrides to call it. This is only meaningful for methods in subclasses of require_super.''' func.__super_required__ = False return func def _get_sub_wrapper(func, class_name, method_name): '''Returns a wrapper function that: 1. Adds key to __supercalls__ 2. Calls the wrapped function 3. Checks for key in __supercalls__ - if there, raises TypeError''' def sub_wrapper(self_or_cls, *args, **kwargs): key = (id(self_or_cls), method_name) self_or_cls.__supercalls__.add(key) retval = func(self_or_cls, *args, **kwargs) if key not in self_or_cls.__supercalls__: return retval self_or_cls.__supercalls__.discard(key) raise TypeError("%s.%s: no super call" % (class_name, method_name)) sub_wrapper.func_name = func.func_name sub_wrapper.func_doc = func.func_doc sub_wrapper.__super_required__ = True # Pass it down return sub_wrapper class _require_super_meta(type): def __new__(typ, cls_name, bases, dct): # Search through all attributes for method_name, func in dct.items(): try: func, rewrap = _unwrap_rewrap(func) except TypeError: continue # unsupported type if hasattr(func, '__super_required__'): continue # decorated - don't wrap it again # See if a base class's method is __super_required__ for base in bases: try: if getattr(base, method_name).__super_required__: break except AttributeError: pass # not there or no __super_required__ else: continue # outer loop # Wrap up the function newfunc = _get_sub_wrapper(func, cls_name, method_name) dct[method_name] = rewrap(newfunc) return type.__new__(typ, cls_name, bases, dct) class require_super(object): '''Inheriting from require_super makes super_required and super_not_required decorators work.''' __metaclass__ = _require_super_meta # This will be visible to classes and instances __supercalls__ = _supercall_set()

On 11/14/07, Neil Toronto <ntoronto@cs.byu.edu> wrote:
Current Issues I Can Think Of:
recipe, yes. Library or more? I'm not sure -- and I don't think this is ready yet. It feels too complicated, as though there may still be plenty of simplifications that should happen before it gets frozen. I don't yet see what those simplifications should actually be, but maybe someone else will if you publish and wait long enough. -jJ

Jim Jewett wrote:
The first thing I noticed was that the naming scheme is confusing. Between required_super and super_required, neither of them indicate to me which is the function decorator and which is the base class. Furthermore, I don't see why required_super (the base class) needs a distinct name. Perhaps I am being a bit to clever, but couldn't we just overload the __new__ method of the base class. def _super_required(func): ... class super_required(object): ... def __new__(cls, *func): if len(func) > 0: return _super_required(*func) return object.__new__(cls) Leaving your example now being spelled as: class A(super_required): @super_required def __init__(self): pass I can't think of a case that the the base class would ever be passed arguments, so this seems ok and rids us of the naming oddities. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
participants (4)
-
Jim Jewett
-
Neil Toronto
-
Oleg Broytmann
-
Scott Dial