* Guido van Rossum <guido@python.org> [2007-04-29 16:30:18 -0700]:

On 4/29/07, Jim Jewett <jimjjewett@gmail.com> wrote:

...

So it is a "keyword" in the sense that None is a keyword; not in the stronger sense that "if" is a keyword?

Um, how do you see those two differ? Is 'if' a keyword in the same sense as 'or', or in a different sense?

In my mind, 'if' and 'or' are "syntax", whereas things like 'None' or 'True' are "values"; even if None becomes an actual keyword, rather than a builtin. -- mithrandi, i Ainil en-Balandor, a faer Ambar

* Guido van Rossum <guido@python.org> [2007-04-29 18:19:20 -0700]:

...

In my mind, 'if' and 'or' are "syntax", whereas things like 'None' or 'True' are "values"; even if None becomes an actual keyword, rather than a builtin.

I'm sorry, but that is such an incredibly subjective difference that I can't do anything with it. String literals and numeric literals are syntax too, even though they are values. A keyword, or reserved word, is simply something that looks like an identifier but is converted into a different token (by the lexer or by something sitting between the lexer and the parse) before the parser sees it.

Let me try a less subjective description. Things like None, 2.3, 'foo', True are values or "expressions"; I'm not certain exactly what the term for these is in Python's grammar, but I basically mean something that can be on the RHS of an assignment.. However, something like 'for' or 'if' is part of some other grammatical construct, generally a statement or operator of some kind, so I tend to think of those differently. -- mithrandi, i Ainil en-Balandor, a faer Ambar

On 4/29/07, Guido van Rossum <guido@python.org> wrote:

On 4/29/07, Jim Jewett <jimjjewett@gmail.com> wrote:

...

So it is a "keyword" in the sense that None is a keyword; not in the stronger sense that "if" is a keyword?

Um, how do you see those two differ? Is 'if' a keyword in the same sense as 'or', or in a different sense?

I realize that in Python 2.5, None is not a full-fledged keyword but cannot be used as an assignment target. But that's only transitional. In 3.0 I imagine it becoming a keyword in the grammar (whose only appearance would be as one of the alternatives for 'atom'). And we're talking 3.0 here.

I think any concerns about it not being fit as a keyword would fall under two catagories or varying validity: 1) Too many keywords is a valid concern, because it complicates the language. 2) It just doesn't "feel" like a keyword. Less valid, unless it _really_ doesn't feel like a keyword. It doesn't feel like a keyword. But it doesn't feel too much not like a keyword. Anyway, I tried to address the concerns laid out, and I'm more than happy to alter the PEP to actually say "Lets implement this as a keyword", and I actually meant to keep more agnostic on that point in the proposal itself. I was more interested in covering the interface, at least to begin, than the actual implementation. Although, being able to have a solid, working reference implementation based on the frame lookups and such is nice, so we can see how it will actually work in real code, and even use it to backport code using the new super to just about any recent Python version. I also checked and PyPy does implement a sys._getframe() and a IronPython currently doesn't, but seems to plan on it (there is a placeholder, at present). I am not sure if notes on this belongs in the PEP or not. Draft Three follows for all. I think I'm turning off e-mail for the rest of this evening, so I'll catch up tomorrow. ------------------------------------------------------- PEP: XXX Title: New Super Version: $Revision$ Last-Modified: $Date$ Author: Calvin Spealman <ironfroggy@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Apr-2007 Python-Version: 2.6 Post-History: 28-Apr-2007, 29-Apr-2007 (1), 29-Apr-2007 (2) Abstract ======== The PEP defines the proposal to enhance the super builtin to work implicitly upon the class within which it is used and upon the instance the current function was called on. The premise of the new super usage suggested is as follows: super.foo(1, 2) to replace the old: super(Foo, self).foo(1, 2) Rationale ========= The current usage of super requires an explicit passing of both the class and instance it must operate from, requiring a breaking of the DRY (Don't Repeat Yourself) rule. This hinders any change in class name, and is often considered a wart by many. Specification ============= Within the specification section, some special terminology will be used to distinguish similar and closely related concepts. "Super type" will refer to the actual builtin type named "super". "Next Class/Type in the MRO" will refer to the class where attribute lookups will be performed by super, for example, in the following, A is the "Next class in the MRO" for the use of super. :: class A(object): def f(self): return 'A' class B(A): def f(self): super(B, self).f() # Here, A would be out "Next class in the # MRO", of course. A "super object" is simply an instance of the super type, which is associated with a class and possibly with an instance of that class. Finally, "new super" refers to the new super type, which will replace the original. Replacing the old usage of super, calls to the next class in the MRO (method resolution order) will be made without an explicit super object creation, by simply accessing an attribute on the super type directly, which will automatically apply the class and instance to perform the proper lookup. The following example demonstrates the use of this. :: class A(object): def f(self): return 'A' class B(A): def f(self): return 'B' + super.f() class C(A): def f(self): return 'C' + super.f() class D(B, C): def f(self): return 'D' + super.f() assert D().f() == 'DBCA' The proposal adds a dynamic attribute lookup to the super type, which will automatically determine the proper class and instance parameters. Each super attribute lookup identifies these parameters and performs the super lookup on the instance, as the current super implementation does with the explicit invokation of a super object upon a class and instance. The enhancements to the super type will define a new __getattr__ classmethod of the super type, which must look backwards to the previous frame and locate the instance object. This can be naively determined by located the local named by the first argument to the function. Using super outside of a function where this is a valid lookup for the instance can be considered undocumented in its behavior. This special method will actually be invoked on attribute lookups to the super type itself, as opposed to super objects, as the current implementation works. This may pose open issues, which are detailed below. "Every class will gain a new special attribute, __super__, which refers to an instance of the associated super object for that class" In this capacity, the new super also acts as its own descriptor, create an instance-specific super upon lookup. Much of this was discussed in the thread of the python-dev list, "Fixing super anyone?" [1]_. Open Issues ----------- __call__ methods '''''''''''''''' Backward compatability of the super type API raises some issues. Names, the lookup of the __call__ of the super type itself, which means a conflict with doing an actual super lookup of the __call__ attribute. Namely, the following is ambiguous in the current proposal: :: super.__call__(arg) Which means the backward compatible API, which involves instansiating the super type, will either not be possible, because it will actually do a super lookup on the __call__ attribute, or there will be no way to perform a super lookup on the __call__ attribute. Both seem unacceptable, so any suggestions are welcome. Actually keeping the old super around in 2.x and creating a completely new super type seperately may be the best option. A future import or even a simple import in 2.x of the new super type from some builtin module may offer a way to choose which each module uses, even mixing uses by binding to different names. Such a builtin module might be called 'newsuper'. This module is also the reference implementation, which I will present below. super type's new getattr '''''''''''''''''''''''' To give the behavior needed, the super type either needs a way to do dynamic lookup of attributes on the super type object itself or define a metaclass for the builtin type. This author is unsure which, if either, is possible with C- defined types. When should we create __super__ attributes? ''''''''''''''''''''''''''''''''''''''''''' They either need to be created on class creation or on __super__ attribute lookup. For the second, they could be cached, of course, which seems like it may be the best idea, if implicit creation of a super object for every class is considered too much overhead. How does it work in inner functions? '''''''''''''''''''''''''''''''''''' If a method defines a function and super is used inside of it, how does this work? The frame looking and instance detection breaks here. However, if there can be some unambiguous way to use both the new super form and still be able to explicitly name the type and instance, I think its an acceptable tradeoff to simply be explicit in these cases, rather than add weird super-specific lookup rules in these cases. An example of such a problematic bit of code is: :: class B(A): def f(self): def g(): return super.f() return g() Should super actually become a keyword? ''''''''''''''''''''''''''''''''''''''' This would solve many of the problems and allow more direct implementation of super into the language proper. However, some are against the actual keyword- ization of super. The simplest solution is often the correct solution and the simplest solution may well not be adding additional keywords to the language when they are not needed. Still, it may solve many of the other open issues. Can we also allow super()? '''''''''''''''''''''''''' There is strong sentiment for and against this, but implementation and style concerns are obvious. Particularly, that its "magical" and that super() would differ from super.__call__(), being very unpythonic. Reference Implementation ======================== This implementation was a cooperative contribution in the original thread [1]_. :: #!/usr/bin/env python # # newsuper.py import sys class SuperMetaclass(type): def __getattr__(cls, attr): calling_frame = sys._getframe().f_back instance_name = calling_frame.f_code.co_varnames[0] instance = calling_frame.f_locals[instance_name] return getattr(instance.__super__, attr) class Super(object): __metaclass__ = SuperMetaclass def __init__(self, type, obj=None): if isinstance(obj, Super): obj = obj.__obj__ self.__type__ = type self.__obj__ = obj def __get__(self, obj, cls=None): if obj is None: raise Exception('only supports instances') else: return Super(self.__type__, obj) def __getattr__(self, attr): mro = iter(self.__obj__.__class__.__mro__) for cls in mro: if cls is self.__type__: break for cls in mro: if attr in cls.__dict__: x = cls.__dict__[attr] if hasattr(x, '__get__'): x = x.__get__(self, cls) return x raise AttributeError, attr class autosuper(type): def __init__(cls, name, bases, clsdict): cls.__super__ = Super(cls) if __name__ == '__main__': class A(object): __metaclass__ = autosuper def f(self): return 'A' class B(A): def f(self): return 'B' + Super.f() class C(A): def f(self): return 'C' + Super.f() class D(B, C): def f(self, arg=None): var = None return 'D' + Super.f() assert D().f() == 'DBCA' Alternative Proposals ===================== No Changes ---------- Although its always attractive to just keep things how they are, people have sought a change in the usage of super calling for some time, and for good reason, all mentioned previously. * Decoupling from the class name (which might not even be bound to the right class anymore! [2]_) * Simpler looking, cleaner super calls would be better super(__this_class__, self) --------------------------- This is nearly an anti-proposal, as it basically relies on the acceptance of the __this_class__ PEP, which proposes a special name that would always be bound to the class within which it is used. If that is accepted, __this_class__ could simply be used instead of the class' name explicitly, solving the name binding issues [2]_. self.__super__.foo(*args) ------------------------- The __super__ attribute is mentioned in this PEP in several places, and could be a candidate for the completel solution, actually using it explicitly instead of any super usage directly. However, double-underscore names are usually an internal detail, and attempted to be kept out of everyday code. super(self, *args) or __super__(self, *args) -------------------------------------------- This solution only solves the problem of the type indication, does not handle differently named super methods, and is explicit about the name of the instance. It is less flexable without being able to enacted on other method names, in cases where that is needed. One use case this fails is where a base- class has a factory classmethod and a subclass has two factory classmethods, both of which needing to properly make super calls to the one in the base- class. super.foo(self, *args) ---------------------- This variation actually eliminates the problems with locating the proper instance, and if any of the alternatives were pushed into the spotlight, I would want it to be this one. super or super() ---------------- This proposal leaves no room for different names, signatures, or application to other classes, or instances. A way to allow some similar use alongside the normal proposal would be favorable, encouraging good design of multiple inheritence trees and compatible methods. History ======= 29-Apr-2007 - Changed title from "Super As A Keyword" to "New Super" - Updated much of the language and added a terminology section for clarification in confusing places. - Added reference implementation and history sections. References ========== .. [1] Fixing super anyone? (http://mail.python.org/pipermail/python-3000/2007-April/006667.html) Copyright ========= This document has been placed in the public domain.  .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

* Guido van Rossum <guido@python.org> [2007-04-29 16:30:18 -0700]:

On 4/29/07, Jim Jewett <jimjjewett@gmail.com> wrote:

...

So it is a "keyword" in the sense that None is a keyword; not in the stronger sense that "if" is a keyword?

Um, how do you see those two differ? Is 'if' a keyword in the same sense as 'or', or in a different sense?

In my mind, 'if' and 'or' are "syntax", whereas things like 'None' or 'True' are "values"; even if None becomes an actual keyword, rather than a builtin. -- mithrandi, i Ainil en-Balandor, a faer Ambar

* Guido van Rossum <guido@python.org> [2007-04-29 18:19:20 -0700]:

...

In my mind, 'if' and 'or' are "syntax", whereas things like 'None' or 'True' are "values"; even if None becomes an actual keyword, rather than a builtin.

I'm sorry, but that is such an incredibly subjective difference that I can't do anything with it. String literals and numeric literals are syntax too, even though they are values. A keyword, or reserved word, is simply something that looks like an identifier but is converted into a different token (by the lexer or by something sitting between the lexer and the parse) before the parser sees it.

Let me try a less subjective description. Things like None, 2.3, 'foo', True are values or "expressions"; I'm not certain exactly what the term for these is in Python's grammar, but I basically mean something that can be on the RHS of an assignment.. However, something like 'for' or 'if' is part of some other grammatical construct, generally a statement or operator of some kind, so I tend to think of those differently. -- mithrandi, i Ainil en-Balandor, a faer Ambar

On 4/29/07, Guido van Rossum <guido@python.org> wrote:

On 4/29/07, Jim Jewett <jimjjewett@gmail.com> wrote:

...

So it is a "keyword" in the sense that None is a keyword; not in the stronger sense that "if" is a keyword?

Um, how do you see those two differ? Is 'if' a keyword in the same sense as 'or', or in a different sense?

I realize that in Python 2.5, None is not a full-fledged keyword but cannot be used as an assignment target. But that's only transitional. In 3.0 I imagine it becoming a keyword in the grammar (whose only appearance would be as one of the alternatives for 'atom'). And we're talking 3.0 here.

I think any concerns about it not being fit as a keyword would fall under two catagories or varying validity: 1) Too many keywords is a valid concern, because it complicates the language. 2) It just doesn't "feel" like a keyword. Less valid, unless it _really_ doesn't feel like a keyword. It doesn't feel like a keyword. But it doesn't feel too much not like a keyword. Anyway, I tried to address the concerns laid out, and I'm more than happy to alter the PEP to actually say "Lets implement this as a keyword", and I actually meant to keep more agnostic on that point in the proposal itself. I was more interested in covering the interface, at least to begin, than the actual implementation. Although, being able to have a solid, working reference implementation based on the frame lookups and such is nice, so we can see how it will actually work in real code, and even use it to backport code using the new super to just about any recent Python version. I also checked and PyPy does implement a sys._getframe() and a IronPython currently doesn't, but seems to plan on it (there is a placeholder, at present). I am not sure if notes on this belongs in the PEP or not. Draft Three follows for all. I think I'm turning off e-mail for the rest of this evening, so I'll catch up tomorrow. ------------------------------------------------------- PEP: XXX Title: New Super Version: $Revision$ Last-Modified: $Date$ Author: Calvin Spealman <ironfroggy@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Apr-2007 Python-Version: 2.6 Post-History: 28-Apr-2007, 29-Apr-2007 (1), 29-Apr-2007 (2) Abstract ======== The PEP defines the proposal to enhance the super builtin to work implicitly upon the class within which it is used and upon the instance the current function was called on. The premise of the new super usage suggested is as follows: super.foo(1, 2) to replace the old: super(Foo, self).foo(1, 2) Rationale ========= The current usage of super requires an explicit passing of both the class and instance it must operate from, requiring a breaking of the DRY (Don't Repeat Yourself) rule. This hinders any change in class name, and is often considered a wart by many. Specification ============= Within the specification section, some special terminology will be used to distinguish similar and closely related concepts. "Super type" will refer to the actual builtin type named "super". "Next Class/Type in the MRO" will refer to the class where attribute lookups will be performed by super, for example, in the following, A is the "Next class in the MRO" for the use of super. :: class A(object): def f(self): return 'A' class B(A): def f(self): super(B, self).f() # Here, A would be out "Next class in the # MRO", of course. A "super object" is simply an instance of the super type, which is associated with a class and possibly with an instance of that class. Finally, "new super" refers to the new super type, which will replace the original. Replacing the old usage of super, calls to the next class in the MRO (method resolution order) will be made without an explicit super object creation, by simply accessing an attribute on the super type directly, which will automatically apply the class and instance to perform the proper lookup. The following example demonstrates the use of this. :: class A(object): def f(self): return 'A' class B(A): def f(self): return 'B' + super.f() class C(A): def f(self): return 'C' + super.f() class D(B, C): def f(self): return 'D' + super.f() assert D().f() == 'DBCA' The proposal adds a dynamic attribute lookup to the super type, which will automatically determine the proper class and instance parameters. Each super attribute lookup identifies these parameters and performs the super lookup on the instance, as the current super implementation does with the explicit invokation of a super object upon a class and instance. The enhancements to the super type will define a new __getattr__ classmethod of the super type, which must look backwards to the previous frame and locate the instance object. This can be naively determined by located the local named by the first argument to the function. Using super outside of a function where this is a valid lookup for the instance can be considered undocumented in its behavior. This special method will actually be invoked on attribute lookups to the super type itself, as opposed to super objects, as the current implementation works. This may pose open issues, which are detailed below. "Every class will gain a new special attribute, __super__, which refers to an instance of the associated super object for that class" In this capacity, the new super also acts as its own descriptor, create an instance-specific super upon lookup. Much of this was discussed in the thread of the python-dev list, "Fixing super anyone?" [1]_. Open Issues ----------- __call__ methods '''''''''''''''' Backward compatability of the super type API raises some issues. Names, the lookup of the __call__ of the super type itself, which means a conflict with doing an actual super lookup of the __call__ attribute. Namely, the following is ambiguous in the current proposal: :: super.__call__(arg) Which means the backward compatible API, which involves instansiating the super type, will either not be possible, because it will actually do a super lookup on the __call__ attribute, or there will be no way to perform a super lookup on the __call__ attribute. Both seem unacceptable, so any suggestions are welcome. Actually keeping the old super around in 2.x and creating a completely new super type seperately may be the best option. A future import or even a simple import in 2.x of the new super type from some builtin module may offer a way to choose which each module uses, even mixing uses by binding to different names. Such a builtin module might be called 'newsuper'. This module is also the reference implementation, which I will present below. super type's new getattr '''''''''''''''''''''''' To give the behavior needed, the super type either needs a way to do dynamic lookup of attributes on the super type object itself or define a metaclass for the builtin type. This author is unsure which, if either, is possible with C- defined types. When should we create __super__ attributes? ''''''''''''''''''''''''''''''''''''''''''' They either need to be created on class creation or on __super__ attribute lookup. For the second, they could be cached, of course, which seems like it may be the best idea, if implicit creation of a super object for every class is considered too much overhead. How does it work in inner functions? '''''''''''''''''''''''''''''''''''' If a method defines a function and super is used inside of it, how does this work? The frame looking and instance detection breaks here. However, if there can be some unambiguous way to use both the new super form and still be able to explicitly name the type and instance, I think its an acceptable tradeoff to simply be explicit in these cases, rather than add weird super-specific lookup rules in these cases. An example of such a problematic bit of code is: :: class B(A): def f(self): def g(): return super.f() return g() Should super actually become a keyword? ''''''''''''''''''''''''''''''''''''''' This would solve many of the problems and allow more direct implementation of super into the language proper. However, some are against the actual keyword- ization of super. The simplest solution is often the correct solution and the simplest solution may well not be adding additional keywords to the language when they are not needed. Still, it may solve many of the other open issues. Can we also allow super()? '''''''''''''''''''''''''' There is strong sentiment for and against this, but implementation and style concerns are obvious. Particularly, that its "magical" and that super() would differ from super.__call__(), being very unpythonic. Reference Implementation ======================== This implementation was a cooperative contribution in the original thread [1]_. :: #!/usr/bin/env python # # newsuper.py import sys class SuperMetaclass(type): def __getattr__(cls, attr): calling_frame = sys._getframe().f_back instance_name = calling_frame.f_code.co_varnames[0] instance = calling_frame.f_locals[instance_name] return getattr(instance.__super__, attr) class Super(object): __metaclass__ = SuperMetaclass def __init__(self, type, obj=None): if isinstance(obj, Super): obj = obj.__obj__ self.__type__ = type self.__obj__ = obj def __get__(self, obj, cls=None): if obj is None: raise Exception('only supports instances') else: return Super(self.__type__, obj) def __getattr__(self, attr): mro = iter(self.__obj__.__class__.__mro__) for cls in mro: if cls is self.__type__: break for cls in mro: if attr in cls.__dict__: x = cls.__dict__[attr] if hasattr(x, '__get__'): x = x.__get__(self, cls) return x raise AttributeError, attr class autosuper(type): def __init__(cls, name, bases, clsdict): cls.__super__ = Super(cls) if __name__ == '__main__': class A(object): __metaclass__ = autosuper def f(self): return 'A' class B(A): def f(self): return 'B' + Super.f() class C(A): def f(self): return 'C' + Super.f() class D(B, C): def f(self, arg=None): var = None return 'D' + Super.f() assert D().f() == 'DBCA' Alternative Proposals ===================== No Changes ---------- Although its always attractive to just keep things how they are, people have sought a change in the usage of super calling for some time, and for good reason, all mentioned previously. * Decoupling from the class name (which might not even be bound to the right class anymore! [2]_) * Simpler looking, cleaner super calls would be better super(__this_class__, self) --------------------------- This is nearly an anti-proposal, as it basically relies on the acceptance of the __this_class__ PEP, which proposes a special name that would always be bound to the class within which it is used. If that is accepted, __this_class__ could simply be used instead of the class' name explicitly, solving the name binding issues [2]_. self.__super__.foo(*args) ------------------------- The __super__ attribute is mentioned in this PEP in several places, and could be a candidate for the completel solution, actually using it explicitly instead of any super usage directly. However, double-underscore names are usually an internal detail, and attempted to be kept out of everyday code. super(self, *args) or __super__(self, *args) -------------------------------------------- This solution only solves the problem of the type indication, does not handle differently named super methods, and is explicit about the name of the instance. It is less flexable without being able to enacted on other method names, in cases where that is needed. One use case this fails is where a base- class has a factory classmethod and a subclass has two factory classmethods, both of which needing to properly make super calls to the one in the base- class. super.foo(self, *args) ---------------------- This variation actually eliminates the problems with locating the proper instance, and if any of the alternatives were pushed into the spotlight, I would want it to be this one. super or super() ---------------- This proposal leaves no room for different names, signatures, or application to other classes, or instances. A way to allow some similar use alongside the normal proposal would be favorable, encouraging good design of multiple inheritence trees and compatible methods. History ======= 29-Apr-2007 - Changed title from "Super As A Keyword" to "New Super" - Updated much of the language and added a terminology section for clarification in confusing places. - Added reference implementation and history sections. References ========== .. [1] Fixing super anyone? (http://mail.python.org/pipermail/python-3000/2007-April/006667.html) Copyright ========= This document has been placed in the public domain.  .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: