A user story concerning things knowing their own names
I just experienced an obscure bug resulting from copying and pasting an overridable_property and forgetting to change the passed-in name: content_size = overridable_property('size', "Size of the content area.") which should have been content_size = overridable_property('content_size', "Size of the content area.") This was quite difficult to track down, because the result was to effectively make it an alias of *another* property I have called 'size'. They happen to be near enough to the same thing that the error went unnoticed for quite some time. If I could write my overridable_property declarations without having to repeat the name, this kind of thing would not be able to happen. -- Greg
On Wed, Mar 16, 2011 at 9:52 AM, Westley Martínez <anikom15@gmail.com> wrote:
If it went unnoticed, what's the matter?
Unnoticed just means it was a silent failure rather than a noisy one. It doesn't mean the bug wasn't resulting erroneous output. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Westley Martínez wrote:
If it went unnoticed, what's the matter?
It only went unnoticed because until recently I hadn't run any code that depended critically on it being right. A bug is still a bug even if you haven't found it yet! When it did became apparent that something was wrong, it still took quite a long time to track down the cause. I was looking for a bug in the get_content_size() method, whereas it was actually calling a different method altogether. Having attribute accesses go astray like that is not something you expect. -- Greg
On 03/16/2011 05:28 AM, Greg Ewing wrote:
As I suggested in my email on the Assignment Decorators thread this morning, you could achieve this in current Python, no extension needed: def assign(fn): return fn(fn.__name__) @assign def content_size(name): return overridable_property(name, "Size of the content area.") How bad do you want it? ;-) /larry/
On 16 March 2011 11:10, Larry Hastings <larry@hastings.org> wrote:
And building on this sightly you could do the following for namedtuple:
All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mar 16, 2011, at 1:49 PM, Terry Reedy wrote:
If make_namedtuple were added to collections, then one could import *that* instead of 'namedtuple' itself.
Post a recipe somewhere. My bet is that it will have a near zero adoption rate. -1 on all of these random ideas to solve what is basically a non-problem. The name of a named tuple, or a class, or a function, or a variable typically is used many times, not just in the definition. If you make up a new factory function that uses inspect magic to guess user's intend target name, you will have introduced unnecessary complexity and will likely introduce unexpected behaviors and bugs. Raymond
On 3/16/2011 4:49 PM, Terry Reedy wrote:
I am not sure how serious I was when I wrote that, but after further thought, I retract it as a serious suggestion, if indeed it was ;-). I think both decorators are 'cute' as examples of what one can do with decorators, and possible cookbook recipes, but not something we really need in the stdlib. I mostly said something because they do not require new syntax and I do not like any of the new syntax proposals. There are many aspects of the rationale for decorators that do not apply to the assignment case. 1. Explicit function wrapping requires the name in tripicate rather than duplicate. Moreover, some interface wrappers for functions need fairly long, multi-component names. I am pretty sure such need is much less for named tuples and properties. 2. The triplicates are *not* on the same line but may be separated by an arbitrary number of lines. This not only impedes checking that all three are the same, but may also impede understanding of the function code. That sometimes really requires knowing how or what the function will be wrapped as. This is greatly aided by bringing the wrapping function back up to (above) the def line. For instance, @classmethod and @staticmethod explain the non-standard signature with 'cls' or '' instead of 'self'. For another example, twisted.internet.defer.inLineCallbacks is a decorator that wraps a generator function as a twisted.internet.defer.Deferred object. Knowing that a particular generator function implements a series of data/error callbacks tor twisted greatly helps in understanding it. The idea is 'twisted' enough as it is ;-). -- Terry Jan Reedy
Larry Hastings wrote:
Not quite badly enough to replace all my existing overridable_property declarations with something that ugly. :-) But I appreciate the suggestion, and I'll keep it in mind next time I'm designing anything similar. -- Greg
On 16 March 2011 19:18, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
You should be able to replace it with something like: @do_something def content_size(): return "Size of the content area." Where do_something calls the function to get the string argument and takes the name from the function.__name__ (and uses them to create and return the overridable_property). Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
I'll note this general problem is also present in any of the declarative ORMs, which use silly hacks to tell descriptors their name. Like you have: class Table(ORM): name = StringColumn() Another case where I've noticed a problem is any kind of descriptor that needs its own storage; the name of the property gives a possible stable namespace for the value, but without the name you either have to pass in a name or the storage area becomes volatile. For instance, a read-only descriptor: http://svn.colorstudy.com/home/ianb/recipes/setonce.py You can solve this in, e.g., the ORM class by doing things when a class is created -- but it requires very specific cooperation between the class and the descriptor. Everyone really does it different ways. One could imagine an extension of the descriptor protocol, where on class creation you called something like attr.__addtoclass__(cls, name) (for all attributes of the class that define that method) -- which if you just want the name you'd simply save that name in your object and return self. On Wed, Mar 16, 2011 at 4:28 AM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
On 3/16/11 12:13 PM, Ian Bicking wrote:
If we were to extend the descriptor protocol, I think it would be better to extend it by providing __get_ex__(self, instance, owner, name), __set_ex__(self, instance, name, value), etc. methods that would be called in preference over the current __get__() and __set__() methods. This allows descriptor objects to be reused. We do this reasonably often with the Traits package (which overrides __getattribute__() to essentially implement this __get_ex__()/__set_ex__() protocol by different names). For example, we have a complicated trait for specifying colors: ColorTrait = Trait("black", Tuple, List, Str, color_table) It would be nice to simply reuse this object everywhere: class Figure(HasTraits): background = ColorTrait foreground = ColorTrait The alternative is to only provide factories such that you always get a new descriptor: ColorTrait = lambda: Trait("black", Tuple, List, Str, color_table) class Figure(HasTraits): background = ColorTrait() foreground = ColorTrait() However, this makes these new descriptors inconsistent with current descriptors, which can be shared between classes. It also prevents you from passing around descriptor objects as first-class citizens. For example, I might want to ask the ColorTrait for its constituents in order to construct a new one that has a different default value. I can't ask that of a factory function. This objection holds for any proposal that requires the descriptor object itself to hold onto its name, no matter how it acquires that name. Personally, I don't want descriptors that know their own name; I want descriptors that can be told what name to use for each operation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Mar 16, 2011 at 1:47 PM, Robert Kern <robert.kern@gmail.com> wrote:
As I was thinking of __addtoclass__ it would address this, though at class instantiation time instead of attribute access time. Specifically it would be like there was a fixup stage that would look like: def fixup(cls): for class_instance in cls.__mro__: for name, value in class_instance.__dict__.items(): method = getattr(value, '__addtoclass__', None) if method is not None: new_value = method(cls, name) if new_value is not value: setattr(cls, name, new_value) If ColorTrait returns a new instance when __addtoclass__ is called, then it can, and all of its instances will be class-specific (and not shared with subclasses). Ian
Robert Kern wrote:
Personally, I don't want descriptors that know their own name; I want descriptors that can be told what name to use for each operation.
This is a somewhat different use case from mine. For overridable_property, I *do* want the name to be known when the descriptor is created, so that it can precompute some things based on it. -- Greg
On Wed, Mar 16, 2011 at 2:28 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I appreciate the user story. I also appreciate what Ian said: that this is often solved using a metaclass (in fact it is now a very common pattern) but that everybody does it somewhat differently. And I appreciate that Greg's use case is not solved by a metaclass (even if we turned the pattern into a metaclass or class decorator in the stdlib, which might be a good idea regardless). At the same time, it seems that there aren't a lot of specific examples besides namedtuple (which seems to cause lots of emotions and is I think best left alone) and Greg's overridable_property. So, unless we can come up with a really nice way (either syntactical or perhaps through a magic builtin) to give functions like overridable_property() access to the LHS name, and find more use cases, I don't see this happening. I really don't like "assignment decorators" (which aren't the same thing at all as class or function decorators, no matter how much I squint) nor most other solutions (e.g. ":=" -- too subtle, and might well mean something else). But I'm not precluding that someone will come up with a better solution. In the mean time, as long as it's just one use case I still like spelling it using a function decorator: @overridable_property def content_size(): "Size of the content are" The overridable_property can then access the __name__ and __doc__ attributes of the function passed into it, assert that it has no arguments using the inspect module, and return an appropriate instance of the OverridableProperty class. Voilà, decorated assignment. :-) PS. Greg: is your current overridable_property part of some open source code that you've published yet? Searches for it mostly seem to turn up recent discussions here... -- --Guido van Rossum (python.org/~guido)
On 3/17/11 11:18 PM, Guido van Rossum wrote:
A sizable portion of Traits needs the name information. http://pypi.python.org/pypi/Traits As I explained elsewhere, we currently implement this by overriding __getattribute__ to implement a descriptor-like protocol that passes along the name to the trait object rather than storing it on the trait object itself, but we could re-engineer Traits make do with Ian's __addtoclass__ proposal. Being able to use plain descriptors would allow us to rewrite Traits to avoid our current C-implemented base class, which would let us interoperate with other frameworks better. I can list all of the specific features of Traits that makes use of this information if you like. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Guido van Rossum wrote:
PS. Greg: is your current overridable_property part of some open source code that you've published yet?
Yes, I'm using it in two projects at the moment: PyGUI: http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/ Albow: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/
Searches for it mostly seem to turn up recent discussions here...
You probably won't find it with a direct search because it's more of an internal detail of those libraries rather than a separately-advertised feature. -- Greg
On Wed, Mar 16, 2011 at 9:52 AM, Westley Martínez <anikom15@gmail.com> wrote:
If it went unnoticed, what's the matter?
Unnoticed just means it was a silent failure rather than a noisy one. It doesn't mean the bug wasn't resulting erroneous output. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Westley Martínez wrote:
If it went unnoticed, what's the matter?
It only went unnoticed because until recently I hadn't run any code that depended critically on it being right. A bug is still a bug even if you haven't found it yet! When it did became apparent that something was wrong, it still took quite a long time to track down the cause. I was looking for a bug in the get_content_size() method, whereas it was actually calling a different method altogether. Having attribute accesses go astray like that is not something you expect. -- Greg
On 03/16/2011 05:28 AM, Greg Ewing wrote:
As I suggested in my email on the Assignment Decorators thread this morning, you could achieve this in current Python, no extension needed: def assign(fn): return fn(fn.__name__) @assign def content_size(name): return overridable_property(name, "Size of the content area.") How bad do you want it? ;-) /larry/
On 16 March 2011 11:10, Larry Hastings <larry@hastings.org> wrote:
And building on this sightly you could do the following for namedtuple:
All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Mar 16, 2011, at 1:49 PM, Terry Reedy wrote:
If make_namedtuple were added to collections, then one could import *that* instead of 'namedtuple' itself.
Post a recipe somewhere. My bet is that it will have a near zero adoption rate. -1 on all of these random ideas to solve what is basically a non-problem. The name of a named tuple, or a class, or a function, or a variable typically is used many times, not just in the definition. If you make up a new factory function that uses inspect magic to guess user's intend target name, you will have introduced unnecessary complexity and will likely introduce unexpected behaviors and bugs. Raymond
On 3/16/2011 4:49 PM, Terry Reedy wrote:
I am not sure how serious I was when I wrote that, but after further thought, I retract it as a serious suggestion, if indeed it was ;-). I think both decorators are 'cute' as examples of what one can do with decorators, and possible cookbook recipes, but not something we really need in the stdlib. I mostly said something because they do not require new syntax and I do not like any of the new syntax proposals. There are many aspects of the rationale for decorators that do not apply to the assignment case. 1. Explicit function wrapping requires the name in tripicate rather than duplicate. Moreover, some interface wrappers for functions need fairly long, multi-component names. I am pretty sure such need is much less for named tuples and properties. 2. The triplicates are *not* on the same line but may be separated by an arbitrary number of lines. This not only impedes checking that all three are the same, but may also impede understanding of the function code. That sometimes really requires knowing how or what the function will be wrapped as. This is greatly aided by bringing the wrapping function back up to (above) the def line. For instance, @classmethod and @staticmethod explain the non-standard signature with 'cls' or '' instead of 'self'. For another example, twisted.internet.defer.inLineCallbacks is a decorator that wraps a generator function as a twisted.internet.defer.Deferred object. Knowing that a particular generator function implements a series of data/error callbacks tor twisted greatly helps in understanding it. The idea is 'twisted' enough as it is ;-). -- Terry Jan Reedy
Larry Hastings wrote:
Not quite badly enough to replace all my existing overridable_property declarations with something that ugly. :-) But I appreciate the suggestion, and I'll keep it in mind next time I'm designing anything similar. -- Greg
On 16 March 2011 19:18, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
You should be able to replace it with something like: @do_something def content_size(): return "Size of the content area." Where do_something calls the function to get the string argument and takes the name from the function.__name__ (and uses them to create and return the overridable_property). Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
I'll note this general problem is also present in any of the declarative ORMs, which use silly hacks to tell descriptors their name. Like you have: class Table(ORM): name = StringColumn() Another case where I've noticed a problem is any kind of descriptor that needs its own storage; the name of the property gives a possible stable namespace for the value, but without the name you either have to pass in a name or the storage area becomes volatile. For instance, a read-only descriptor: http://svn.colorstudy.com/home/ianb/recipes/setonce.py You can solve this in, e.g., the ORM class by doing things when a class is created -- but it requires very specific cooperation between the class and the descriptor. Everyone really does it different ways. One could imagine an extension of the descriptor protocol, where on class creation you called something like attr.__addtoclass__(cls, name) (for all attributes of the class that define that method) -- which if you just want the name you'd simply save that name in your object and return self. On Wed, Mar 16, 2011 at 4:28 AM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
On 3/16/11 12:13 PM, Ian Bicking wrote:
If we were to extend the descriptor protocol, I think it would be better to extend it by providing __get_ex__(self, instance, owner, name), __set_ex__(self, instance, name, value), etc. methods that would be called in preference over the current __get__() and __set__() methods. This allows descriptor objects to be reused. We do this reasonably often with the Traits package (which overrides __getattribute__() to essentially implement this __get_ex__()/__set_ex__() protocol by different names). For example, we have a complicated trait for specifying colors: ColorTrait = Trait("black", Tuple, List, Str, color_table) It would be nice to simply reuse this object everywhere: class Figure(HasTraits): background = ColorTrait foreground = ColorTrait The alternative is to only provide factories such that you always get a new descriptor: ColorTrait = lambda: Trait("black", Tuple, List, Str, color_table) class Figure(HasTraits): background = ColorTrait() foreground = ColorTrait() However, this makes these new descriptors inconsistent with current descriptors, which can be shared between classes. It also prevents you from passing around descriptor objects as first-class citizens. For example, I might want to ask the ColorTrait for its constituents in order to construct a new one that has a different default value. I can't ask that of a factory function. This objection holds for any proposal that requires the descriptor object itself to hold onto its name, no matter how it acquires that name. Personally, I don't want descriptors that know their own name; I want descriptors that can be told what name to use for each operation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Mar 16, 2011 at 1:47 PM, Robert Kern <robert.kern@gmail.com> wrote:
As I was thinking of __addtoclass__ it would address this, though at class instantiation time instead of attribute access time. Specifically it would be like there was a fixup stage that would look like: def fixup(cls): for class_instance in cls.__mro__: for name, value in class_instance.__dict__.items(): method = getattr(value, '__addtoclass__', None) if method is not None: new_value = method(cls, name) if new_value is not value: setattr(cls, name, new_value) If ColorTrait returns a new instance when __addtoclass__ is called, then it can, and all of its instances will be class-specific (and not shared with subclasses). Ian
Robert Kern wrote:
Personally, I don't want descriptors that know their own name; I want descriptors that can be told what name to use for each operation.
This is a somewhat different use case from mine. For overridable_property, I *do* want the name to be known when the descriptor is created, so that it can precompute some things based on it. -- Greg
On Wed, Mar 16, 2011 at 2:28 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I appreciate the user story. I also appreciate what Ian said: that this is often solved using a metaclass (in fact it is now a very common pattern) but that everybody does it somewhat differently. And I appreciate that Greg's use case is not solved by a metaclass (even if we turned the pattern into a metaclass or class decorator in the stdlib, which might be a good idea regardless). At the same time, it seems that there aren't a lot of specific examples besides namedtuple (which seems to cause lots of emotions and is I think best left alone) and Greg's overridable_property. So, unless we can come up with a really nice way (either syntactical or perhaps through a magic builtin) to give functions like overridable_property() access to the LHS name, and find more use cases, I don't see this happening. I really don't like "assignment decorators" (which aren't the same thing at all as class or function decorators, no matter how much I squint) nor most other solutions (e.g. ":=" -- too subtle, and might well mean something else). But I'm not precluding that someone will come up with a better solution. In the mean time, as long as it's just one use case I still like spelling it using a function decorator: @overridable_property def content_size(): "Size of the content are" The overridable_property can then access the __name__ and __doc__ attributes of the function passed into it, assert that it has no arguments using the inspect module, and return an appropriate instance of the OverridableProperty class. Voilà, decorated assignment. :-) PS. Greg: is your current overridable_property part of some open source code that you've published yet? Searches for it mostly seem to turn up recent discussions here... -- --Guido van Rossum (python.org/~guido)
On 3/17/11 11:18 PM, Guido van Rossum wrote:
A sizable portion of Traits needs the name information. http://pypi.python.org/pypi/Traits As I explained elsewhere, we currently implement this by overriding __getattribute__ to implement a descriptor-like protocol that passes along the name to the trait object rather than storing it on the trait object itself, but we could re-engineer Traits make do with Ian's __addtoclass__ proposal. Being able to use plain descriptors would allow us to rewrite Traits to avoid our current C-implemented base class, which would let us interoperate with other frameworks better. I can list all of the specific features of Traits that makes use of this information if you like. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Guido van Rossum wrote:
PS. Greg: is your current overridable_property part of some open source code that you've published yet?
Yes, I'm using it in two projects at the moment: PyGUI: http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/ Albow: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Albow/
Searches for it mostly seem to turn up recent discussions here...
You probably won't find it with a direct search because it's more of an internal detail of those libraries rather than a separately-advertised feature. -- Greg
participants (11)
-
Greg Ewing -
Guido van Rossum -
Ian Bicking -
Larry Hastings -
Masklinn -
Michael Foord -
Nick Coghlan -
Raymond Hettinger -
Robert Kern -
Terry Reedy -
Westley Martínez