descriptors outside of classes

Here's another outlandish idea. How about if descriptors could be used outside of classes. I.e. any global or local variable could be assigned a descriptor object and the descriptor protocol would be respected for that variable. This would be a pretty messy change, and I have no illusions that the idea will go anywhere. However, would there be room for this in python? The advantage is that it would allow for greater flexibility in hacking things up. The downside is that it hides what's really going on. Descriptors on classes are already less-than-obvious if you aren't familiar with how variables are handled on objects. I would expect that descriptors in the global namespace would be even more-so. -eric

On Wed, Mar 30, 2011 at 3:59 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Not really, because globals() both promises to return a normal dictionary and to respect changes to the module globals made via that dictionary. All bets are off with locals(), but the globals() aspect already spikes the idea, as it does many other ideas to do with speeding or otherwise enhancing namespace lookups. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 30 March 2011 17:07, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Yeah, I figured as much. I am sure there is some convoluted way to make it work, but it would not nearly be worth it for what we would get out of it.
Well, it's still a *nice idea* even if it's impractical. For example to allow for deprecation warnings on module variables Twisted creates a subclass of modules (I believe), so that accessing the variable raises the appropriate warning. In the standard library we are unable to apply deprecation warnings to module variables because we don't have a mechanism like this. All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Eric Snow wrote:
I was just thinking along those same lines. Sounds like twisted already does it. Does it amount to using a custom __import__?
I don't know what Twisted does, but I was thinking of an attribute called __moduleclass__ that works a bit like the old __metaclass__ attribute. Then you could do class __moduleclass__: ... descriptor definitions go here ... -- Greg

I suppose that is more specific than the __import__ builtin. Classes have __build_class__. Functions don't have an equivalent in the global builtins. For imports you have to go through __import__. So a __module_class__ would dictate which class for import to use. By default it would be types.ModuleType. Makes sense. -eric p.s. a __build_function__ would be meaningful addition particularly if the def-from syntax were feasible. I'm just saying... :) On Wed, Mar 30, 2011 at 3:13 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:

Eric Snow wrote:
There's one tricky point, though -- you need a module object before you can execute the module's code, and it's the module's code that creates the __moduleclass__ entry. What should probably happen is that a standard module object gets created initially, and then after executing the module body, replace the module's __class__ with the __moduleclass__, if any. You can't currently do this in Python code, because it won't let you change the __class__ of a builtin module object. So either that restriction would have to be lifted, or the machinery implementing this would have to be written in C. An alternative (which is what PyGUI currently does) is to create a new module object of the specified class, copy the __dict__ of the original module into it, and then replace the entry in sys.modules. This would be second-best, though, because it would mean that if the module imported itself, it would end up with the old module object instead of the new one. The same thing would also happen to any other modules that imported the first module while it was still loading. -- Greg

On 30 March 2011 22:13, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I'm pretty sure it creates a module subclass that forwards all attribute access to the real module and inserts itself into sys.modules in place of the "real" module. Pretty evil really. :-) I may be mistaken about this, it is based off my memory of a previous discussion. Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Thu, Mar 31, 2011 at 8:04 AM, Michael Foord <fuzzyman@gmail.com> wrote:
That's certainly the trick people use to implement lazy import handlers in the absence of a proper implementation of post-import hooks (ala PEP 369). (I'll point out that the object inserted into sys.modules doesn't need to be, and often isn't, an instance of the module type) The module code itself doesn't even need to know that the intervening class exists. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mar 29, 2011, at 10:59 AM, Eric Snow wrote:
Here's another outlandish idea. How about if descriptors could be used outside of classes. I.e. any global or local variable could be assigned a descriptor object and the descriptor protocol would be respected for that variable. This would be a pretty messy change, and I have no illusions that the idea will go anywhere. However, would there be room for this in python?
FWIW, you can already do this with locals (am not saying you should do it, am just saying that you can do it). Remember, the essential mechanism for descriptors is in the lookup function, not in the descriptor itself. For example, property() objects are descriptors only because they define one of the descriptor protocol methods (__get__, et al). Whether it gets invoked solely depends on how you look it up. If you use regular dictionary lookup, a.__class__.__dict__['x'], then the property object is retrieved but no special action occurs. If you use dotted lookup, a.x, then the property's __get__ method is called. This is because the lookup function, object.__getattribute__(), has code to detect and invoke descriptors. A ultra-simplified version of the lookup functions's psuedo-code looks like this: value = kls.__dict__[key] if hasattr(value, '__get__'): return call_the_getter(kls ,key) else: return the value Knowing this, it is possible to emulate that behavior with a dictionary whose lookup function, __getitem__(), can detect and invoke some sort of descriptor protocol. Since eval/exec can use arbitrary mappings for locals, you can use your custom dictionary while executing arbitrary python code. Essentially, you're executing python code in an environment where the lookup function for locals has been trained to handle your custom descriptor protocol. Raymond ----- simple example ----- class MyDict: def __init__(self, mapping): self.mapping = mapping def __getitem__(self, key): value = self.mapping[key] if hasattr(value, '__get__'): print('Invoking descriptor on', key) return value.__get__(key) print('Getting', key) return value def __setitem__(self, key, value): self.mapping[key] = value class Property: def __init__(self, getter): self.getter = getter def __get__(self, key): return self.getter(key) if __name__ == '__main__': md = MyDict({}) md['x'] = 10 md['_y'] = 20 md['y'] = Property(lambda key: md['_'+key]) print(eval('x+y+1', {}, md))

The same could be applied to the globals if module subclassing were practical. Then you could just use descriptors on that subclass. I expect that custom import functionality could provide this right now. Naturally, this would affect that promise Nick was talking about regarding globals, which could be confusing. But only in the same way that descriptors can be for classes already. Even if messing with the module class's __dict__ were legal, adding decorators there would probably not be effective since all modules would get those attributes. However, with module subclasses that would be more practical. Of course, the application of all this would be to let a module control what happens when another module tries to use the first module's namespace. But that is what descriptors are all about. -eric On Wed, Mar 30, 2011 at 2:26 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

Eric Snow wrote:
BTW, if anyone's wondering about use cases for this, I have one in PyGUI where the top-level module auto-imports names from submodules the first time you refer to them. This avoids the overhead of loading modules that an application doesn't use, without requiring the user to memorise which names come from which submodules. It also gives me the flexibility to move things around between submodules if I want. This is currently done using a custom module subclass with a __getattr__ method. Although I'm thinking about using a different strategy, because the current one confuses the heck out of py2app and py2exe. :-( -- Greg

On Wed, Mar 30, 2011 at 2:40 PM, Don Spaulding <donspauldingii@gmail.com> wrote:
I just signed up for python-ideas a month or two ago, how many awesome(ly dangerous) hacks like this have I missed over the years?
We need a python-hacks list. :) -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
We need a python-hacks list. :)
Very tangentially related, here's my latest piece of dubious hackery, written the other day to work around the fact that multiple inheritance seems to be broken in conjunction with gobject introspection. It plays fast and loose with the method resolution order, but it got me out of a tight corner. def mix_in(*src_classes): # Workaround for do_xxx method overrides not working properly # with multiple inheritance. # # Usage: # # class MyClass(Gtk.SomeBaseClass): # mix_in(Class1, Class2, ...) # import sys frame = sys._getframe(1) dst_dict = frame.f_locals for src_class in src_classes: for name, value in src_class.__dict__.iteritems(): if name not in dst_dict: dst_dict[name] = value -- Greg

On Wed, Mar 30, 2011 at 3:59 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Not really, because globals() both promises to return a normal dictionary and to respect changes to the module globals made via that dictionary. All bets are off with locals(), but the globals() aspect already spikes the idea, as it does many other ideas to do with speeding or otherwise enhancing namespace lookups. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 30 March 2011 17:07, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Yeah, I figured as much. I am sure there is some convoluted way to make it work, but it would not nearly be worth it for what we would get out of it.
Well, it's still a *nice idea* even if it's impractical. For example to allow for deprecation warnings on module variables Twisted creates a subclass of modules (I believe), so that accessing the variable raises the appropriate warning. In the standard library we are unable to apply deprecation warnings to module variables because we don't have a mechanism like this. All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Eric Snow wrote:
I was just thinking along those same lines. Sounds like twisted already does it. Does it amount to using a custom __import__?
I don't know what Twisted does, but I was thinking of an attribute called __moduleclass__ that works a bit like the old __metaclass__ attribute. Then you could do class __moduleclass__: ... descriptor definitions go here ... -- Greg

I suppose that is more specific than the __import__ builtin. Classes have __build_class__. Functions don't have an equivalent in the global builtins. For imports you have to go through __import__. So a __module_class__ would dictate which class for import to use. By default it would be types.ModuleType. Makes sense. -eric p.s. a __build_function__ would be meaningful addition particularly if the def-from syntax were feasible. I'm just saying... :) On Wed, Mar 30, 2011 at 3:13 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:

Eric Snow wrote:
There's one tricky point, though -- you need a module object before you can execute the module's code, and it's the module's code that creates the __moduleclass__ entry. What should probably happen is that a standard module object gets created initially, and then after executing the module body, replace the module's __class__ with the __moduleclass__, if any. You can't currently do this in Python code, because it won't let you change the __class__ of a builtin module object. So either that restriction would have to be lifted, or the machinery implementing this would have to be written in C. An alternative (which is what PyGUI currently does) is to create a new module object of the specified class, copy the __dict__ of the original module into it, and then replace the entry in sys.modules. This would be second-best, though, because it would mean that if the module imported itself, it would end up with the old module object instead of the new one. The same thing would also happen to any other modules that imported the first module while it was still loading. -- Greg

On 30 March 2011 22:13, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I'm pretty sure it creates a module subclass that forwards all attribute access to the real module and inserts itself into sys.modules in place of the "real" module. Pretty evil really. :-) I may be mistaken about this, it is based off my memory of a previous discussion. Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On Thu, Mar 31, 2011 at 8:04 AM, Michael Foord <fuzzyman@gmail.com> wrote:
That's certainly the trick people use to implement lazy import handlers in the absence of a proper implementation of post-import hooks (ala PEP 369). (I'll point out that the object inserted into sys.modules doesn't need to be, and often isn't, an instance of the module type) The module code itself doesn't even need to know that the intervening class exists. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mar 29, 2011, at 10:59 AM, Eric Snow wrote:
Here's another outlandish idea. How about if descriptors could be used outside of classes. I.e. any global or local variable could be assigned a descriptor object and the descriptor protocol would be respected for that variable. This would be a pretty messy change, and I have no illusions that the idea will go anywhere. However, would there be room for this in python?
FWIW, you can already do this with locals (am not saying you should do it, am just saying that you can do it). Remember, the essential mechanism for descriptors is in the lookup function, not in the descriptor itself. For example, property() objects are descriptors only because they define one of the descriptor protocol methods (__get__, et al). Whether it gets invoked solely depends on how you look it up. If you use regular dictionary lookup, a.__class__.__dict__['x'], then the property object is retrieved but no special action occurs. If you use dotted lookup, a.x, then the property's __get__ method is called. This is because the lookup function, object.__getattribute__(), has code to detect and invoke descriptors. A ultra-simplified version of the lookup functions's psuedo-code looks like this: value = kls.__dict__[key] if hasattr(value, '__get__'): return call_the_getter(kls ,key) else: return the value Knowing this, it is possible to emulate that behavior with a dictionary whose lookup function, __getitem__(), can detect and invoke some sort of descriptor protocol. Since eval/exec can use arbitrary mappings for locals, you can use your custom dictionary while executing arbitrary python code. Essentially, you're executing python code in an environment where the lookup function for locals has been trained to handle your custom descriptor protocol. Raymond ----- simple example ----- class MyDict: def __init__(self, mapping): self.mapping = mapping def __getitem__(self, key): value = self.mapping[key] if hasattr(value, '__get__'): print('Invoking descriptor on', key) return value.__get__(key) print('Getting', key) return value def __setitem__(self, key, value): self.mapping[key] = value class Property: def __init__(self, getter): self.getter = getter def __get__(self, key): return self.getter(key) if __name__ == '__main__': md = MyDict({}) md['x'] = 10 md['_y'] = 20 md['y'] = Property(lambda key: md['_'+key]) print(eval('x+y+1', {}, md))

The same could be applied to the globals if module subclassing were practical. Then you could just use descriptors on that subclass. I expect that custom import functionality could provide this right now. Naturally, this would affect that promise Nick was talking about regarding globals, which could be confusing. But only in the same way that descriptors can be for classes already. Even if messing with the module class's __dict__ were legal, adding decorators there would probably not be effective since all modules would get those attributes. However, with module subclasses that would be more practical. Of course, the application of all this would be to let a module control what happens when another module tries to use the first module's namespace. But that is what descriptors are all about. -eric On Wed, Mar 30, 2011 at 2:26 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

Eric Snow wrote:
BTW, if anyone's wondering about use cases for this, I have one in PyGUI where the top-level module auto-imports names from submodules the first time you refer to them. This avoids the overhead of loading modules that an application doesn't use, without requiring the user to memorise which names come from which submodules. It also gives me the flexibility to move things around between submodules if I want. This is currently done using a custom module subclass with a __getattr__ method. Although I'm thinking about using a different strategy, because the current one confuses the heck out of py2app and py2exe. :-( -- Greg

On Wed, Mar 30, 2011 at 2:40 PM, Don Spaulding <donspauldingii@gmail.com> wrote:
I just signed up for python-ideas a month or two ago, how many awesome(ly dangerous) hacks like this have I missed over the years?
We need a python-hacks list. :) -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
We need a python-hacks list. :)
Very tangentially related, here's my latest piece of dubious hackery, written the other day to work around the fact that multiple inheritance seems to be broken in conjunction with gobject introspection. It plays fast and loose with the method resolution order, but it got me out of a tight corner. def mix_in(*src_classes): # Workaround for do_xxx method overrides not working properly # with multiple inheritance. # # Usage: # # class MyClass(Gtk.SomeBaseClass): # mix_in(Class1, Class2, ...) # import sys frame = sys._getframe(1) dst_dict = frame.f_locals for src_class in src_classes: for name, value in src_class.__dict__.iteritems(): if name not in dst_dict: dst_dict[name] = value -- Greg
participants (7)
-
Don Spaulding
-
Eric Snow
-
Greg Ewing
-
Guido van Rossum
-
Michael Foord
-
Nick Coghlan
-
Raymond Hettinger