Add a way to test for a descriptor in Python Code
Greetings, I am trying to emulate attribute lookup, and want to test if symbol found in __dict__ is an attribute or not in python code (i.e.: does its have a `tp_descr_get` slot?) Reading the documentation carefully, I am supposed to test if the attribute has a `.__get__` method; however, I see no way to test for this properly (easily). Currently the only way I know to test for this is (See code at end of this message): any('__get__' in m.__dict__ for m in type(v).__mro__) Which seems terribly inefficient. The documentation at: https://docs.python.org/2/howto/descriptor.html https://docs.python.org/3/howto/descriptor.html Both says: """ For classes, the machinery is in type.__getattribute__() which transforms B.x into B.__dict__['x'].__get__(None, B). In pure Python, it looks like: def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v """ However, the call to `hasattr(v, '__get__')` appears to me to be incorrect. The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* whether `v` has a symbol `__get__` in any of the classes in it's method resolution order. Looking at `type_getattro` in "Objects/typeobject.c" here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L3177 Reads: meta_get = Py_TYPE(meta_attribute)->tp_descr_get So I really want to know if the `tp_descr_get` slot is set or not. (Which is a different question than whether `v` has a `__get__` attribute). The code below shows that: 1. The valid value of `Point.y` is <Not_A_Descriptor> 2. The valid value is returned by `Point.y`, `type.__getattribute__(Point, y)`, and `fixed__Type__getattribute` 3. The invalid value of `Point.y` is `2` as returned by the [emulated] `__getattribute__` documented https://docs.python.org/3/howto/descriptor.html So I am requesting: 1. An efficient python way to test for `tp_descr_get` slot (and other descriptor) slots. 2. Fix the documentation referenced above. Thanks, Joy Diamond. NOTE #1: This email describes the very subtle difference between whether an instance has a `__get__` symbol or not, which I believe is *NOT* the same question as whether `hasattr(instance, "__get__")` returns true or not. The first question is does it have the symbol `__get_` [Which python put in the `tp_descr_slot`] while `hasattr` answers the question does it have the `__get__` attribute. NOTE #2: Also using `hasattr(type(v), "__get__")` would not answer the question I want, because then it might find a `__get__` method in the meta-class of `type(v)` which again would return an incorrect answer. Example program that shows that using `hasattr(v, '__get__')` is not a valid way to test if something is a descriptor (works in python 2, python 3, and pypy): def get_1(self, a, b): return 1 def get_2(a, b): return 2 class Descriptor(object): __get__ = get_1 class Not_A_Descriptor(object): def __init__(self): self.__get__ = get_2 def __repr__(self): return '<Not_A_Descriptor instance>' class Point(object): __slots__ = (()) x = Descriptor() y = Not_A_Descriptor() # # Copied from https://docs.python.org/3/howto/descriptor.html # def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v # # My fixed version # def fixed__Type__getattribute(self, key): "FIXED: Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if any('__get__' in m.__dict__ for m in type(v).__mro__): return v.__get__(None, self) return v print('Point.x: %s' % Point.x) print('Point.y: %s' % Point.y) print("type.__getattribute__(Point, 'x'): %s" % type.__getattribute__(Point, 'x')) print("type.__getattribute__(Point, 'y'): %s" % type.__getattribute__(Point, 'y')) print("__getattribute__(Point, 'x'): %s" % __getattribute__(Point, 'x')) print("__getattribute__(Point, 'y'): %s ***WRONG***" % __getattribute__(Point, 'y')) print("fixed__Type__getattribute(Point, 'x'): %s" % fixed__Type__getattribute(Point, 'x')) print("fixed__Type__getattribute(Point, 'y'): %s ***CORRECT***" % fixed__Type__getattribute(Point, 'y'))
On Tue, Oct 30, 2018 at 04:40:40AM -0400, Joy Diamond wrote:
""" For classes, the machinery is in type.__getattribute__() which transforms B.x into B.__dict__['x'].__get__(None, B). In pure Python, it looks like:
def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v
""" [...] However, the call to `hasattr(v, '__get__')` appears to me to be incorrect.
I agree, but only because it fails to take into account that dunder methods like __get__ are only looked up on the class, not the instance. I believe a more accurate eumulation would be: if hasattr(type(v), '__get__'): return type(v).__get__(None, self) Actually, on further investigation, I think it ought to be: if inspect.hasattr_static(type(v), '__get__') except that there is no hasattr_static, there's only a getattr_static. So perhaps there ought to be a hasattr_static as well.
The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* whether `v` has a symbol `__get__` in any of the classes in it's method resolution order.
What's the difference as you see it? -- Steve
Clarifications: 1. I miswrote part of my first post where I wrote "I want to test if symbol found in __dict__ is an attribute or not in python code". I meant to write "is a DESCRIPTOR" or not. 2. The example in https://docs.python.org/3/howto/descriptor.html for reproducing `type.__getattribute__` has a second bug, in that it does not look at the class inheritance properly. (See fixed example below named `type_getattro` and based on the same function in the C code). In the example below if you call ` __getattribute__(Child, 'x')` it will incorrectly fail with "Catch `AttributeError: 'type' object has no attribute 'x'`" Responding to Steve: On Tue, Oct 30, 2018 at 6:31 AM Steven D'Aprano <steve@pearwood.info> wrote:
Actually, on further investigation, I think it ought to be:
if inspect.hasattr_static(type(v), '__get__')
except that there is no hasattr_static, there's only a getattr_static. So perhaps there ought to be a hasattr_static as well.
`inspect.hasattr_static` gets me half way there (but it still looks in two chains of inheritance, where I only want to look in one). In particular if the metaclass of `type(v)` has a `.__get__` method it will incorrectly find that. So it will still misidentify if an instance is a descriptor or not.
The question is *NOT* whether 'v' has an attribute '__get__'; *BUT* whether `v` has a symbol `__get__` in any of the classes in it's method resolution order.
What's the difference as you see it?
I want to be able to look in the method resolution order (one inheritance chain). `getattr` and `inspect.getattr_static` both look in two inheritance chains (the instance & it's type; or the case of a class, the class and it's metaclass). I need to look in only one chain (and disable descriptors like `inspect.getattr_static` does). To put it succinctly: I am trying to reproduce the behavior of `_PyType_Lookup` from "Objects/typeobject.c" (see example below). Below is a full reproduction of `object.__getattribute__` and `type.__getattribute__` based on reading the Python source code. Note this reproduction of `type.__getattribute__` is much more accurate than what is at: https://docs.python.org/2/howto/descriptor.html https://docs.python.org/3/howto/descriptor.html Both of which need to be updated. (This is not yet filed as a bug report; as first I am requesting a call to something like `_PyType_Lookup` that is efficent; and once we agree on that, we can created an updated reproduction of `type.__getattribute__`). Thanks, Joy Diamond. # # The following reproduces (and tests) `object.__getattribute__` and `type.__getattribute__` based on reading the C source code. # absent = object() def _PyType_Lookup(model, name): '''Based on `_PyType_Lookup` in "Objects/typeobject.c"''' mro = model.__mro__ if mro is None: return absent for m in mro: symbol_table = m.__dict__ if name in symbol_table: return symbol_table[name] return absent def lookup__tp_descr_get(model): tp_descr_get = _PyType_Lookup(model, '__get__') return tp_descr_get def has__tp_descr_set(model): tp_descr_set = _PyType_Lookup(model, '__set__') return tp_descr_set is not absent # # Reproduction of `object.__getattribute__` # def PyObject_GenericGetAttr(instance, name): '''Based on `PyObject_GenericGetAttr` in "Objects/object.c"''' instance_type = type(instance) descriptor = _PyType_Lookup(instance_type, name) if descriptor is absent: get = absent else: descriptor_type = type(descriptor) get = lookup__tp_descr_get(descriptor_type) if (get is not absent) and (has__tp_descr_set(descriptor_type)): # # "Data Descriptor" (a `__set__` method exists) has precedence. # return get(descriptor, instance, instance_type) if instance_type.__dictoffset__: instance__mapping = instance.__dict__ if name in instance__mapping: return instance__mapping[name] if get is not absent: return get(descriptor, instance, instance_type) raise AttributeError("cannot find attribute `{}` in instance of `{}`".format(name, instance_type.__name__)) # # Reproduction of `type.__getattribute__` # def type_getattro(model, name): '''Based on `type_getattro` in "Objects/type_object.c"''' metatype = type(model) descriptor = _PyType_Lookup(metatype, name) if descriptor is absent: get = absent else: descriptor_type = type(descriptor) get = lookup__tp_descr_get(descriptor_type) if (get is not absent) and (has__tp_descr_set(descriptor_type)): # # "Data Descriptor" (a `__set__` method exists) has precedence. # return get(descriptor, instance, instance_type) symbol = _PyType_Lookup(model, name) if symbol is not absent: # # Implement descriptor functionality, if any # symbol_type = type(symbol) symbol_get = lookup__tp_descr_get(symbol_type) if symbol_get is not absent: # # None 2nd argument indicates the descriptor was # found on the target object itself (or a base) # return symbol_get(symbol, None, model) return symbol if get is not absent: return get(descriptor, instance, instance_type) raise AttributeError("cannot find attribute `{}` in class `{}`".format(name, instance_type.__name__)) # # object Example # print('=== object example ===') class Readonly_Descriptor(object): __slots__ = (( 'value', )) def __init__(self, value): self.value = value def __get__(self, object, object_type): return self.value class Data_Descriptor(object): __slots__ = (( 'value', )) def __init__(self, value): self.value = value def __get__(self, object, object_type): return self.value def __set__(self, object, value): self.value = value class Point(object): __slots__ = (( '__dict__', '_y', )) def __init__(self, x, y): self.x = x self.y = y self._y = 1 p23 = Point(2, 3) Point.x = Readonly_Descriptor(4) Point.y = Data_Descriptor(5) Point.z = Readonly_Descriptor(6) p23.y = 7 # Uses the Data_Descriptor print("p23.x: %d # p23.__dict__['x']; *IGNORES* ReadOnly_Descriptor" % p23.x) print("p23.y: %d # type(p23).__dict__['y'].__get__(p23, Point)" % p23.y) print("p23.z: %d # type(p23).__dict__['z'].__get__(p23, Point)" % p23.z) print('') print("PyObject_GenericGetAttr(p23, 'x'): %d" % PyObject_GenericGetAttr(p23, 'x')) print("PyObject_GenericGetAttr(p23, 'y'): %d" % PyObject_GenericGetAttr(p23, 'y')) print("PyObject_GenericGetAttr(p23, 'z'): %d" % PyObject_GenericGetAttr(p23, 'z')) # # Type Example # print('') print('=== Type example ===') def get_1(self, a, b): return 1 def get_2(a, b): return 2 class Descriptor(object): __get__ = get_1 class Not_A_Descriptor(object): def __init__(self): self.__get__ = get_2 def __repr__(self): return '<Not_A_Descriptor instance>' class Parent(object): __slots__ = (()) x = Descriptor() y = Not_A_Descriptor() class Child(Parent): __slots__ = (()) # # Copied from https://docs.python.org/3/howto/descriptor.html # def __getattribute__(self, key): "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(None, self) return v print('Child.x: %s' % Child.x) print('Child.y: %s' % Child.y) print("type.__getattribute__(Child, 'x'): %s" % type.__getattribute__(Child, 'x')) print("type.__getattribute__(Child, 'y'): %s" % type.__getattribute__(Child, 'y')) try: print("__getattribute__(Child, 'x'): %s" % __getattribute__(Child, 'x')) print("__getattribute__(Child, 'y'): %s ***WRONG***" % __getattribute__(Child, 'y')) except AttributeError as e: # # Catch `AttributeError: 'type' object has no attribute 'x'` # pass print("type_getattro(Child, 'x'): %s" % type_getattro(Child, 'x')) print("type_getattro(Child, 'y'): %s ***CORRECT***" % type_getattro(Child, 'y'))
participants (2)
-
Joy Diamond -
Steven D'Aprano