Weak-referencing/weak-proxying of (bound) methods
Hello, Today, I encountered a surprising bug in my code which creates some weakref.proxies to instance methods... The actual Python behaviour related to the issue can be ilustrated with the following example: >>> import weakref >>> class A: ... def method(self): print(self) ... >>> A.method <function method at 0xb732926c> >>> a = A() >>> a.method <bound method A.method of <__main__.A object at 0xb7326bec>> >>> r = weakref.ref(a.method) # creating a weak reference >>> r # ...but it appears to be dead <weakref at 0xb7327d9c; dead> >>> w = weakref.proxy(a.method) # the same with a weak proxy >>> w <weakproxy at 0xb7327d74 to NoneType at 0x829f7d0> >>> w() Traceback (most recent call last): File "<stdin>", line 1, in <module> ReferenceError: weakly-referenced object no longer exists This behaviour is perfectly correct -- but still surprising, especially for people who know little about method creation machinery, descriptors etc. I think it would be nice to make this 'trap' less painful -- for example, by doing one or both of the following: 1. Describe and explain this behaviour in the weakref module documentation. 2. Provide (in functools?) a type-and-decorator that do the same what func_descr_get() does (transforms a function into a method) *plus* caches the created method (e.g. at the instance object). A prototype implementation: class InstanceCachedMethod(object): def __init__(self, func): self.func = func (self.instance_attr_name ) = '__{0}_method_ref'.format(func.__name__) def __get__(self, instance, owner): if instance is None: return self.func try: return getattr(instance, self.instance_attr_name) except AttributeError: method = types.MethodType(self.func, instance) setattr(instance, self.instance_attr_name, method) return method A simplified version that reuses the func.__name__ (works well as long as func.__name__ is the actual instance attribute name...): class InstanceCachedMethod(object): def __init__(self, func): self.func = func def __get__(self, instance, owner): if instance is None: return self.func method = types.MethodType(self.func, instance) setattr(instance, self.func.__name__, method) return method Both versions work well with weakref.proxy()/ref() objects: >>> class B: ... @InstanceCachedMethod ... def method(self): print(self) ... >>> B.method <function method at 0xb7329d6c> >>> b = B() >>> b.method <bound method B.method of <__main__.B object at 0xb7206ccc>> >>> r = weakref.ref(b.method) >>> r <weakref at 0xb72c611c; to 'method' at 0xb736c40c (method)> >>> w = weakref.proxy(b.method) >>> w <weakproxy at 0xb7327e14 to method at 0xb736c40c> >>> w() <__main__.B object at 0xb7206ccc> What do you think about it? Cheers. *j
On Mon, Jun 11, 2012 at 2:16 AM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
I was bitten by this issue a while ago as well. It made working with weakref proxies much more involved than I expected it would be. Wouldn't it be better to approach the issue from the opposite end, and improve/wrap/replace weakref.proxy with something that can handle bound methods? - Tal
Tal Einat dixit (2012-06-15, 15:41):
Indeed, probably could it be done by wrapping weakref.ref()/proxy() with something like the following: # here `obj` is the object that is being weak-referenced... if isinstance(obj, types.MethodType): try: cache = obj.__self__.__method_cache__ except AttributeError: cache = obj.__self__.__method_cache__ = WeakKeyDictionary() method_cache.setdefault(obj.__func__, set()).add(obj) (Using WeakKeyDictionary with corresponding function objects as weak keys -- to provide automagic cleanup when a function is deleted, e.g. replaced with another one. In other words: the actual weak ref/proxy to a method lives as long as the corresponding function does). Any thoughts? Cheers. *j
Jan Kaliszewski dixit (2012-06-16, 01:41):
On second thought -- no, it shouldn't be done on the side of weakref.ref()/proxy(). Why? My last idea described just above has such a bug: each time you create a new weak reference to the method another method object is cached (added to __method_cache__[func] set). You could think that caching only one object (just in __method_cache__[func]) would be a better idea, but it wouldn't: such a behaviour would be strange and unstable: after creating a new weakref to the method, the old weakref would became invalid... And yes, we can prevent it by ensuring that each time you take the method from a class instance you get the same object (per class instance) -- but then we come back to my previous idea of a descriptor-decorator. And IMHO such a decorator should not be applied on the class dictionary implicitly by weakref.ref()/proxy() but explicitly in the class body with the decorator syntax (applying such a decorater, i.e. replacing a function with a caching descriptor is a class dict, is too invasive operation to be done silently). So I renew (and update) my previous descriptor-decorator that could be added to functools (or to weakref as a helper?) and applied explicitly by programmers, when needed: class CachedMethod(object): def __init__(self, func): self.func = func def __get__(self, instance, owner): if instance is None: return self.func try: cache = instance.__method_cache__ except AttributeError: # not thread-safe :-( cache = instance.__method_cache__ = WeakKeyDictionary() return cache.setdefault( self.func, types.MethodType(self.func, instance)) Usage: class MyClass(object): @CachedMethod def my_method(self): ... instance = MyClass() method_weak_proxy = weakref.proxy(instance.my_method) method_weak_proxy() # works! It should be noted that caching a reference to a method in an instance causes circular referencing (class <-> instance). However, ofter it is not a problem and can help avoiding circular references involving other objects which we want to have circular-ref-free (typical use case: passing a bound method as a callback). Cheers. *j
On Mon, Jun 11, 2012 at 2:16 AM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
I was bitten by this issue a while ago as well. It made working with weakref proxies much more involved than I expected it would be. Wouldn't it be better to approach the issue from the opposite end, and improve/wrap/replace weakref.proxy with something that can handle bound methods? - Tal
Tal Einat dixit (2012-06-15, 15:41):
Indeed, probably could it be done by wrapping weakref.ref()/proxy() with something like the following: # here `obj` is the object that is being weak-referenced... if isinstance(obj, types.MethodType): try: cache = obj.__self__.__method_cache__ except AttributeError: cache = obj.__self__.__method_cache__ = WeakKeyDictionary() method_cache.setdefault(obj.__func__, set()).add(obj) (Using WeakKeyDictionary with corresponding function objects as weak keys -- to provide automagic cleanup when a function is deleted, e.g. replaced with another one. In other words: the actual weak ref/proxy to a method lives as long as the corresponding function does). Any thoughts? Cheers. *j
Jan Kaliszewski dixit (2012-06-16, 01:41):
On second thought -- no, it shouldn't be done on the side of weakref.ref()/proxy(). Why? My last idea described just above has such a bug: each time you create a new weak reference to the method another method object is cached (added to __method_cache__[func] set). You could think that caching only one object (just in __method_cache__[func]) would be a better idea, but it wouldn't: such a behaviour would be strange and unstable: after creating a new weakref to the method, the old weakref would became invalid... And yes, we can prevent it by ensuring that each time you take the method from a class instance you get the same object (per class instance) -- but then we come back to my previous idea of a descriptor-decorator. And IMHO such a decorator should not be applied on the class dictionary implicitly by weakref.ref()/proxy() but explicitly in the class body with the decorator syntax (applying such a decorater, i.e. replacing a function with a caching descriptor is a class dict, is too invasive operation to be done silently). So I renew (and update) my previous descriptor-decorator that could be added to functools (or to weakref as a helper?) and applied explicitly by programmers, when needed: class CachedMethod(object): def __init__(self, func): self.func = func def __get__(self, instance, owner): if instance is None: return self.func try: cache = instance.__method_cache__ except AttributeError: # not thread-safe :-( cache = instance.__method_cache__ = WeakKeyDictionary() return cache.setdefault( self.func, types.MethodType(self.func, instance)) Usage: class MyClass(object): @CachedMethod def my_method(self): ... instance = MyClass() method_weak_proxy = weakref.proxy(instance.my_method) method_weak_proxy() # works! It should be noted that caching a reference to a method in an instance causes circular referencing (class <-> instance). However, ofter it is not a problem and can help avoiding circular references involving other objects which we want to have circular-ref-free (typical use case: passing a bound method as a callback). Cheers. *j
participants (3)
-
Jan Kaliszewski -
shibturn -
Tal Einat