Issue 643841: Including a new-style proxy base class in 2.6/3.0
One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. With classic classes, this is trivial, since __getattr__ is always consulted, even for retrieval of special methods. With new-style classes, however, the __getattribute__ machinery can be bypassed, meaning the only way to proxy an arbitrary instance is to define all of the special methods that have C-level slots. This issue was actually first raised five and a half years ago [1], but has never been a particularly pressing problem, as anyone with any sense that needed to write a proxy object just used a classic class instead of a new-style one. In 3.0, with the demise of classic classes, that workaround goes away. So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance? For some proxy class implementers, it would be possible to use this class as a base-class to get the special method delegation 'for free', and for others with more esoteric needs, it would at least provide a reference for which special methods needed to be provided explicitly by the proxy type. I attached a sample implementation to [1] which is essentially equivalent to weakref.proxy, but with a strong reference to the target, and written in Python rather than C. I expect the target audience for such a feature to be quite small, but without better support for it in the standard library, I also suspect it could prove to be a show-stopper for the affected developers as far as Py3k migration is concerned. Cheers, Nick. [1] http://bugs.python.org/issue643841 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. I've needed to do this a few times when wrapping libraries.
Michael Foord
With classic classes, this is trivial, since __getattr__ is always consulted, even for retrieval of special methods.
With new-style classes, however, the __getattribute__ machinery can be bypassed, meaning the only way to proxy an arbitrary instance is to define all of the special methods that have C-level slots.
This issue was actually first raised five and a half years ago [1], but has never been a particularly pressing problem, as anyone with any sense that needed to write a proxy object just used a classic class instead of a new-style one. In 3.0, with the demise of classic classes, that workaround goes away.
So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance? For some proxy class implementers, it would be possible to use this class as a base-class to get the special method delegation 'for free', and for others with more esoteric needs, it would at least provide a reference for which special methods needed to be provided explicitly by the proxy type.
I attached a sample implementation to [1] which is essentially equivalent to weakref.proxy, but with a strong reference to the target, and written in Python rather than C.
I expect the target audience for such a feature to be quite small, but without better support for it in the standard library, I also suspect it could prove to be a show-stopper for the affected developers as far as Py3k migration is concerned.
Cheers, Nick.
Nick Coghlan wrote:
So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance?
Sounds good to me. -- Greg
Nick Coghlan wrote:
So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance?
On May 20, 2008, at 7:55 PM, Greg Ewing wrote:
Sounds good to me.
Same here. There's an implementation in zope.proxy which might be useful as a starting point: http://pypi.python.org/pypi/zope.proxy/3.4.0 I'd be willing to write the required documentation, since I'm partly to blame for the implementation. The short description of the package on PyPI contains a typo; that's been fixed in Subversion. :-) -Fred -- Fred Drake <fdrake at acm.org>
Fred Drake wrote:
Nick Coghlan wrote:
So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance?
On May 20, 2008, at 7:55 PM, Greg Ewing wrote:
Sounds good to me.
Same here. There's an implementation in zope.proxy which might be useful as a starting point:
While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name). One other issue is where to put it - none of the existing stdlib modules seemed appropriate, so my sample implementation in the tracker issue just invents a new 'typetools' module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
On May 21, 2008, at 5:41 AM, Nick Coghlan wrote:
While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name).
The idea that it would serve as an example seems odd; the reason to make things part of the standard library is because their implementation needs to track the Python core closely. For a proxy, it would be the need to reflect additional slot methods as they're added. A Python implementation may be able to do this just fine, but the performance penalty would make it uninteresting for many large applications. For what it's worth, zope.proxy does support subclassing. -Fred -- Fred Drake <fdrake at acm.org>
Fred Drake wrote:
On May 21, 2008, at 5:41 AM, Nick Coghlan wrote:
While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name).
The idea that it would serve as an example seems odd; the reason to make things part of the standard library is because their implementation needs to track the Python core closely. For a proxy, it would be the need to reflect additional slot methods as they're added.
It does that too - the unit tests I just added to the tracker issue for this proposal will actually start to fail if something is added to the operator module without the unit tests for the proposed ProxyMixin being updated appropriately (it would actually probably be a good thing to have something similar in the weakref.proxy test suite to prevent a repeat of our effort with forgetting to update that when the operator.index slot was added)
A Python implementation may be able to do this just fine, but the performance penalty would make it uninteresting for many large applications.
It should still be faster than the classic class __getattr__ based proxy implementations it is primarily intended to replace. People that are already using a C-based implementation like zope.proxy aren't going to be affected by the removal of classic classes in 3.0, since they weren't relying on them anyway. In many ways, the TestProxyMixin test suite may prove more useful in the long run than the ProxyMixin class itself, since the test suite provides a template for how to test that a proxy class is doing its job, and also to automatically detect when new C-level operations have been added to the interpreter that the proxy class doesn't support.
For what it's worth, zope.proxy does support subclassing.
But not in a mixin style - since zope.proxy is an extension class in its own right, it can't be combined with other extension classes. The Python version can be combined with anything. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Fred Drake wrote:
On May 21, 2008, at 5:41 AM, Nick Coghlan wrote:
While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name).
The idea that it would serve as an example seems odd; the reason to make things part of the standard library is because their implementation needs to track the Python core closely. For a proxy, it would be the need to reflect additional slot methods as they're added. A Python implementation may be able to do this just fine, but the performance penalty would make it uninteresting for many large applications.
I've added a documentation file to the tracker item and kicked it in Barry's direction (I also bumped the priority to 'release blocker' because I think we need an explicit decision on this one from Barry or Guido before the first beta release of 3.0). Here's what I included in the proposed documentation for ProxyMixin (note that I modelled the implicit unwrapping behaviour directly on the behaviour of weakref.proxy): class:: ProxyMixin(target) A proxy class that ensures all special method invocations which may otherwise bypass the normal :method:`__getattribute__` lookup process are correctly delegated to the specified target object. Normal attribute manipulation operations are also delegated to the specified target object. All operations on a :class:`ProxyMixin` instance return an unproxied result. Operations involving multiple :class:`ProxyMixin` instances (e.g. addition) are permitted, and endeavour to return the same result as would be calculated if the proxy objects were not involved. Custom proxy class implementations may inherit from this class in order to automatically delegate all such special methods that the custom proxy class does not need to provide special handling for. This may not be practical for more complex delegation scenarios (e.g. a local interface to a remote object, or delegating to a weakly referenced target as in :code:`weakref.proxy`), but :class:`ProxyMixin` may still be used as a reference when developing such classes to identify which methods must be defined explicitly on the proxy type in order for them to be delegated correctly by the Python interpreter. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Hi, Nick Coghlan <ncoghlan <at> gmail.com> writes:
One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. Yes. I stumbled upon that multiple times when writing proxies for thread locals. The solution I came up with is rather ugly but works at least:
http://dev.pocoo.org/projects/werkzeug/browser/werkzeug/local.py#L202 The main problem with such proxies is that especially builtin types perform various type checks which break proxies sooner or later anyways (currently even abstract base classes implementing a compatible interface). So as soon as you start doing something like some_list + proxy_to_list you get a TypeError. Regards, Armin
Armin Ronacher wrote:
Hi,
Nick Coghlan <ncoghlan <at> gmail.com> writes:
One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. Yes. I stumbled upon that multiple times when writing proxies for thread locals. The solution I came up with is rather ugly but works at least:
http://dev.pocoo.org/projects/werkzeug/browser/werkzeug/local.py#L202
The main problem with such proxies is that especially builtin types perform various type checks which break proxies sooner or later anyways (currently even abstract base classes implementing a compatible interface). So as soon as you start doing something like some_list + proxy_to_list you get a TypeError.
What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? The link to the ugly proxy is in the mail :-) And no, I'm currently not
Hi, Nick Coghlan <ncoghlan <at> gmail.com> writes: providing any __r*__ methods as I was too lazy to test on each call if the method that is proxied is providing an __rsomething__ or not, and if not come up with an ad-hoc implementation by calling __something__ and reversing the arguments passed.
While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) I also think that the stdlib should mention NotImplemented with a big warning. I see countless classes raising TypeError()s if __add__ or something fails which seem to work alright as long as someone tries to __radd__ it.
Regards, Armin
Armin Ronacher wrote:
Hi,
Nick Coghlan <ncoghlan <at> gmail.com> writes:
What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? The link to the ugly proxy is in the mail :-) And no, I'm currently not providing any __r*__ methods as I was too lazy to test on each call if the method that is proxied is providing an __rsomething__ or not, and if not come up with an ad-hoc implementation by calling __something__ and reversing the arguments passed.
I had another look - indeed, it is almost certainly the missing __r*__ methods which are causing problems for you when interacting with unproxied objects. The other differences between what you're doing and the proposed typetools.ProxyMixin are that: - ProxyMixin delegates the whole __getattribute__ operation to the target object so that descriptors work correctly (invoking object.__getattribute__ explicitly when it needs to retrieve the proxy object's _target attribute). - ProxyMixin correctly delegates in-place assignment operations via the __i*__ methods (note however that using in-place assignment will remove the proxy wrapper, just as it does for weakref.proxy) - ProxyMixin implicitly unwraps other ProxyMixin instances when it encounters them as an argument to an explicitly proxied operation - ProxyMixin doesn't support deprecated operations like __getslice__ or __methods__
While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) I also think that the stdlib should mention NotImplemented with a big warning. I see countless classes raising TypeError()s if __add__ or something fails which seem to work alright as long as someone tries to __radd__ it.
Returning NotImplemented is already mentioned in the documentation for the binary special methods, but I agree that it could be made more obvious that explicitly raising a TypeError from these methods is almost certainly the wrong thing to do. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
Armin Ronacher wrote:
Hi,
Nick Coghlan <ncoghlan <at> gmail.com> writes:
What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? The link to the ugly proxy is in the mail :-) And no, I'm currently not providing any __r*__ methods as I was too lazy to test on each call if the method that is proxied is providing an __rsomething__ or not, and if not come up with an ad-hoc implementation by calling __something__ and reversing the arguments passed.
I had another look - indeed, it is almost certainly the missing __r*__ methods which are causing problems for you when interacting with unproxied objects.
The other differences between what you're doing and the proposed typetools.ProxyMixin are that: - ProxyMixin delegates the whole __getattribute__ operation to the target object so that descriptors work correctly (invoking object.__getattribute__ explicitly when it needs to retrieve the proxy object's _target attribute). - ProxyMixin correctly delegates in-place assignment operations via the __i*__ methods (note however that using in-place assignment will remove the proxy wrapper, just as it does for weakref.proxy)
Couldn't in place operations wrap the return value with a proxy? Michael Foord
- ProxyMixin implicitly unwraps other ProxyMixin instances when it encounters them as an argument to an explicitly proxied operation - ProxyMixin doesn't support deprecated operations like __getslice__ or __methods__
While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) I also think that the stdlib should mention NotImplemented with a big warning. I see countless classes raising TypeError()s if __add__ or something fails which seem to work alright as long as someone tries to __radd__ it.
Returning NotImplemented is already mentioned in the documentation for the binary special methods, but I agree that it could be made more obvious that explicitly raising a TypeError from these methods is almost certainly the wrong thing to do.
Cheers, Nick.
Michael Foord wrote:
Nick Coghlan wrote:
- ProxyMixin correctly delegates in-place assignment operations via the __i*__ methods (note however that using in-place assignment will remove the proxy wrapper, just as it does for weakref.proxy)
Couldn't in place operations wrap the return value with a proxy?
They could, but I tried to copy as much behaviour as possible from weakref.proxy which loses the proxy wrapper when you do an in-place operation:
from weakref import proxy as p class MyInt(int): pass ... x = MyInt(1) px = p(x) px <weakproxy at 0xb768f11c to MyInt at 0xb768d86c> px += 1 px 2
class MyList(list): pass ... y = MyList([]) py = p(y) py <weakproxy at 0xb768f16c to MyList at 0xb768f0f4> py += [] py []
That said, it wouldn't be hard (and might make more sense) to redefine the proxied in-place operations along the following lines: # See tracker item for _deref/_unwrap definitions def __iadd__(self, other): target = _deref(self) result = target + _unwrap(other) if result is target: # Mutated in-place, keep same proxy result = self else: # Returned a different object, make a new proxy result = type(self)(result) return result However, if we did something like that, I'd prefer to see the weakref.proxy implementation change to provide similar semantics. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
else: # Returned a different object, make a new proxy result = type(self)(result)
You might want to check that the result has the same type as the proxied object before doing that. -- Greg
Greg Ewing wrote:
Nick Coghlan wrote:
else: # Returned a different object, make a new proxy result = type(self)(result)
You might want to check that the result has the same type as the proxied object before doing that.
Yep - and I really think it would need to do this. To have a proxy where: proxy_instance += 1 unwraps the proxy is really no good! (At least not for my use cases...) Michael Foord -- http://www.ironpythoninaction.com/ http://www.theotherdelia.co.uk/ http://www.voidspace.org.uk/ http://www.ironpython.info/ http://www.resolverhacks.net/
Armin Ronacher wrote:
I'm currently not providing any __r*__ methods as I was too lazy to test on each call if the method that is proxied is providing an __rsomething__ or not, and if not come up with an ad-hoc implementation by calling __something__ and reversing the arguments passed.
I don't see why you should have to do that, as the reversed method of the proxy will only be called if a prior non-reversed call failed. All the proxy should have to do is delegate any lookups of its own reversed methods to corresponding methods of the proxied object, no differently from any other method. -- Greg
participants (5)
-
Armin Ronacher -
Fred Drake -
Greg Ewing -
Michael Foord -
Nick Coghlan