Nick Coghlan wrote:

One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. I've needed to do this a few times when wrapping libraries.

Michael Foord

With classic classes, this is trivial, since __getattr__ is always consulted, even for retrieval of special methods.

With new-style classes, however, the __getattribute__ machinery can be bypassed, meaning the only way to proxy an arbitrary instance is to define all of the special methods that have C-level slots.

This issue was actually first raised five and a half years ago [1], but has never been a particularly pressing problem, as anyone with any sense that needed to write a proxy object just used a classic class instead of a new-style one. In 3.0, with the demise of classic classes, that workaround goes away.

So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance? For some proxy class implementers, it would be possible to use this class as a base-class to get the special method delegation 'for free', and for others with more esoteric needs, it would at least provide a reference for which special methods needed to be provided explicitly by the proxy type.

I attached a sample implementation to [1] which is essentially equivalent to weakref.proxy, but with a strong reference to the target, and written in Python rather than C.

I expect the target audience for such a feature to be quite small, but without better support for it in the standard library, I also suspect it could prove to be a show-stopper for the affected developers as far as Py3k migration is concerned.

Cheers, Nick.

[1] http://bugs.python.org/issue643841

Nick Coghlan wrote:

So what do people think of including a ProxyBase implementation in 2.6 and 3.0 that explicitly delegates all of the C-level slots to a designated target instance?

On May 20, 2008, at 7:55 PM, Greg Ewing wrote:

Sounds good to me.

Same here. There's an implementation in zope.proxy which might be useful as a starting point: http://pypi.python.org/pypi/zope.proxy/3.4.0 I'd be willing to write the required documentation, since I'm partly to blame for the implementation. The short description of the package on PyPI contains a typo; that's been fixed in Subversion. :-) -Fred -- Fred Drake <fdrake at acm.org>

On May 21, 2008, at 5:41 AM, Nick Coghlan wrote:

While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name).

The idea that it would serve as an example seems odd; the reason to make things part of the standard library is because their implementation needs to track the Python core closely. For a proxy, it would be the need to reflect additional slot methods as they're added. A Python implementation may be able to do this just fine, but the performance penalty would make it uninteresting for many large applications. For what it's worth, zope.proxy does support subclassing. -Fred -- Fred Drake <fdrake at acm.org>

Fred Drake wrote:

On May 21, 2008, at 5:41 AM, Nick Coghlan wrote:

...

While a proxy class written in C would no doubt be faster than one written in Python, one of the things I'm hoping to achieve is for the stdlib generic proxy to serve as an example for people writing their own new-style proxy classes in addition to being usable as a base class (Adam Olsen suggested ProxyMixin instead of ProxyBase as a possible name).

The idea that it would serve as an example seems odd; the reason to make things part of the standard library is because their implementation needs to track the Python core closely. For a proxy, it would be the need to reflect additional slot methods as they're added. A Python implementation may be able to do this just fine, but the performance penalty would make it uninteresting for many large applications.

I've added a documentation file to the tracker item and kicked it in Barry's direction (I also bumped the priority to 'release blocker' because I think we need an explicit decision on this one from Barry or Guido before the first beta release of 3.0). Here's what I included in the proposed documentation for ProxyMixin (note that I modelled the implicit unwrapping behaviour directly on the behaviour of weakref.proxy): class:: ProxyMixin(target) A proxy class that ensures all special method invocations which may otherwise bypass the normal :method:`__getattribute__` lookup process are correctly delegated to the specified target object. Normal attribute manipulation operations are also delegated to the specified target object. All operations on a :class:`ProxyMixin` instance return an unproxied result. Operations involving multiple :class:`ProxyMixin` instances (e.g. addition) are permitted, and endeavour to return the same result as would be calculated if the proxy objects were not involved. Custom proxy class implementations may inherit from this class in order to automatically delegate all such special methods that the custom proxy class does not need to provide special handling for. This may not be practical for more complex delegation scenarios (e.g. a local interface to a remote object, or delegating to a weakly referenced target as in :code:`weakref.proxy`), but :class:`ProxyMixin` may still be used as a reference when developing such classes to identify which methods must be defined explicitly on the proxy type in order for them to be delegated correctly by the Python interpreter. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Armin Ronacher wrote:

Hi,

Nick Coghlan writes:

...

One of the tasks where classic classes are currently far superior to new-style classes is in writing proxy classes like weakref.proxy - cases where nearly all operations need to be delegated to some other object, with only a few being handled via the proxy type object itself. Yes. I stumbled upon that multiple times when writing proxies for thread locals. The solution I came up with is rather ugly but works at least:

http://dev.pocoo.org/projects/werkzeug/browser/werkzeug/local.py#L202

The main problem with such proxies is that especially builtin types perform various type checks which break proxies sooner or later anyways (currently even abstract base classes implementing a compatible interface). So as soon as you start doing something like some_list + proxy_to_list you get a TypeError.

What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Armin Ronacher wrote:

Hi,

Nick Coghlan writes:

...

What version are you using, and are your proxies correctly implementing all the __r*__ versions of the methods? The link to the ugly proxy is in the mail :-) And no, I'm currently not providing any __r*__ methods as I was too lazy to test on each call if the method that is proxied is providing an __rsomething__ or not, and if not come up with an ad-hoc implementation by calling __something__ and reversing the arguments passed.

I had another look - indeed, it is almost certainly the missing __r*__ methods which are causing problems for you when interacting with unproxied objects. The other differences between what you're doing and the proposed typetools.ProxyMixin are that: - ProxyMixin delegates the whole __getattribute__ operation to the target object so that descriptors work correctly (invoking object.__getattribute__ explicitly when it needs to retrieve the proxy object's _target attribute). - ProxyMixin correctly delegates in-place assignment operations via the __i*__ methods (note however that using in-place assignment will remove the proxy wrapper, just as it does for weakref.proxy) - ProxyMixin implicitly unwraps other ProxyMixin instances when it encounters them as an argument to an explicitly proxied operation - ProxyMixin doesn't support deprecated operations like __getslice__ or __methods__

...

While there are still some cases where types in the standard library raise TypeError directly instead of returning NotImplemented, they're generally pretty good about playing well with others (see the test_typetools.py file attached to the tracker item for #643841) I also think that the stdlib should mention NotImplemented with a big warning. I see countless classes raising TypeError()s if __add__ or something fails which seem to work alright as long as someone tries to __radd__ it.

Returning NotImplemented is already mentioned in the documentation for the binary special methods, but I agree that it could be made more obvious that explicitly raising a TypeError from these methods is almost certainly the wrong thing to do. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Michael Foord wrote:

Nick Coghlan wrote:

...

- ProxyMixin correctly delegates in-place assignment operations via the __i*__ methods (note however that using in-place assignment will remove the proxy wrapper, just as it does for weakref.proxy)

Couldn't in place operations wrap the return value with a proxy?

They could, but I tried to copy as much behaviour as possible from weakref.proxy which loses the proxy wrapper when you do an in-place operation:

...

...

from weakref import proxy as p class MyInt(int): pass ... x = MyInt(1) px = p(x) px px += 1 px 2

...

...

class MyList(list): pass ... y = MyList([]) py = p(y) py py += [] py []

That said, it wouldn't be hard (and might make more sense) to redefine the proxied in-place operations along the following lines: # See tracker item for _deref/_unwrap definitions def __iadd__(self, other): target = _deref(self) result = target + _unwrap(other) if result is target: # Mutated in-place, keep same proxy result = self else: # Returned a different object, make a new proxy result = type(self)(result) return result However, if we did something like that, I'd prefer to see the weakref.proxy implementation change to provide similar semantics. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Nick Coghlan wrote:

else: # Returned a different object, make a new proxy result = type(self)(result)

You might want to check that the result has the same type as the proxied object before doing that. -- Greg