Making proxy types easier to write and maintain
Hello, It is known to be cumbersome to write a proxy type that will correctly proxy all special methods (this has to be done manually on the type, since special methods are not looked up on the instance: a __getattr__ method would not work). Recently we've had reports of two stdlib types that forgot to implement some special methods: - weakref.proxy doesn't implement __reversed__: http://bugs.python.org/issue19359 - mock.MagicMock doesn't implement __truediv__: http://bugs.python.org/issue20968 In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.: def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name) What do you think? Regards Antoine.
On Wed Mar 19 2014 at 2:46:48 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hello,
It is known to be cumbersome to write a proxy type that will correctly proxy all special methods (this has to be done manually on the type, since special methods are not looked up on the instance: a __getattr__ method would not work).
Recently we've had reports of two stdlib types that forgot to implement some special methods: - weakref.proxy doesn't implement __reversed__: http://bugs.python.org/issue19359 - mock.MagicMock doesn't implement __truediv__: http://bugs.python.org/issue20968
In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Without having the code in front of me, would this only be for magic methods and attributes, or all attributes? IOW would this mean that if I assign an object to __proxy__ it would take care of the uses of __getattr__ for proxying?
On Wed, 19 Mar 2014 19:32:50 +0000 Brett Cannon <bcannon@gmail.com> wrote:
In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Without having the code in front of me, would this only be for magic methods and attributes, or all attributes? IOW would this mean that if I assign an object to __proxy__ it would take care of the uses of __getattr__ for proxying?
In a first approach it would be only for magic methods and attributes. It if proves successful, perhaps it can be later expanded to also be used for regular lookups. By the way, Benjamin pointed me to a prior discussion: http://bugs.python.org/issue643841 Regards Antoine.
On 20 Mar 2014 06:24, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Wed, 19 Mar 2014 19:32:50 +0000 Brett Cannon <bcannon@gmail.com> wrote:
In http://bugs.python.org/issue19359#msg213530 I proposed to
introduce a
"proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Without having the code in front of me, would this only be for magic methods and attributes, or all attributes? IOW would this mean that if I assign an object to __proxy__ it would take care of the uses of __getattr__ for proxying?
In a first approach it would be only for magic methods and attributes. It if proves successful, perhaps it can be later expanded to also be used for regular lookups.
By the way, Benjamin pointed me to a prior discussion: http://bugs.python.org/issue643841
Graeme Dumpleton has also subsequently written a library to handle easier creation of correct proxy types: https://pypi.python.org/pypi/wrapt It isn't as simple as changing one lookup function though - abstract.c reads the type slots directly, and those are already hairy performance critical code paths (especially the binary operator type dispatch). Cheers, Nick.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On Thu, 20 Mar 2014 07:54:39 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Graeme Dumpleton has also subsequently written a library to handle easier creation of correct proxy types: https://pypi.python.org/pypi/wrapt
(is "Graeme" another spelling for "Graham"?)
It isn't as simple as changing one lookup function though - abstract.c reads the type slots directly, and those are already hairy performance critical code paths (especially the binary operator type dispatch).
Oh indeed, you're right. I think my proposal is taking water. Regards Antoine.
On Wed, Mar 19, 2014 at 11:01:39PM +0100, Antoine Pitrou wrote:
On Thu, 20 Mar 2014 07:54:39 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Graeme Dumpleton has also subsequently written a library to handle easier creation of correct proxy types: https://pypi.python.org/pypi/wrapt
(is "Graeme" another spelling for "Graham"?)
Yes.
It isn't as simple as changing one lookup function though - abstract.c reads the type slots directly, and those are already hairy performance critical code paths (especially the binary operator type dispatch).
Oh indeed, you're right. I think my proposal is taking water.
Nevertheless, I would like to see a standard way to proxy dunder methods. -- Steven
On 20 Mar 2014 10:34, "Steven D'Aprano" <steve@pearwood.info> wrote:
On Wed, Mar 19, 2014 at 11:01:39PM +0100, Antoine Pitrou wrote:
On Thu, 20 Mar 2014 07:54:39 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Graeme Dumpleton has also subsequently written a library to handle
easier
creation of correct proxy types: https://pypi.python.org/pypi/wrapt
(is "Graeme" another spelling for "Graham"?)
Yes.
It isn't as simple as changing one lookup function though - abstract.c reads the type slots directly, and those are already hairy performance critical code paths (especially the binary operator type dispatch).
Oh indeed, you're right. I think my proposal is taking water.
Nevertheless, I would like to see a standard way to proxy dunder methods.
I had vague plans to bug Graham (yay, spelling!) about proposing wrapt for inclusion in 3.5 (possibly split between functools and types rather than as a completely new module, though). functools.wraps was always intended as a near term workaround that, like most sufficiently effective workarounds, ended lasting years before its limitations irritated anyone enough for them to go back and do it right :) Cheers, Nick.
-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 03/19/2014 08:49 PM, Nick Coghlan wrote:
I had vague plans to bug Graham (yay, spelling!) about proposing wrapt for inclusion in 3.5 (possibly split between functools and types rather than as a completely new module, though).
I pinged him about it for 3.4. He didn't have the time for it. I hope we can talk him into it for 3.5, wrapt is pretty sweet! //arry/
On 20 March 2014 17:17, Larry Hastings <larry@hastings.org> wrote:
On 03/19/2014 08:49 PM, Nick Coghlan wrote:
I had vague plans to bug Graham (yay, spelling!) about proposing wrapt for inclusion in 3.5 (possibly split between functools and types rather than as a completely new module, though).
I pinged him about it for 3.4. He didn't have the time for it. I hope we can talk him into it for 3.5, wrapt is pretty sweet!
So long as Graham's willing to go along with it, he doesn't have to to be the one to write the PEP. Just needs someone that understands the problem it's designed to solve and is willing to write it up as a PEP, and look at the question of whether it makes more sense to drop it in wholesale as a new module, or to break it up amongst existing modules. (and python-ideas discussion can help with that). One particularly thorny question to investigate: would it be possible to *implicitly* switch functools.wraps() over to using it? I suspect the backwards compatibility implications wouldn't be acceptable, but the idea at least needs to be looked into to identify the specific problems with the concept. Cheers, Nick.
/arry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 03/20/2014 12:49 AM, Nick Coghlan wrote:
So long as Graham's willing to go along with it, he doesn't have to to be the one to write the PEP.
PEP? Why does it need a PEP? I didn't think it'd even be a new top-level library module, it'd be an addition / additions to functools. //arry/
pypy's transparent proxy feature: http://pypy.readthedocs.org/en/latest/objspace-proxies.html#transparent-prox... On Thu, Mar 20, 2014 at 1:56 PM, Larry Hastings <larry@hastings.org> wrote:
On 03/20/2014 12:49 AM, Nick Coghlan wrote:
So long as Graham's willing to go along with it, he doesn't have to to be the one to write the PEP.
PEP? Why does it need a PEP? I didn't think it'd even be a new top-level library module, it'd be an addition / additions to functools.
/arry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
On Thu, Mar 20, 2014 at 11:56 AM, Larry Hastings <larry@hastings.org> wrote:
On 03/20/2014 12:49 AM, Nick Coghlan wrote:
So long as Graham's willing to go along with it, he doesn't have to to be the one to write the PEP.
PEP? Why does it need a PEP? I didn't think it'd even be a new top-level library module, it'd be an addition / additions to functools.
Would it make sense to have a new module named classtools rather than add more class-related stuff to functools. For instance, total_ordering() is the sort of thing I would expect to find in classtools rather than functools. -eric
On 19 March 2014 18:46, Antoine Pitrou <solipsis@pitrou.net> wrote:
In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Would that increase the size of type objects? Would that matter? (There's a similar question that came up in the thread about adding the @ operator over on python-ideas, which is what made me think of it...) Paul
On Wed, 19 Mar 2014 20:15:15 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 19 March 2014 18:46, Antoine Pitrou <solipsis@pitrou.net> wrote:
In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Would that increase the size of type objects? Would that matter? (There's a similar question that came up in the thread about adding the @ operator over on python-ideas, which is what made me think of it...)
One additional slot is one additional pointer field, which is mostly trivial in a type object. Regards Antoine.
On Wed, Mar 19, 2014 at 12:46 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.:
def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name)
What do you think?
Regardless of the attr lookup in the interpreter, it would be nice to have a standardized explicit __proxied__ attribute on proxy objects (similar to __wrapped__). -eric
participants (8)
-
Antoine Pitrou -
Brett Cannon -
Daniel Holth -
Eric Snow -
Larry Hastings -
Nick Coghlan -
Paul Moore -
Steven D'Aprano