[Python-Dev] The role of NotImplemented: What is it for and when should it be used?
Guido van Rossum
guido at python.org
Mon Nov 3 17:48:07 CET 2014
Gotta be brief, but NotImplemented is for all binary ops. Power may be an
exception because it's ternary?
On Nov 3, 2014 8:08 AM, "Brett Cannon" <brett at python.org> wrote:
>
>
> On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> Just to be clear, this is about NotImplemented, not NotImplementedError.
>>
>> tl;dr When a binary operation fails, should an exception be raised or
>> NotImplemented returned?
>>
>
> The docs for NotImplemented suggest it's only for rich comparison methods
> and not all binary operators:
> https://docs.python.org/3/library/constants.html#NotImplemented . But
> then had I not read that I would have said all binary operator methods
> should return NotImplemented when the types are incompatible.
>
> -Brett
>
>
>>
>>
>> When a binary operation in Python is attempted, there are two
>> possibilities:
>>
>> - it can work
>> - it can't work
>>
>> The main reason [1] that it can't work is that the two operands are of
>> different types, and the first type does not know
>> how to deal with the second type.
>>
>> The question then becomes: how does the first type tell Python that it
>> cannot perform the requested operation? The most
>> obvious answer is to raise an exception, and TypeError is a good
>> candidate. The problem with the exception raising
>> approach is that once an exception is raised, Python doesn't try anything
>> else to make the operation work.
>>
>> What's wrong with that? Well, the second type might know how to perform
>> the operation, and in fact that is why we have
>> the reflected special methods, such as __radd__ and __rmod__ -- but if
>> the first type raises an exception the __rxxx__
>> methods will not be tried.
>>
>> Okay, how can the first type tell Python that it cannot do what is
>> requested, but to go ahead and check with the second
>> type to see if it does? That is where NotImplemented comes in -- if a
>> special method (and only a special method)
>> returns NotImplemented then Python will check to see if there is anything
>> else it can do to make the operation succeed;
>> if all attempts return NotImplemented, then Python itself will raise an
>> appropriate exception [2].
>>
>> In an effort to see how often NotImplemented is currently being returned
>> I crafted a test script [3] to test the types
>> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict,
>> deque, and OrderedDict with the operations for
>> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__,
>> __ilshift__, __imod__, __imul__, __ior__, __ipow__,
>> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__,
>> __mul__, __or__, __pow__, __rshift__, __sub__,
>> __truediv__, and __xor__.
>>
>> Here are the results of the 275 tests:
>> ------------------------------------------------------------
>> --------------------
>> testing control...
>>
>> ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control'
>> and 'subtype'> raised
>> errors in Control -- misunderstanding or bug?
>>
>> testing types against a foreign class
>>
>> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute
>> 'items'> raised instead of TypeError
>> iand(Counter()) -- NotImplemented not returned, TypeError not raised
>> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute
>> 'items'> raised instead of TypeError
>> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute
>> 'items'> raised instead of TypeError
>>
>>
>> testing types against a subclass
>>
>> mod(str()) -- NotImplemented not returned, TypeError not raised
>>
>> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'>
>> raised (should have worked)
>> iand(Counter()) -- NotImplemented not returned, TypeError not raised
>> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'>
>> raised (should have worked)
>> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'>
>> raised (should have worked)
>> ------------------------------------------------------------
>> --------------------
>>
>> Two observations:
>>
>> - __ipow__ doesn't seem to behave properly in the 3.x line (that error
>> doesn't show up when testing against 2.7)
>>
>> - Counter should be returning NotImplemented instead of raising an
>> AttributeError, for three reasons [4]:
>> - a TypeError is more appropriate
>> - subclasses /cannot/ work with the current implementation
>> - __iand__ is currently a silent failure if the Counter is empty,
>> and the other operand should trigger a failure
>>
>> Back to the main point...
>>
>> So, if my understanding is correct:
>>
>> - NotImplemented is used to signal Python that the requested operation
>> could not be performed
>> - it should be used by the binary special methods to signal type
>> mismatch failure, so any subclass gets a chance to work.
>>
>> Is my understanding correct? Is this already in the docs somewhere, and
>> I just missed it?
>>
>> --
>> ~Ethan~
>>
>> [1] at least, it's the main reason in my code
>> [2] usually a TypeError, stating either that the operation is not
>> supported, or the types are unorderable
>> [3] test script at the end
>> [4] https://bugs.python.org/issue22766 [returning NotImplemented was
>> rejected]
>>
>> -- 8< ------------------------------------------------------------
>> ----------------
>> from collections import Counter, defaultdict, deque, OrderedDict
>> from fractions import Fraction
>> from decimal import Decimal
>> from enum import Enum
>> import operator
>> import sys
>>
>> py_ver = sys.version_info[:2]
>>
>> types = (
>> bytes, bytearray, str, dict, list, tuple,
>> Enum, Counter, defaultdict, deque, OrderedDict,
>> )
>> numeric_types = int, float, Decimal, Fraction
>>
>> operators = (
>> '__add__', '__and__', '__floordiv__',
>> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
>> '__imod__', '__imul__', '__ior__', '__ipow__',
>> '__irshift__', '__isub__', '__itruediv__', '__ixor__',
>> '__lshift__', '__mod__', '__mul__',
>> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
>> '__xor__',
>> )
>>
>> if py_ver >= (3, 0):
>> operators += ('__gt__', '__ge__', '__le__','__lt__')
>>
>> ordered_reflections = {
>> '__le__': '__ge__',
>> '__lt__': '__gt__',
>> '__ge__': '__le__',
>> '__gt__': '__lt__',
>> }
>>
>>
>> # helpers
>>
>> class SomeOtherClass:
>> """"
>> used to test behavior when a different type is passed in to the
>> special methods
>> """
>> def __repr__(self):
>> return 'SomeOtherClass'
>> some_other_class = SomeOtherClass()
>>
>> class MainClassHandled(Exception):
>> """
>> called by base class if both operands are of type base class
>> """
>>
>> class SubClassCalled(Exception):
>> """
>> called by reflected operations for testing
>> """
>>
>> def create_control(test_op):
>> def _any(self, other):
>> if not type(other) is self.__class__:
>> return NotImplemented
>> raise MainClassHandled
>> class Control:
>> "returns NotImplemented when other object is not supported"
>> _any.__name__ = op
>> setattr(Control, test_op, _any)
>> return Control()
>>
>> def create_subtype(test_op, base_class=object):
>> def _any(*a):
>> global subclass_called
>> subclass_called = True
>> raise SubClassCalled
>> class subtype(base_class):
>> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any
>> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any
>> __and__ = __xor__ = __or__ = _any
>> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ =
>> _any
>> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ =
>> _any
>> __rand__ = __rxor__ = __ror__ = _any
>> __le__ = __lt__ = __gt__ = __ge__ = _any
>> if issubclass(subtype, (bytes, bytearray)):
>> value = b'hello'
>> elif issubclass(subtype, str):
>> value = 'goodbye'
>> elif issubclass(subtype, (list, tuple)):
>> value = (1, 2, 3)
>> elif issubclass(subtype, (int, float, Decimal, Fraction)):
>> value = 42
>> else:
>> # ignore value
>> return subtype()
>> return subtype(value)
>>
>>
>> # test exceptions
>>
>> # control against some other class
>> print('testing control...\n')
>> errors = False
>> for op in operators:
>> control = create_control(op)
>> op = getattr(operator, op)
>> try:
>> op(control, some_other_class)
>> except TypeError:
>> # the end result of no method existing, or each method called
>> returning
>> # NotImplemented because it does not know how to perform the
>> requested
>> # operation between the two types
>> pass
>> except Exception as exc:
>> errors = True
>> print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
>> (op.__name__, test_type.__name__, exc))
>> else:
>> errors = True
>> print('Control -- TypeError not raised for op %r' % op)
>> if errors:
>> print('errors in Control -- misunderstanding or bug?\n')
>>
>> # control against a subclass
>> errors = False
>> for op in operators:
>> subclass_called = False
>> control = create_control(op)
>> subtype = create_subtype(op, control.__class__)
>> op = getattr(operator, op)
>> try:
>> op(control, subtype)
>> except SubClassCalled:
>> # if the control class properly signals that it doesn't know how
>> to
>> # perform the operation, of if Python notices that a reflected
>> # operation exists, we get here (which is good)
>> pass
>> except MainClassHandled:
>> errors = True
>> print('Control did not yield to subclass for op %r' % op)
>> except Exception as exc:
>> if subclass_called:
>> # exception was subverted to something more appropriate (like
>> # unorderable types)
>> pass
>> errors = True
>> print('%s -- Exception <%s> raised' %
>> (op.__name__, exc))
>> else:
>> errors = True
>> print('Control -- op %r appears to have succeeded (it should not
>> have)' % op)
>> if errors:
>> print('errors in Control -- misunderstanding or bug?\n')
>>
>>
>> # tests
>> print('testing types against a foreign class\n')
>> for test_type in types + numeric_types:
>> errors = False
>> for op in operators:
>> op = getattr(operator, op)
>> try:
>> op(test_type(), some_other_class)
>> except TypeError:
>> pass
>> except Exception as exc:
>> errors = True
>> print('%s(%s()) -- Exception <%s> raised instead of
>> TypeError' %
>> (op.__name__, test_type.__name__, exc))
>> else:
>> print('%s(%s()) -- NotImplemented not returned, TypeError
>> not raised' %
>> (op.__name__, test_type.__name__))
>> if errors:
>> print()
>>
>> print()
>>
>> # test subclasses
>> print('testing types against a subclass\n')
>> for test_type in types:
>> errors = False
>> for op in operators:
>> subclass_called = False
>> if not test_type.__dict__.get(op):
>> continue
>> subclass = create_subtype(op, test_type)
>> op = getattr(operator, op)
>> try:
>> if test_type is str:
>> op('%s', subtype)
>> else:
>> op(test_type(), subtype)
>> except SubClassCalled:
>> # expected, ignore
>> pass
>> except Exception as exc:
>> if subclass_called:
>> # exception raised by subclass was changed
>> pass
>> errors = True
>> print('%s(%s()) -- Exception <%s> raised (should have
>> worked)' %
>> (op.__name__, test_type.__name__, exc))
>> else:
>> errors = True
>> print('%s(%s()) -- NotImplemented not returned, TypeError
>> not raised' %
>> (op.__name__, test_type.__name__))
>> if errors:
>> print()
>> for test_type in numeric_types:
>> errors = False
>> for op in operators:
>> subclass_called = False
>> if not test_type.__dict__.get(op):
>> continue
>> subtype = create_subtype(op, test_type)
>> op = getattr(operator, op)
>> try:
>> op(test_type(), subtype)
>> except SubClassCalled:
>> # expected, ignore
>> pass
>> except Exception as exc:
>> if subclass_called:
>> # exception raised by subclass was changed
>> pass
>> errors = True
>> print('%s(%s()) -- Exception <%s> raised (should have
>> worked)' %
>> (op.__name__, test_type.__name__, exc))
>> else:
>> errors = True
>> print('%s(%s)) -- NotImplemented not returned' %
>> (op.__name__, test_type.__name__))
>> if errors:
>> print()
>> -- 8< ------------------------------------------------------------
>> ----------------
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141103/b15efd92/attachment.html>
More information about the Python-Dev
mailing list