[Python-Dev] The role of NotImplemented: What is it for and when should it be used?
Brett Cannon
brett at python.org
Mon Nov 3 16:05:31 CET 2014
On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <ethan at stoneleaf.us> wrote:
> Just to be clear, this is about NotImplemented, not NotImplementedError.
>
> tl;dr When a binary operation fails, should an exception be raised or
> NotImplemented returned?
>
The docs for NotImplemented suggest it's only for rich comparison methods
and not all binary operators:
https://docs.python.org/3/library/constants.html#NotImplemented . But then
had I not read that I would have said all binary operator methods should
return NotImplemented when the types are incompatible.
-Brett
>
>
> When a binary operation in Python is attempted, there are two
> possibilities:
>
> - it can work
> - it can't work
>
> The main reason [1] that it can't work is that the two operands are of
> different types, and the first type does not know
> how to deal with the second type.
>
> The question then becomes: how does the first type tell Python that it
> cannot perform the requested operation? The most
> obvious answer is to raise an exception, and TypeError is a good
> candidate. The problem with the exception raising
> approach is that once an exception is raised, Python doesn't try anything
> else to make the operation work.
>
> What's wrong with that? Well, the second type might know how to perform
> the operation, and in fact that is why we have
> the reflected special methods, such as __radd__ and __rmod__ -- but if the
> first type raises an exception the __rxxx__
> methods will not be tried.
>
> Okay, how can the first type tell Python that it cannot do what is
> requested, but to go ahead and check with the second
> type to see if it does? That is where NotImplemented comes in -- if a
> special method (and only a special method)
> returns NotImplemented then Python will check to see if there is anything
> else it can do to make the operation succeed;
> if all attempts return NotImplemented, then Python itself will raise an
> appropriate exception [2].
>
> In an effort to see how often NotImplemented is currently being returned I
> crafted a test script [3] to test the types
> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict,
> deque, and OrderedDict with the operations for
> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__,
> __ilshift__, __imod__, __imul__, __ior__, __ipow__,
> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__,
> __mul__, __or__, __pow__, __rshift__, __sub__,
> __truediv__, and __xor__.
>
> Here are the results of the 275 tests:
> ------------------------------------------------------------
> --------------------
> testing control...
>
> ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control'
> and 'subtype'> raised
> errors in Control -- misunderstanding or bug?
>
> testing types against a foreign class
>
> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> 'items'> raised instead of TypeError
> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> 'items'> raised instead of TypeError
> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> 'items'> raised instead of TypeError
>
>
> testing types against a subclass
>
> mod(str()) -- NotImplemented not returned, TypeError not raised
>
> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> raised (should have worked)
> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> raised (should have worked)
> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> raised (should have worked)
> ------------------------------------------------------------
> --------------------
>
> Two observations:
>
> - __ipow__ doesn't seem to behave properly in the 3.x line (that error
> doesn't show up when testing against 2.7)
>
> - Counter should be returning NotImplemented instead of raising an
> AttributeError, for three reasons [4]:
> - a TypeError is more appropriate
> - subclasses /cannot/ work with the current implementation
> - __iand__ is currently a silent failure if the Counter is empty, and
> the other operand should trigger a failure
>
> Back to the main point...
>
> So, if my understanding is correct:
>
> - NotImplemented is used to signal Python that the requested operation
> could not be performed
> - it should be used by the binary special methods to signal type
> mismatch failure, so any subclass gets a chance to work.
>
> Is my understanding correct? Is this already in the docs somewhere, and I
> just missed it?
>
> --
> ~Ethan~
>
> [1] at least, it's the main reason in my code
> [2] usually a TypeError, stating either that the operation is not
> supported, or the types are unorderable
> [3] test script at the end
> [4] https://bugs.python.org/issue22766 [returning NotImplemented was
> rejected]
>
> -- 8< ------------------------------------------------------------
> ----------------
> from collections import Counter, defaultdict, deque, OrderedDict
> from fractions import Fraction
> from decimal import Decimal
> from enum import Enum
> import operator
> import sys
>
> py_ver = sys.version_info[:2]
>
> types = (
> bytes, bytearray, str, dict, list, tuple,
> Enum, Counter, defaultdict, deque, OrderedDict,
> )
> numeric_types = int, float, Decimal, Fraction
>
> operators = (
> '__add__', '__and__', '__floordiv__',
> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
> '__imod__', '__imul__', '__ior__', '__ipow__',
> '__irshift__', '__isub__', '__itruediv__', '__ixor__',
> '__lshift__', '__mod__', '__mul__',
> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
> '__xor__',
> )
>
> if py_ver >= (3, 0):
> operators += ('__gt__', '__ge__', '__le__','__lt__')
>
> ordered_reflections = {
> '__le__': '__ge__',
> '__lt__': '__gt__',
> '__ge__': '__le__',
> '__gt__': '__lt__',
> }
>
>
> # helpers
>
> class SomeOtherClass:
> """"
> used to test behavior when a different type is passed in to the
> special methods
> """
> def __repr__(self):
> return 'SomeOtherClass'
> some_other_class = SomeOtherClass()
>
> class MainClassHandled(Exception):
> """
> called by base class if both operands are of type base class
> """
>
> class SubClassCalled(Exception):
> """
> called by reflected operations for testing
> """
>
> def create_control(test_op):
> def _any(self, other):
> if not type(other) is self.__class__:
> return NotImplemented
> raise MainClassHandled
> class Control:
> "returns NotImplemented when other object is not supported"
> _any.__name__ = op
> setattr(Control, test_op, _any)
> return Control()
>
> def create_subtype(test_op, base_class=object):
> def _any(*a):
> global subclass_called
> subclass_called = True
> raise SubClassCalled
> class subtype(base_class):
> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any
> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any
> __and__ = __xor__ = __or__ = _any
> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ =
> _any
> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ =
> _any
> __rand__ = __rxor__ = __ror__ = _any
> __le__ = __lt__ = __gt__ = __ge__ = _any
> if issubclass(subtype, (bytes, bytearray)):
> value = b'hello'
> elif issubclass(subtype, str):
> value = 'goodbye'
> elif issubclass(subtype, (list, tuple)):
> value = (1, 2, 3)
> elif issubclass(subtype, (int, float, Decimal, Fraction)):
> value = 42
> else:
> # ignore value
> return subtype()
> return subtype(value)
>
>
> # test exceptions
>
> # control against some other class
> print('testing control...\n')
> errors = False
> for op in operators:
> control = create_control(op)
> op = getattr(operator, op)
> try:
> op(control, some_other_class)
> except TypeError:
> # the end result of no method existing, or each method called
> returning
> # NotImplemented because it does not know how to perform the
> requested
> # operation between the two types
> pass
> except Exception as exc:
> errors = True
> print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
> (op.__name__, test_type.__name__, exc))
> else:
> errors = True
> print('Control -- TypeError not raised for op %r' % op)
> if errors:
> print('errors in Control -- misunderstanding or bug?\n')
>
> # control against a subclass
> errors = False
> for op in operators:
> subclass_called = False
> control = create_control(op)
> subtype = create_subtype(op, control.__class__)
> op = getattr(operator, op)
> try:
> op(control, subtype)
> except SubClassCalled:
> # if the control class properly signals that it doesn't know how
> to
> # perform the operation, of if Python notices that a reflected
> # operation exists, we get here (which is good)
> pass
> except MainClassHandled:
> errors = True
> print('Control did not yield to subclass for op %r' % op)
> except Exception as exc:
> if subclass_called:
> # exception was subverted to something more appropriate (like
> # unorderable types)
> pass
> errors = True
> print('%s -- Exception <%s> raised' %
> (op.__name__, exc))
> else:
> errors = True
> print('Control -- op %r appears to have succeeded (it should not
> have)' % op)
> if errors:
> print('errors in Control -- misunderstanding or bug?\n')
>
>
> # tests
> print('testing types against a foreign class\n')
> for test_type in types + numeric_types:
> errors = False
> for op in operators:
> op = getattr(operator, op)
> try:
> op(test_type(), some_other_class)
> except TypeError:
> pass
> except Exception as exc:
> errors = True
> print('%s(%s()) -- Exception <%s> raised instead of
> TypeError' %
> (op.__name__, test_type.__name__, exc))
> else:
> print('%s(%s()) -- NotImplemented not returned, TypeError not
> raised' %
> (op.__name__, test_type.__name__))
> if errors:
> print()
>
> print()
>
> # test subclasses
> print('testing types against a subclass\n')
> for test_type in types:
> errors = False
> for op in operators:
> subclass_called = False
> if not test_type.__dict__.get(op):
> continue
> subclass = create_subtype(op, test_type)
> op = getattr(operator, op)
> try:
> if test_type is str:
> op('%s', subtype)
> else:
> op(test_type(), subtype)
> except SubClassCalled:
> # expected, ignore
> pass
> except Exception as exc:
> if subclass_called:
> # exception raised by subclass was changed
> pass
> errors = True
> print('%s(%s()) -- Exception <%s> raised (should have
> worked)' %
> (op.__name__, test_type.__name__, exc))
> else:
> errors = True
> print('%s(%s()) -- NotImplemented not returned, TypeError not
> raised' %
> (op.__name__, test_type.__name__))
> if errors:
> print()
> for test_type in numeric_types:
> errors = False
> for op in operators:
> subclass_called = False
> if not test_type.__dict__.get(op):
> continue
> subtype = create_subtype(op, test_type)
> op = getattr(operator, op)
> try:
> op(test_type(), subtype)
> except SubClassCalled:
> # expected, ignore
> pass
> except Exception as exc:
> if subclass_called:
> # exception raised by subclass was changed
> pass
> errors = True
> print('%s(%s()) -- Exception <%s> raised (should have
> worked)' %
> (op.__name__, test_type.__name__, exc))
> else:
> errors = True
> print('%s(%s)) -- NotImplemented not returned' %
> (op.__name__, test_type.__name__))
> if errors:
> print()
> -- 8< ------------------------------------------------------------
> ----------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141103/dac47e0f/attachment.html>
More information about the Python-Dev
mailing list