[Python-Dev] The role of NotImplemented: What is it for and when should it be used?
Guido van Rossum
guido at python.org
Mon Nov 3 18:00:06 CET 2014
Not those.
On Nov 3, 2014 8:56 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> On Mon, 3 Nov 2014 08:48:07 -0800
> Guido van Rossum <guido at python.org> wrote:
> > Gotta be brief, but NotImplemented is for all binary ops.
>
> Even in-place ops?
>
> Regards
>
> Antoine.
>
>
> > Power may be an
> > exception because it's ternary?
> > On Nov 3, 2014 8:08 AM, "Brett Cannon" <brett at python.org> wrote:
> >
> > >
> > >
> > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <ethan at stoneleaf.us>
> wrote:
> > >
> > >> Just to be clear, this is about NotImplemented, not
> NotImplementedError.
> > >>
> > >> tl;dr When a binary operation fails, should an exception be raised or
> > >> NotImplemented returned?
> > >>
> > >
> > > The docs for NotImplemented suggest it's only for rich comparison
> methods
> > > and not all binary operators:
> > > https://docs.python.org/3/library/constants.html#NotImplemented . But
> > > then had I not read that I would have said all binary operator methods
> > > should return NotImplemented when the types are incompatible.
> > >
> > > -Brett
> > >
> > >
> > >>
> > >>
> > >> When a binary operation in Python is attempted, there are two
> > >> possibilities:
> > >>
> > >> - it can work
> > >> - it can't work
> > >>
> > >> The main reason [1] that it can't work is that the two operands are of
> > >> different types, and the first type does not know
> > >> how to deal with the second type.
> > >>
> > >> The question then becomes: how does the first type tell Python that it
> > >> cannot perform the requested operation? The most
> > >> obvious answer is to raise an exception, and TypeError is a good
> > >> candidate. The problem with the exception raising
> > >> approach is that once an exception is raised, Python doesn't try
> anything
> > >> else to make the operation work.
> > >>
> > >> What's wrong with that? Well, the second type might know how to
> perform
> > >> the operation, and in fact that is why we have
> > >> the reflected special methods, such as __radd__ and __rmod__ -- but if
> > >> the first type raises an exception the __rxxx__
> > >> methods will not be tried.
> > >>
> > >> Okay, how can the first type tell Python that it cannot do what is
> > >> requested, but to go ahead and check with the second
> > >> type to see if it does? That is where NotImplemented comes in -- if a
> > >> special method (and only a special method)
> > >> returns NotImplemented then Python will check to see if there is
> anything
> > >> else it can do to make the operation succeed;
> > >> if all attempts return NotImplemented, then Python itself will raise
> an
> > >> appropriate exception [2].
> > >>
> > >> In an effort to see how often NotImplemented is currently being
> returned
> > >> I crafted a test script [3] to test the types
> > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict,
> > >> deque, and OrderedDict with the operations for
> > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__,
> > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__,
> > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__,
> > >> __mul__, __or__, __pow__, __rshift__, __sub__,
> > >> __truediv__, and __xor__.
> > >>
> > >> Here are the results of the 275 tests:
> > >> ------------------------------------------------------------
> > >> --------------------
> > >> testing control...
> > >>
> > >> ipow -- Exception <unsupported operand type(s) for ** or pow():
> 'Control'
> > >> and 'subtype'> raised
> > >> errors in Control -- misunderstanding or bug?
> > >>
> > >> testing types against a foreign class
> > >>
> > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> > >> 'items'> raised instead of TypeError
> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> > >> 'items'> raised instead of TypeError
> > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> > >> 'items'> raised instead of TypeError
> > >>
> > >>
> > >> testing types against a subclass
> > >>
> > >> mod(str()) -- NotImplemented not returned, TypeError not raised
> > >>
> > >> iadd(Counter()) -- Exception <'subtype' object has no attribute
> 'items'>
> > >> raised (should have worked)
> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> > >> ior(Counter()) -- Exception <'subtype' object has no attribute
> 'items'>
> > >> raised (should have worked)
> > >> isub(Counter()) -- Exception <'subtype' object has no attribute
> 'items'>
> > >> raised (should have worked)
> > >> ------------------------------------------------------------
> > >> --------------------
> > >>
> > >> Two observations:
> > >>
> > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that
> error
> > >> doesn't show up when testing against 2.7)
> > >>
> > >> - Counter should be returning NotImplemented instead of raising an
> > >> AttributeError, for three reasons [4]:
> > >> - a TypeError is more appropriate
> > >> - subclasses /cannot/ work with the current implementation
> > >> - __iand__ is currently a silent failure if the Counter is empty,
> > >> and the other operand should trigger a failure
> > >>
> > >> Back to the main point...
> > >>
> > >> So, if my understanding is correct:
> > >>
> > >> - NotImplemented is used to signal Python that the requested
> operation
> > >> could not be performed
> > >> - it should be used by the binary special methods to signal type
> > >> mismatch failure, so any subclass gets a chance to work.
> > >>
> > >> Is my understanding correct? Is this already in the docs somewhere,
> and
> > >> I just missed it?
> > >>
> > >> --
> > >> ~Ethan~
> > >>
> > >> [1] at least, it's the main reason in my code
> > >> [2] usually a TypeError, stating either that the operation is not
> > >> supported, or the types are unorderable
> > >> [3] test script at the end
> > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was
> > >> rejected]
> > >>
> > >> -- 8< ------------------------------------------------------------
> > >> ----------------
> > >> from collections import Counter, defaultdict, deque, OrderedDict
> > >> from fractions import Fraction
> > >> from decimal import Decimal
> > >> from enum import Enum
> > >> import operator
> > >> import sys
> > >>
> > >> py_ver = sys.version_info[:2]
> > >>
> > >> types = (
> > >> bytes, bytearray, str, dict, list, tuple,
> > >> Enum, Counter, defaultdict, deque, OrderedDict,
> > >> )
> > >> numeric_types = int, float, Decimal, Fraction
> > >>
> > >> operators = (
> > >> '__add__', '__and__', '__floordiv__',
> > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
> > >> '__imod__', '__imul__', '__ior__', '__ipow__',
> > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__',
> > >> '__lshift__', '__mod__', '__mul__',
> > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
> > >> '__xor__',
> > >> )
> > >>
> > >> if py_ver >= (3, 0):
> > >> operators += ('__gt__', '__ge__', '__le__','__lt__')
> > >>
> > >> ordered_reflections = {
> > >> '__le__': '__ge__',
> > >> '__lt__': '__gt__',
> > >> '__ge__': '__le__',
> > >> '__gt__': '__lt__',
> > >> }
> > >>
> > >>
> > >> # helpers
> > >>
> > >> class SomeOtherClass:
> > >> """"
> > >> used to test behavior when a different type is passed in to the
> > >> special methods
> > >> """
> > >> def __repr__(self):
> > >> return 'SomeOtherClass'
> > >> some_other_class = SomeOtherClass()
> > >>
> > >> class MainClassHandled(Exception):
> > >> """
> > >> called by base class if both operands are of type base class
> > >> """
> > >>
> > >> class SubClassCalled(Exception):
> > >> """
> > >> called by reflected operations for testing
> > >> """
> > >>
> > >> def create_control(test_op):
> > >> def _any(self, other):
> > >> if not type(other) is self.__class__:
> > >> return NotImplemented
> > >> raise MainClassHandled
> > >> class Control:
> > >> "returns NotImplemented when other object is not supported"
> > >> _any.__name__ = op
> > >> setattr(Control, test_op, _any)
> > >> return Control()
> > >>
> > >> def create_subtype(test_op, base_class=object):
> > >> def _any(*a):
> > >> global subclass_called
> > >> subclass_called = True
> > >> raise SubClassCalled
> > >> class subtype(base_class):
> > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ =
> _any
> > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ =
> _any
> > >> __and__ = __xor__ = __or__ = _any
> > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ =
> __rfloordiv__ =
> > >> _any
> > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ =
> __rrshift__ =
> > >> _any
> > >> __rand__ = __rxor__ = __ror__ = _any
> > >> __le__ = __lt__ = __gt__ = __ge__ = _any
> > >> if issubclass(subtype, (bytes, bytearray)):
> > >> value = b'hello'
> > >> elif issubclass(subtype, str):
> > >> value = 'goodbye'
> > >> elif issubclass(subtype, (list, tuple)):
> > >> value = (1, 2, 3)
> > >> elif issubclass(subtype, (int, float, Decimal, Fraction)):
> > >> value = 42
> > >> else:
> > >> # ignore value
> > >> return subtype()
> > >> return subtype(value)
> > >>
> > >>
> > >> # test exceptions
> > >>
> > >> # control against some other class
> > >> print('testing control...\n')
> > >> errors = False
> > >> for op in operators:
> > >> control = create_control(op)
> > >> op = getattr(operator, op)
> > >> try:
> > >> op(control, some_other_class)
> > >> except TypeError:
> > >> # the end result of no method existing, or each method called
> > >> returning
> > >> # NotImplemented because it does not know how to perform the
> > >> requested
> > >> # operation between the two types
> > >> pass
> > >> except Exception as exc:
> > >> errors = True
> > >> print('%s(%s()) -- Exception <%s> raised instead of
> TypeError' %
> > >> (op.__name__, test_type.__name__, exc))
> > >> else:
> > >> errors = True
> > >> print('Control -- TypeError not raised for op %r' % op)
> > >> if errors:
> > >> print('errors in Control -- misunderstanding or bug?\n')
> > >>
> > >> # control against a subclass
> > >> errors = False
> > >> for op in operators:
> > >> subclass_called = False
> > >> control = create_control(op)
> > >> subtype = create_subtype(op, control.__class__)
> > >> op = getattr(operator, op)
> > >> try:
> > >> op(control, subtype)
> > >> except SubClassCalled:
> > >> # if the control class properly signals that it doesn't know
> how
> > >> to
> > >> # perform the operation, of if Python notices that a
> reflected
> > >> # operation exists, we get here (which is good)
> > >> pass
> > >> except MainClassHandled:
> > >> errors = True
> > >> print('Control did not yield to subclass for op %r' % op)
> > >> except Exception as exc:
> > >> if subclass_called:
> > >> # exception was subverted to something more appropriate
> (like
> > >> # unorderable types)
> > >> pass
> > >> errors = True
> > >> print('%s -- Exception <%s> raised' %
> > >> (op.__name__, exc))
> > >> else:
> > >> errors = True
> > >> print('Control -- op %r appears to have succeeded (it should
> not
> > >> have)' % op)
> > >> if errors:
> > >> print('errors in Control -- misunderstanding or bug?\n')
> > >>
> > >>
> > >> # tests
> > >> print('testing types against a foreign class\n')
> > >> for test_type in types + numeric_types:
> > >> errors = False
> > >> for op in operators:
> > >> op = getattr(operator, op)
> > >> try:
> > >> op(test_type(), some_other_class)
> > >> except TypeError:
> > >> pass
> > >> except Exception as exc:
> > >> errors = True
> > >> print('%s(%s()) -- Exception <%s> raised instead of
> > >> TypeError' %
> > >> (op.__name__, test_type.__name__, exc))
> > >> else:
> > >> print('%s(%s()) -- NotImplemented not returned, TypeError
> > >> not raised' %
> > >> (op.__name__, test_type.__name__))
> > >> if errors:
> > >> print()
> > >>
> > >> print()
> > >>
> > >> # test subclasses
> > >> print('testing types against a subclass\n')
> > >> for test_type in types:
> > >> errors = False
> > >> for op in operators:
> > >> subclass_called = False
> > >> if not test_type.__dict__.get(op):
> > >> continue
> > >> subclass = create_subtype(op, test_type)
> > >> op = getattr(operator, op)
> > >> try:
> > >> if test_type is str:
> > >> op('%s', subtype)
> > >> else:
> > >> op(test_type(), subtype)
> > >> except SubClassCalled:
> > >> # expected, ignore
> > >> pass
> > >> except Exception as exc:
> > >> if subclass_called:
> > >> # exception raised by subclass was changed
> > >> pass
> > >> errors = True
> > >> print('%s(%s()) -- Exception <%s> raised (should have
> > >> worked)' %
> > >> (op.__name__, test_type.__name__, exc))
> > >> else:
> > >> errors = True
> > >> print('%s(%s()) -- NotImplemented not returned, TypeError
> > >> not raised' %
> > >> (op.__name__, test_type.__name__))
> > >> if errors:
> > >> print()
> > >> for test_type in numeric_types:
> > >> errors = False
> > >> for op in operators:
> > >> subclass_called = False
> > >> if not test_type.__dict__.get(op):
> > >> continue
> > >> subtype = create_subtype(op, test_type)
> > >> op = getattr(operator, op)
> > >> try:
> > >> op(test_type(), subtype)
> > >> except SubClassCalled:
> > >> # expected, ignore
> > >> pass
> > >> except Exception as exc:
> > >> if subclass_called:
> > >> # exception raised by subclass was changed
> > >> pass
> > >> errors = True
> > >> print('%s(%s()) -- Exception <%s> raised (should have
> > >> worked)' %
> > >> (op.__name__, test_type.__name__, exc))
> > >> else:
> > >> errors = True
> > >> print('%s(%s)) -- NotImplemented not returned' %
> > >> (op.__name__, test_type.__name__))
> > >> if errors:
> > >> print()
> > >> -- 8< ------------------------------------------------------------
> > >> ----------------
> > >> _______________________________________________
> > >> Python-Dev mailing list
> > >> Python-Dev at python.org
> > >> https://mail.python.org/mailman/listinfo/python-dev
> > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> > >> brett%40python.org
> > >>
> > >
> > > _______________________________________________
> > > Python-Dev mailing list
> > > Python-Dev at python.org
> > > https://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe:
> > > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> > >
> > >
> >
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141103/64913c14/attachment.html>
More information about the Python-Dev
mailing list