More appropriate behavior for the NotImplemented object
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though? It seems to me that `not NotImplemented` should result in `NotImplemented` and attempting to convert it to `bool` should raise a `TypeError` exception. Take the following example: ``` def __lt__(self, other): return not self.__ge__(other): def __le__(self, other): return not self.__gt__(other): def __ge__(self, other): <some code that might or might not return NotImplemented> ``` Currently, this will not work because `NotImplemented` is truthy and `not NotImplemented` is `False`, so it is necessary to complicate the implementations of `__lt__` and `__le__` to specifically check whether the value returned from the complementary method returned `NotImplemented` or not. If the value of `not NotImplemented` was `NotImplemented` then the coding pattern above would simply work.
NotImplemented is not supposed to be used in any operation. It's just a special value used in coercion since raising exceptions would be too costly. In your example you would need to check for this special value using the "is" comparison. See https://www.python.org/dev/peps/pep-0208/ and https://www.python.org/dev/peps/pep-0207/ for details. On 3/11/2020 10:42 AM, Steve Jorgensen wrote:
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though?
It seems to me that `not NotImplemented` should result in `NotImplemented` and attempting to convert it to `bool` should raise a `TypeError` exception.
Take the following example:
``` def __lt__(self, other): return not self.__ge__(other):
def __le__(self, other): return not self.__gt__(other):
def __ge__(self, other): <some code that might or might not return NotImplemented> ```
Currently, this will not work because `NotImplemented` is truthy and `not NotImplemented` is `False`, so it is necessary to complicate the implementations of `__lt__` and `__le__` to specifically check whether the value returned from the complementary method returned `NotImplemented` or not.
If the value of `not NotImplemented` was `NotImplemented` then the coding pattern above would simply work. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/U7GOYM... Code of Conduct: http://python.org/psf/codeofconduct/
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 11 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Steve Jorgensen wrote:
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though? It seems to me that not NotImplemented should result in NotImplemented and attempting to convert it to bool should raise a TypeError exception. Take the following example: def __lt__(self, other): return not self.__ge__(other):
def __le__(self, other): return not self.__gt__(other):
def __ge__(self, other): <some code that might or might not return NotImplemented>
Currently, this will not work because NotImplemented is truthy and not NotImplemented is False, so it is necessary to complicate the implementations of __lt__ and __le__ to specifically check whether the value returned from the complementary method returned NotImplemented or not. If the value of not NotImplemented was NotImplemented then the coding pattern above would simply work.
Yeah, it seems contra-intuitive. But there is some logic in that behaviour. `not NotImplemented` is a boolean test and, I think, it follows the "Truth Value Testing" (https://docs.python.org/3/library/stdtypes.html#truth). According to that, `bool(NotImplemented)` returns True so `not bool(NotImplemented)` returns False. Ultimately, the NotImplemented singleton is like any other Python object and, from some point of view, it is logical that it behaves like other objects. From that point of view, `not NotImplemented` must not return `NotImplemented` ever --it is not a boolean value! Raising a TypeError when executing `bool(NotImplemented)`? Why? In my understanding, if you are comparing two object with different types (which cannot be compared), False is not a wrong answer. Remember that NotImplemented is a "special value which should be returned to indicate that the operation is not implemented with respect to the other type" and its purpose is that "the interpreter will try the reflected operation on the other type" (https://docs.python.org/3/library/constants.html#NotImplemented). So, IMHO, you should check if __ge__ returns a NotImplemented singleton or not and react accordingly to the result of that test.
On Wed, Mar 11, 2020 at 09:42:15AM -0000, Steve Jorgensen wrote:
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though?
It seems to me that `not NotImplemented` should result in `NotImplemented` and attempting to convert it to `bool` should raise a `TypeError` exception.
To my disappointment, you will get your wish, at least for the second part: https://bugs.python.org/issue35712 I am disappointed because, to me, it is a fundamental part of Python's object model that *everything* can be interpreted as a truthy/falsey object (in the absence of bugs). Anyway, the deprecation warning has been checked in, so it's probably only a matter of time before you will get your TypeError :-) -- Steven
Steven D'Aprano wrote:
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though? It seems to me that not NotImplemented should result in NotImplemented and attempting to convert it to bool should raise a TypeError exception. To my disappointment, you will get your wish, at least for the second
On Wed, Mar 11, 2020 at 09:42:15AM -0000, Steve Jorgensen wrote: part: https://bugs.python.org/issue35712 I am disappointed because, to me, it is a fundamental part of Python's object model that everything can be interpreted as a truthy/falsey object (in the absence of bugs). Anyway, the deprecation warning has been checked in, so it's probably only a matter of time before you will get your TypeError :-)
Oouh, I also feel disappointed. Indeed, I feel that NotImplemented is a fiasco and that a TypeError exception should be raised instead. But I suppose it is central to duck typing concept...
Looks like NotImplemented is widely misunderstood. It should *ONLY* be
returned from binary operator overloads, like __add__.
On Wed, Mar 11, 2020 at 03:49
Steven D'Aprano wrote:
I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though? It seems to me that not NotImplemented should result in NotImplemented and attempting to convert it to bool should raise a TypeError exception. To my disappointment, you will get your wish, at least for the second
On Wed, Mar 11, 2020 at 09:42:15AM -0000, Steve Jorgensen wrote: part: https://bugs.python.org/issue35712 I am disappointed because, to me, it is a fundamental part of Python's object model that everything can be interpreted as a truthy/falsey object (in the absence of bugs). Anyway, the deprecation warning has been checked in, so it's probably only a matter of time before you will get your TypeError :-)
Oouh, I also feel disappointed. Indeed, I feel that NotImplemented is a fiasco and that a TypeError exception should be raised instead. But I suppose it is central to duck typing concept... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7VCQ6M... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Guido van Rossum wrote:
Looks like NotImplemented is widely misunderstood. It should ONLY be returned from binary operator overloads, like __add__. On Wed, Mar 11, 2020 at 03:49 jdveiga@gmail.com wrote:
Steven D'Aprano wrote: On Wed, Mar 11, 2020 at 09:42:15AM -0000, Steve Jorgensen wrote: I realize this is probably something that would be hard to change for compatibility reasons. Maybe someone can think of a way around that though? It seems to me that not NotImplemented should result in NotImplemented and attempting to convert it to bool should raise a TypeError exception. To my disappointment, you will get your wish, at least for the second part: https://bugs.python.org/issue35712 I am disappointed because, to me, it is a fundamental part of Python's object model that everything can be interpreted as a truthy/falsey object (in the absence of bugs). Anyway, the deprecation warning has been checked in, so it's probably only a matter of time before you will get your TypeError :-) Oouh, I also feel disappointed. Indeed, I feel that NotImplemented is a fiasco and that a TypeError exception should be raised instead. But I suppose it is central to duck typing concept...
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7VCQ6M... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile)
Widely misunderstood and little known. But documentation is very clear if read carefully. Even an example is provided. However, it is a special case and special cases, you know, are... special. They can be difficult to understand by non-advanced users --like me.
11.03.20 12:39, Steven D'Aprano пише:
https://bugs.python.org/issue35712
I am disappointed because, to me, it is a fundamental part of Python's object model that *everything* can be interpreted as a truthy/falsey object (in the absence of bugs).
NotImplemented is special. It is more special than even None. It is special enough to break the rule. It's only purpose is to be a signal value for special methods like __add__ or __eq__, and errors related to interpreting it as boolean are pretty common (there was ones even in the stdlib). This is a clear case case of "Practicality beats purity." There is a precedence (although not in the stdlib): NumPy array with dtype=bool.
import numpy bool(numpy.array([False, False])) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
On Wed, Mar 11, 2020 at 07:06:20PM +0200, Serhiy Storchaka wrote:
11.03.20 12:39, Steven D'Aprano пише:
https://bugs.python.org/issue35712
I am disappointed because, to me, it is a fundamental part of Python's object model that *everything* can be interpreted as a truthy/falsey object (in the absence of bugs).
NotImplemented is special. It is more special than even None. It is special enough to break the rule.
Thanks Serhiy, I've been watching the relevent b.p.o. issue and while I'm disappointed about the break to the clean truthiness model, I never came up with a strong enough argument against the change to argue :-) so I have to accept it. As you say, practicality beats purity. -- Steven
On Wed, 11 Mar 2020 at 18:08, Serhiy Storchaka
There is a precedence (although not in the stdlib): NumPy array with dtype=bool.
import numpy bool(numpy.array([False, False])) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Actually, this is the behaviour of ndarray with any dtype. And IMHO ithis is quite.... terrible? I was so used to have False for an empty iterable that ndarray surprised me. I had to add a boolean() function in my little msutils module, that works also for ndarrays. And the reason ndarray do this is because it also override __eq__(): https://github.com/numpy/numpy/issues/15573
On Thu, Mar 12, 2020 at 5:50 AM Marco Sulla via Python-ideas < python-ideas@python.org> wrote:
Actually, this is the behaviour of ndarray with any dtype. And IMHO ithis is quite.... terrible?
I can see how you would think that. But the fact is that element-wise operations are very important to numpy. I was so used to have False for an empty iterable I hate to be pedantic, but it doesn't work for iterables anyway: In [2]: def iterable(): ...: for i in range(2): ...: yield i ...: In [3]: it = iterable() In [4]: bool(it) Out[4]: True In [5]: next(it) Out[5]: 0 In [6]: next(it) Out[6]: 1 In [7]: next(it) --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) <ipython-input-7-bc1ab118995a> in <module> ----> 1 next(it) StopIteration: In [8]: bool(it) Out[8]: True And I don't think it could -- the only way to tell if a iterable is empty (really, exhausted) is to call next on it, and then you will have changed its state. It does work for most sequences, however. Honestly though, I've always been uncomfortable with the whole "truthiness" thing. I understand that in most cases, a empty sequence is logically Falsey, but the fact is that depending on the context other "Falsey" values, like 0 for intetrgers, may be perfectly reasonable. I do, in fact, often use: if not a_sequence: do_something_special but If I really want to know if a sequence is empty, maybe if len(sequence) == 0: is really a more clear test. And as the above demonstrates, if you get an non-sequence iterable, then len() test will raise, which is much better than it always returning True. in contrast, a numpy array will raise if you explicitly (or implicitly) call bool() on it, whereas the len() test will work as expected. And that is much safer behavior: much better to raise than pass silently with an incorrect result. And the fact is that numpy arrays will not replace an arbitrary sequence in many contexts -- at least in this case, you get an Exception the very first time to try to use one that way. And by the way, numpy arrays are not Sequences as far as collections.abc is concerned: isinstance(arr, collections.abc.Sequence) A new point: maybe the default behavior for non-sequence iterables (like generators) should be to raise on a bool() call? Note: that is NOT well thought out, it seems the default for any object is for bool(an_object) to return True -- there may well be good reason for that. -CHB
On Thu, 12 Mar 2020 at 17:09, Christopher Barker
I was so used to have False for an empty iterable I hate to be pedantic, but it doesn't work for iterables anyway:
Yes, I know that it does not work for iterators. This is another problem :-)
but If I really want to know if a sequence is empty, maybe if len(sequence) == 0: is really a more clear test.
But it's against PEP 8.
And by the way, numpy arrays are not Sequences as far as collections.abc is concerned
But it quacks like a sequence. IMHO numpy arrays should result as they implements collections.abc.Sequence, even if it's really not true. Like dict:
help(dict) Help on class dict in module builtins:
class dict(object) [...]
issubclass(dict, MutableMapping) True
On Thu, Mar 12, 2020 at 09:08:57AM -0700, Christopher Barker via Python-ideas wrote:
On Thu, Mar 12, 2020 at 5:50 AM Marco Sulla via Python-ideas < python-ideas@python.org> wrote:
Actually, this is the behaviour of ndarray with any dtype. And IMHO ithis is quite.... terrible?
I can see how you would think that. But the fact is that element-wise operations are very important to numpy.
If I could go back to 1995 or thereabouts when Python was first starting, I would make a clear distinction between scalar and vector operations, like Julia does. Borrowing Julia's syntax: a == b # compare a to b, returns True or False a .== b # elementwise comparison a*b # multiply a by b a.*b # elementwise multiplication -- Steven
If I could go back to 1995 or thereabouts when Python was first starting, I would make a clear distinction between scalar and vector operations, like Julia does.
Borrowing Julia's syntax:
a == b # compare a to b, returns True or False a .== b # elementwise comparison
a*b # multiply a by b a.*b # elementwise multiplication
That was borrowed from MATLAB, and as a former MATLAB user, I can tell you it sucks ;-). In the Python world, it would mean double as many operators, and that was discussed for years, until it was realized that the only truly useful new operation was matrix multiplication — and thus @ was introduced. So in this case, maybe there is another use case: elementwise vs vector-wise bool(). But I don’t think it’s important— numpy arrays are different enough from other sequences that you really need to know which you have anyway. In fact, having bool() raise on numpy arrays means code that isn’t expecting them will fail early, which is a good thing. We need to remember that PEP-8 is a set of recommendations that do not apply to all cases. Making numpy arrays able to conform to PEP-8 is a non-goal. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Mar 11, 2020 at 10:09 AM Serhiy Storchaka
There is a precedence (although not in the stdlib): NumPy array with dtype=bool.
import numpy bool(numpy.array([False, False])) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
There are some cases in the standard library, e.g.
import dbm.dumb db = dbm.dumb.open('foo') db.close() not db Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: DBM object has already been closed
mmap.mmap also does this. Maybe it's a bug, or maybe the objects are considered to be invalid and no longer fit for any use. -- Ben
12.03.20 08:08, Ben Rudiak-Gould пише:
There are some cases in the standard library, e.g.
import dbm.dumb db = dbm.dumb.open('foo') db.close() not db Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: DBM object has already been closed
mmap.mmap also does this. Maybe it's a bug, or maybe the objects are considered to be invalid and no longer fit for any use.
Uhm... Thanks, I did not know about these examples. This behavior is explainable (the result depends on the context which is not available at that moment), although may be not intended. I think it is not a bug.
On Mar 11, 2020, at 02:42, Steve Jorgensen
Take the following example:
``` def __lt__(self, other): return not self.__ge__(other):
def __le__(self, other): return not self.__gt__(other):
def __ge__(self, other): <some code that might or might not return NotImplemented> ```
Usually you can just use @total_ordering. Or, if you’re doing something almost but not quite like total_ordering and you need to do it for 20 types, you write your own variation on total_ordering (or a base class or metaclass instead of a decorator, if appropriate). When you can’t, often you just want `return not self >= other`, much like a delegating __iter__ usually wants `yield from iter(other)` rather than `yield from other.__iter__()`. When you really do need to call the dunder methods directly for a good reason, you really do need to deal with their API, including testing for NotImplemented with is. The hard part of the design is thinking through the reverse-direction cases that NotImplemented implies (both with subclasses and with unrelated classes), not knowing how to write the extra line or two of implementation. So this doesn’t seem like much of a problem, much less a problem worth breaking fundamental truthiness for a builtin type.
On 03/11/2020 11:16 AM, Andrew Barnert via Python-ideas wrote:
On Mar 11, 2020, at 02:42, Steve Jorgensen wrote:
Take the following example:
``` def __lt__(self, other): return not self.__ge__(other):
def __le__(self, other): return not self.__gt__(other):
def __ge__(self, other): <some code that might or might not return NotImplemented> ```
Usually you can just use @total_ordering.
Even @total_ordering suffered from this bug for a number of years.
[...] you [may] just want `return not self >= other` [in the `__dunder__`]
It's been my experience that when working with `__dunders__`, I'm better off sticking with `__dunders__`, at least with the rich comparison operators.
[...] you really do need to deal with their API, including testing for NotImplemented with it.
Forgetting to deal with `NotImplemented` is a common mistake. Is there any context in which `bool(NotImplemented)` actually makes sense?
So this doesn’t seem like much of a problem, much less a problem worth breaking fundamental truthiness for a builtin type.
Considering that the stdlib itself has suffered from it, I would say it's more than a small problem. -- ~Ethan~
participants (12)
-
Andrew Barnert -
Ben Rudiak-Gould -
Christopher Barker -
Christopher Barker -
Ethan Furman -
Guido van Rossum -
jdveiga＠gmail.com -
M.-A. Lemburg -
Marco Sulla -
Serhiy Storchaka -
Steve Jorgensen -
Steven D'Aprano