Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not?
I'm curious why the return value of __contains__ is coerced to True or False, whereas the return value of "normal" comparison operators like __lt__ and the like are not. The latter return the value directly without forcing it to be True or False. This makes overriding __contains__ significantly less flexible, so I'm wondering why it's designed or implemented that way. (I believe it's the cmp_outcome() function in Python/ceval.c that does this: http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545) For example, the "peewee" ORM overloads __lt__ and the like so it can map Python expressions to SQL. But it can't do this with the "in" operator due to the result of "x in y" always returning True or False in Python. So it (ab)uses the "<<" operator to do this instead (See the peewee docs at http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups ). I'm sure there's a good reason for why "in" is different here, but I can't see why right now. -Ben
Oh, just to show what I mean here with some code (same in both Python 2.x and 3.x): ----- from __future__ import print_function class C(object): def __contains__(self, x): print('__contains__({0!r})'.format(x)) return x def __lt__(self, x): print('__lt__({0!r})'.format(x)) return x c = C() print(42 in c) print(0 in c) print(c < 42) print(c < 0) ----- This prints the following: __contains__(42) True __contains__(0) False __lt__(42) 42 __lt__(0) 0 Whereas I'd kinda expect it to print: __contains__(42) 42 __contains__(0) 0 __lt__(42) 42 __lt__(0) 0 -Ben On Mon, Jul 15, 2013 at 12:21 PM, Ben Hoyt <benhoyt@gmail.com> wrote:
I'm curious why the return value of __contains__ is coerced to True or False, whereas the return value of "normal" comparison operators like __lt__ and the like are not. The latter return the value directly without forcing it to be True or False.
This makes overriding __contains__ significantly less flexible, so I'm wondering why it's designed or implemented that way. (I believe it's the cmp_outcome() function in Python/ceval.c that does this: http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545)
For example, the "peewee" ORM overloads __lt__ and the like so it can map Python expressions to SQL. But it can't do this with the "in" operator due to the result of "x in y" always returning True or False in Python. So it (ab)uses the "<<" operator to do this instead (See the peewee docs at http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups ).
I'm sure there's a good reason for why "in" is different here, but I can't see why right now.
-Ben
On 15 July 2013 10:21, Ben Hoyt <benhoyt@gmail.com> wrote:
I'm sure there's a good reason for why "in" is different here, but I can't see why right now.
It depends on what you mean by "good reason" - PEP 207 (which is what allows arbitrary objects to be returned from comparison operations) was entirely about replacing __cmp__ with the rich comparison methods, it doesn't mention __contains__ at all. At this point the main limitations are backwards compatibility (having existing containment tests suddenly start returning anything other than True or False would be problematic), along with the signature of CPython's sq_contains slot (it returns an integer rather than a PyObject pointer). Accordingly, to convert containment testing to a rich comparison operation would require a new protocol. That said, there is potential value in redefining containment in terms of a symmetric protocol (rather than the current only-controlled-by-the-container behaviour), so such a PEP may be worth writing. (it would initially be a topic for python-ideas rather than python-dev, though) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but I think it'd be beyond me (I'm not familiar enough with CPython internals, protocols, etc). Can you explain what you mean by "symmetric protocol rather than the current only-controlled-by-the-container behaviour"? -Ben On Mon, Jul 15, 2013 at 12:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 15 July 2013 10:21, Ben Hoyt <benhoyt@gmail.com> wrote:
I'm sure there's a good reason for why "in" is different here, but I can't see why right now.
It depends on what you mean by "good reason" - PEP 207 (which is what allows arbitrary objects to be returned from comparison operations) was entirely about replacing __cmp__ with the rich comparison methods, it doesn't mention __contains__ at all.
At this point the main limitations are backwards compatibility (having existing containment tests suddenly start returning anything other than True or False would be problematic), along with the signature of CPython's sq_contains slot (it returns an integer rather than a PyObject pointer).
Accordingly, to convert containment testing to a rich comparison operation would require a new protocol. That said, there is potential value in redefining containment in terms of a symmetric protocol (rather than the current only-controlled-by-the-container behaviour), so such a PEP may be worth writing. (it would initially be a topic for python-ideas rather than python-dev, though)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jul 15, 2013 at 03:34:08PM +1200, Ben Hoyt wrote:
Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but I think it'd be beyond me (I'm not familiar enough with CPython internals, protocols, etc).
Can you explain what you mean by "symmetric protocol rather than the current only-controlled-by-the-container behaviour"?
Most operators can be controlled by either the left-hand or right-hand operand. For example, x + y can end up calling either x.__add__(y) or y.__radd_(x). The `in` operator is an exception, it only ever calls the container: x in y => y.__contains__(x) but never x.__contained_by__(y) -- Steven
participants (3)
-
Ben Hoyt -
Nick Coghlan -
Steven D'Aprano