[issue16011] "in" should be consistent with return value of __contains__
New submission from Neal Parikh: The Python 2.7.3 documentation says the following about defining __contains__: "Called to implement membership test operators. Should return true if item is in self, false otherwise. For mapping objects, this should consider the keys of the mapping rather than the values or the key-item pairs." This suggests that while you should return True/False, you don't need to. Also, the documentation doesn't say that if you return a value other than True or False, it will silently coerce other values to be True or False when you use "in". For example:
class Foo(object): ... def __contains__(self, item): return 42 ... foo = Foo() 3 in foo True foo.__contains__(3) 42
When __contains__ is defined, "in" should return whatever __contains__ returns, even if this value is neither True nor False. That would be consistent with, for example, the comparison operators: You can return anything from __lt__ and "x < 4" will correctly pass along that value regardless of what it is. ---------- assignee: docs@python components: Documentation, Interpreter Core messages: 171085 nosy: docs@python, nparikh priority: normal severity: normal status: open title: "in" should be consistent with return value of __contains__ type: behavior versions: Python 2.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Christian Heimes added the comment: The internal API is limited to flags as the sq_contains slot and the API function can only return an int: PyAPI_FUNC(int) PySequence_In(PyObject *o, PyObject *value) In order to return the Python object we'd have to alter the API which we can't do as the type slot and the function is part of the stable API and ABI. A doc update should explain why the code doesn't behave as you expect it. ---------- nosy: +christian.heimes _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Ezio Melotti added the comment: http://docs.python.org/py3k/reference/datamodel.html#object.__contains__ says that __contain__ "Should return true if item is in self, false otherwise.". Here the lowercase true and false mean any true or false value, not just True and False and it's indeed possible to return any value. The fact that 'in' only returns True/False, possibly converting the return value of __contains__, should be documented in the 'in' documentation. http://docs.python.org/py3k/reference/expressions.html#membership-test-detai... says "For user-defined classes which define the __contains__() method, x in y is true if and only if y.__contains__(x) is true.". This could be changed to say that the return value is converted to True/False. ---------- keywords: +easy nosy: +ezio.melotti stage: -> needs patch versions: +Python 3.2, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Mark Dickinson added the comment: There was a related discussion on python-ideas a while back: http://mail.python.org/pipermail/python-ideas/2010-July/007733.html ---------- nosy: +mark.dickinson _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Neal Parikh added the comment: Thanks for passing along the thread, it was interesting. Oddly, it seemed to die off with no real resolution. I realize it is now too late to change __contains__ to return a non-boolean value, but for reference, the reason I wanted to return something different from __contains__ was also to implement a DSL, though not a SQL-related one. Basically, I have some kind of abstract mathematical set S, and I wanted "x in S" to return a constraint that the variable x must lie in the set S for use in a larger problem description. (In fact, one could implement __contains__ so it returned True/False with a constant numeric argument and a constraint object with a variable argument.) This would have mirrored the way one would write all this mathematically. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
STINNER Victor added the comment: Do you need this PEP? http://www.python.org/dev/peps/pep-0335/ ---------- nosy: +haypo _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Neal Parikh added the comment: I don't, but thanks for passing that along. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Eric Lafontaine added the comment: Hi, For user-defined class, it's up to the class to do the right implementation in my opinion. It's true the description is wrong though. x in y means that x exist inside of y (so that the execution of y.__contain__(x) is executed successfully and (I guess) doesn't return None,False or 0). I'll modify the doc to be : For user-defined classes which define the __contains__() method, x in y is false if y.__contains__(x) is returning either None,False or 0. Otherwise, x in y return true. Regards, Eric Lafontaine ---------- nosy: +Eric Lafontaine _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
R. David Murray added the comment: Eric: that is not precise enough, I'm afraid :) See msg171093 for the correct documentation update. Specifically, in returns True if __contains__ returns a true value, and False otherwise (not the difference in case, it matters). There are more things than just None, False, and 0 that are false in Python. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Eric Lafontaine added the comment: Hi all, Here are the test I've made to understand the behavior : class Foo_42(object): def __contains__(self,item): return 42 class Foo_neg(object): def __contains__(self,item): return -42 class Foo_None(object): def __contains__(self,item): return class Foo_false(object): def __contains__(self,item): return False class Foo_true(object): def __contains__(self,item): return True for foo in [Foo_false(),Foo_None(),Foo_neg(),Foo_true(),Foo_42()]: print("3 in foo:" + str(3 in foo)) print("foo.__contains__(3)" + str(foo.__contains__(3))) which output this : 3 in foo:False foo.__contains__(3)False 3 in foo:False foo.__contains__(3)None 3 in foo:True foo.__contains__(3)-42 3 in foo:True foo.__contains__(3)True 3 in foo:True foo.__contains__(3)42 So as long as __contains__ return False or None, the 'in' operator will be False. Otherwise true. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Eric Lafontaine added the comment: Hi David, sorry for the delay on my part for providing how I was getting to that conclusion. I've also resurrected an old post as I want to start contributing more seriously :). As this is documentation only (not changing the code behavior), I didn't take a look at the implementation of "in". Could you enlighten me on what else would be considered "False" in this case? Regards, Eric Lafontaine ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Eric Lafontaine added the comment: oh, I've got what you meant! Proposed change : For user-defined classes which define the __contains__() method, the in operator will convert to False "x in y" if y.__contains__(x) return False, 0 or None. Otherwise, the in operator will return True for any other value being returned by y.__contains__(x). Would that make more sense? Regards, Eric Lafontaine ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
R. David Murray added the comment:
bool(()) False bool([]) False bool('') False
What you want to say is that 'in' coerces the result returned by __contains__ to a boolean value. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Eric Lafontaine added the comment: (first time trying to reply through email) thanks for the example and you are right : class Foo_emptylist(object): def __contains__(self,item): return [] class Foo_emptydict(object): def __contains__(self,item): return {} class Foo_emptystring(object): def __contains__(self,item): return '' for foo in [Foo_false(),Foo_None(),Foo_emptylist(),Foo_emptydict(),Foo_emptystring(),Foo_neg(),Foo_true(),Foo_42()]: print("3 in foo:" + str(3 in foo)) print("foo.__contains__(3)" + str(foo.__contains__(3))) 3 in foo:False foo.__contains__(3)False 3 in foo:False foo.__contains__(3)None 3 in foo:False foo.__contains__(3)[] 3 in foo:False foo.__contains__(3){} 3 in foo:False foo.__contains__(3) 3 in foo:True foo.__contains__(3)-42 3 in foo:True foo.__contains__(3)True 3 in foo:True foo.__contains__(3)42 So the proposition should be this then? For user-defined classes which define a __contains__() method, the in operator will apply bool() on the __contains__() method. In other words, "x in y" is equivalent to "bool(y.__contains__(x))" and will return False if bool(y.__contains__(x)) is equivalent to false. Éric Lafontaine | Membre du Projet VUE, Groupe Contrôle Génie électrique, 54ème promotion UdeS | Étudiant en maitrise TI à l'ETS VAS OPS chez Bell Mobility « Nous voulons proposer une alternative de transport en présentant un véhicule électrique spécifiquement conçu pour les déplacements urbains. » 2017-02-07 21:10 GMT-05:00 R. David Murray <report@bugs.python.org>:
R. David Murray added the comment:
bool(()) False bool([]) False bool('') False
What you want to say is that 'in' coerces the result returned by __contains__ to a boolean value.
----------
_______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
R. David Murray added the comment: You've got the right idea, but you are repeating yourself. Keep it as short as possible while still conveying the correct information. "coerce to boolean" is better than "apply bool", because the code may not in fact be using the bool function to do it. Your "equivalent to" phrase would be OK as an alternative, but you only need to show the equivalence, no need to also explain it in words. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Changes by Amit Kumar <dtu.amit@gmail.com>: ---------- pull_requests: +116 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Changes by Berker Peksag <berker.peksag@gmail.com>: ---------- nosy: +berker.peksag stage: needs patch -> patch review versions: +Python 3.5, Python 3.6, Python 3.7 -Python 3.2, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
R. David Murray added the comment: New changeset 0ae7c8bd614d3aa1fcaf2d71a10ff1148c80d9b5 by R. David Murray (Amit Kumar) in branch 'master': bpo-16011 clarify that 'in' always returns a boolean value https://github.com/python/cpython/commit/0ae7c8bd614d3aa1fcaf2d71a10ff1148c8... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Changes by Mariatta Wijaya <mariatta.wijaya@gmail.com>: ---------- pull_requests: +772 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Changes by Mariatta Wijaya <mariatta.wijaya@gmail.com>: ---------- pull_requests: +773 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
R. David Murray added the comment: Thanks, ---------- stage: patch review -> backport needed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Mariatta Wijaya added the comment: New changeset 0957f262c5e47167efd520624557aebdc61bfda8 by Mariatta in branch '3.5': bpo-16011: clarify that 'in' always returns a boolean value (GH-152) (GH-875) https://github.com/python/cpython/commit/0957f262c5e47167efd520624557aebdc61... ---------- nosy: +Mariatta _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Mariatta Wijaya added the comment: New changeset c4021af50526f488c0c280e7c7eaa83ef80ae1df by Mariatta in branch '3.6': bpo-16011: clarify that 'in' always returns a boolean value (GH-874) https://github.com/python/cpython/commit/c4021af50526f488c0c280e7c7eaa83ef80... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Changes by Mariatta Wijaya <mariatta.wijaya@gmail.com>: ---------- pull_requests: +785 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Mariatta Wijaya added the comment: New changeset fd704a02ca8713b0dae644f7f182f3e3d1218dbf by Mariatta in branch '2.7': bpo-16011: clarify that 'in' always returns a boolean value (GH-152) (GH-883) https://github.com/python/cpython/commit/fd704a02ca8713b0dae644f7f182f3e3d12... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
Mariatta Wijaya added the comment: Thanks everyone :) Patch has been merged and backported to 3.6, 3.5, and 2.7. ---------- resolution: -> fixed stage: backport needed -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16011> _______________________________________
participants (10)
-
Amit Kumar -
Berker Peksag
-
Christian Heimes
-
Eric Lafontaine
-
Ezio Melotti
-
Mariatta Wijaya
-
Mark Dickinson
-
Neal Parikh
-
R. David Murray
-
STINNER Victor