[issue12067] Doc: remove errors about mixed-type comparisons.
New submission from Terry J. Reedy <tjreedy@udel.edu>: Current 3.2 doc, 5.9. Comparisons, has this paragraph about mixed-type comparisons. "The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects need not have the same type. If both are numbers, they are converted to a common type. Otherwise, the == and != operators *always* consider objects of different types to be unequal, while the <, >, >= and <= operators raise a TypeError when comparing objects of different types that do not implement these operators for the given pair of types. You can control comparison behavior of objects of non-built-in types by defining rich comparison methods like __gt__(), described in section Basic customization." Sentence 3: "If both are numbers, they are converted to a common type." I suspect it would be more true to say 'common internal type' as I would not think it a language requirement to produce Python objects. In any case, I think it is only true for built-in number types, and I do not see that qualification anywhere previously. That aside, it does not appear to be true for Decimals and Fractions in 2.7.1. Sentence 4: first clause is only true for built-in types. That qualification is not obvious to everyone, as evidenced by a current python-list sub thread. For 2.7, which has a different continuation, I suggest adding 'built-in' before 'objects of'. For 3.2/3, I suggest deleting '*always*' and adding a comma after 'TypeError' so that the 'when' condition applies to equality comparisons also. After discussion about same-type comparisons, there is another paragraph about mixed-type comparison: "Comparison of objects of the differing types depends on whether either of the types provide explicit support for the comparison. Most numeric types can be compared with one another, but comparisons of float and Decimal are not supported to avoid the inevitable confusion arising from representation issues such as float('1.1') being inexactly represented and therefore not exactly equal to Decimal('1.1') which is. When cross-type comparison is not supported, the comparison method returns NotImplemented. This can create the illusion of non-transitivity between supported cross-type comparisons and unsupported comparisons. For example, Decimal(2) == 2 and 2 == float(2) but Decimal(2) != float(2)." I suggest deleting this entirely. The first sentence and first clause of the second repeat what was said above. The rest is obsolete as float/decimal comparisons *are* implemented in 2.7.1 and 3.2.0. ---------- assignee: docs@python components: Documentation keywords: easy messages: 135873 nosy: docs@python, rhettinger, terry.reedy priority: normal severity: normal stage: needs patch status: open title: Doc: remove errors about mixed-type comparisons. versions: Python 2.7, Python 3.2, Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Chris Rebert <pybugs@rebertia.com>: ---------- nosy: +cvrebert _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Ezio Melotti <ezio.melotti@gmail.com> added the comment: Can you provide a patch? ---------- nosy: +ezio.melotti _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson <dickinsm@gmail.com> added the comment: [Docs] "If both are numbers, they are converted to a common type." [Terry] "In any case, I think it is only true for built-in number types," It's not even true for built-in number types. When comparing an int with a float, it's definitely *not* the case that the int is converted to a float and the floats compared. And that's for good reason: the int -> float conversion is lossy for large integers, so if int <-> float comparisons just converted the int to a float before comparing, we'd have (for example):
10**16 == 1e16 == 10**16 + 1
leading to broken transitivity of equality, and strange dict and set behaviour. So int <-> float comparisons do a complicated dance under the hood to compare the exact numerical values of the two objects and produce the correct result. I'm not sure what the intent of the original sentence was, or how to reword it. The key point is simply that it *is* possible to compare an int with a float, and that the result is sensible, based on numeric values. ---------- nosy: +mark.dickinson _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Ezio Melotti <ezio.melotti@gmail.com> added the comment: Would it be ok to state that: 1) <, >, ==, >=, <=, and != compare the values of two objects; 2) the two objects don't necessarily have to be of the same type; 3) with == and !=, objects of different types compare unequal, unless they define a specific __eq__ and/or __ne__; 4) with <, >, <=, and >=, the comparison of objects of different types raises a TypeError, unless they define specific __lt__, __gt__, __le__, and __ge__; 5) some built-in types define these operations, so it's possible to compare e.g. int and floats; This should summarize the possible behaviors. There's no reason IMHO to expose implementation details and to special case built-in types (unless their comparison is actually different and doesn't depend on __eq__, __ne__, etc.). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: In Python 3, where all classes inherit from object, the default rules are, by experiment, (which someone can verify from the code) simpler than you stated. 3. By default, == and /= compare identities. 4. By default, order comparisons raise TypeError. ob <= ob raises even though ob == ob because ob is ob. I am not sure of the method look-up rules for rich comparisons, but perhaps the following are true: 3) with == and !=, an object is equal to itself and different objects (a is not b) compare unequal, unless the class of the first define a specific __eq__ and __ne__; 4) with <, >, <=, and >=, comparison raises a TypeError, unless the class of the first object defines specific __lt__, __gt__, __le__, and __ge__, or the class of the second defines the reflected method (__ge__ reflects __lt__, etcetera); What is not clear to me is whether the reflected method is called if the first raises TypeError. The special method names doc (reference 3.3) says "A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments. ... There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection." Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section. I think point 5 should say a bit more: builtin numbers compare as expected, even when of different types; builtin sequences compare lexicographically. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Mike Hoy <mhoy09@gmail.com>: ---------- nosy: +mikehoy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- type: -> enhancement _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: http://bugs.python.org/issue15997 is this issue related to what Terry has mentioned: Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section. I am working on this patch and need confirmation as to whether or not this has to be included in my patch. I'm not clear on it so I may just pass on making a patch if it is required for this issue. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: After further thought: This section is about the syntax operators, not the special methods. The syntax operators never evaluate to NotImplemented as they (apparently) interpret its return from a special method the same as a raising of TypeError, and always raise TypeError when neither the op or its reflection is supported. So there should be no mention of NotImplemented here. Just a reference to 3.3. #15997 is related to my 'wonder' but not directly relevant to a patch for this. Please submit a draft patch when you have one. I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method, at least for a couple of cases. (And same seems true for arithmetic ops too.) class C(): def __ge__(self, other): # print("in C.__ge__", end='') return True def __add__(self, other): return 44 __radd__ = __add__ class O(): def __le__(self, other): # print ("in O.__le__") return NotImplemented def __add__(self, other): return NotImplemented c = C() o = O() ob = object() print(c >= o, o <= c, ob <= c) # True True True # print(ob <= ob) # raises TypeError print(c + o, o + c, ob + c) # 44 44 44 # print(ob + ob) # raises TypeError # print(ob >= o) # with O.__le__ print uncommented # in O.__le__ # so non-implemented reflected o <= ob *is* called # TypeError: unorderable types: object() >= O() ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: I've attempted to incorporate both Terry's and Ezio's suggestions. Here is a patch to get started with. There is a section that has been deleted. Patch uploaded. ---------- keywords: +patch Added file: http://bugs.python.org/file27256/issue12067-expressions.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Chris Jerdonek added the comment: Some minor comments: -The operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the +``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the values of two I think it reads better to start a sentence (and in this case a paragraph) with a word rather than a symbol. -values of two objects. The objects need not have the same type. If both are +objects. The two objects don't necessarily have to be of the same type. With The replacement sentence seems wordier to me. +(:meth:`__ge__()` reflects :meth:`__lt__()`, etcetera). Builtin numbers compare +as expected, even when of different types. Builtin sequences compare "Built-in" is hyphenated in the docs. See, for example, here: http://docs.python.org/dev/library/functions.html ---------- nosy: +cjerdonek _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson added the comment:
I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method
Are you sure? raise TypeError *should* result in the operation being abandoned, with the reflected operation not tried. Python 3.3.0rc2+ (default:3504cbb3e1d8, Sep 20 2012, 22:08:44) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information.
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: You are right, I did not look deep enough. I was fooled by the conversion of NotImplemented, returned from object.__le__, etc, to TypeError. Sorry for that noise. For comparison and arithmetic, the actual alternative to defining a function that returns NotImplemented seems to be to not define it at all. class C(): def __ge__(self, other): return True def __add__(self, other): return 44 __radd__ = __add__ class O(): pass # removed NotImplemented defs c = C() o = O() print(c >= o, o <= c) # True True print(c + o, o + c) # 44 44 (I looked at the codes for binary_op1 in abstract.c and do_richcompare in object.c and do not yet see any effective difference between not defined and a NotImplemented return.) I'll take a look at the patch later. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: Changed patch to include suggestions by Chris Jerdonek. http://bugs.python.org/issue12067#msg170953 ---------- hgrepos: +153 Added file: http://bugs.python.org/file27442/issue12067-expressions_v2.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Mike Hoy <mhoy09@gmail.com>: ---------- hgrepos: -153 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: Considering that the docs have changed does this issue still need to be open? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Hi, I'd like to revive this issue. IMHO, the changes in issue12067-expressions_v2.diff go too far. I don't think that deleting the entire section about the details of comparing objects of the same type makes sense. I agree with Terry's statement in msg170936 that the chapter is about the operators and not about the ways to customize them, so some of what the patch introduces in that area should not be introduced. So far, that means that I'm pretty much against that patch entirely... Having said that, I do believe that there are still issues: 1. both the 2.7 and 3.x sections about the comparison operators are sufficiently convoluted and could be organized better by grouping the various statements that are made, into categories like this: - comparisons involving objects of user-defined types - comparing objects of same built-in type - comparing objects of differing built-in type 2. There are still some errors, ambiguities and omissions that need to be fixed. For example, in the 3.x version: a) omission about treatment of NaN for numbers of different type (could in theory be implied from the statement "are converted to same type", but that statement is problematic as was pointed out in Mark's comment on this issue). b) Amgiguous statement "Most numeric types [of same type] can be compared with one another.". I think what is true is that all built-in numeric types (including Fraction and Decimal) compare mathematically correct in 3.x, and that the only non-support is that complex numbers are not considered orderable and an attempt to use an ordering operator raises TypeError. c) Ambiguous statement "When cross-type comparison is not supported, the comparison method returns NotImplemented.". If this is about the customization methods, it should not be here, but there. Here, it is relevant that a TypeError is raised when using the operator. d) Terminology in "Bytes objects are compared lexicographically using the numeric values of their elements.": Chapter [4.8.1. Bytes] defines bytes objects as immutable sequences of single bytes (not elements). e) Terminology in "Tuples and lists are compared lexicographically using comparison of corresponding elements.": lists and tuples contain "items" not "elements", and an item-wise comparison should not be called "lexicographically" because that makes sense only when the items are characters. f) Ambiguity in "If not equal, the sequences are ordered the same as their first differing elements.": "Are ordered" could be interpreted (e.g. by non-native speakers) to mean that the sequence is changed to achieve that ordering, which is not the case of course. g) Editorial: In the list item about sets and froze sets, the example set {2,3} is not in example font. h) Omission: Range types are not covered. i) Omission: The section "Comparison of objects of differing types..." is silent about which built-in types support comparison across types (except for numeric types where that is covered). I think that should be explicitly listed for those built-in types that are also listed explicitly in the section about comparing objects of same type. I'll try to come up with a patch for 3.x, and once that is agreed, with one for 2.x. Andy ---------- nosy: +andymaier _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- versions: +Python 3.5 -Python 3.2, Python 3.3 Added file: http://bugs.python.org/file35843/issue12067-expressions_v3.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded issue12067-expressions_v3.diff for the 3.5 tip. Please review. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: docs@python -> rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded issue12067-expressions_v4.diff to improve the unicode footnote 3, and to revert to using the term "lexicographical" for sequences (after learning that it applies there as well). Also, this version was produced using "hg diff" to make it properly reviewable. Please review. ---------- Added file: http://bugs.python.org/file35845/issue12067-expressions_v4.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: PS: The v4 patch does not address comments f) and h) from msg222204, and it seems to me they do not need to be addressed. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Terry, I'd like to comment on your statement:
3. By default, == and /= compare identities. in msg148774.
What experiment lead you to that conclusion? Here is one that contradicts it (using cpython 3.4.1):
Is it possible, that your experiment got influenced by the optimization that attempts to reuse existing objects of immutable types? Like in this:
Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v5 of the patch. Changes: 1. The statement that comparison of different built-in types (always) raises TypeError, was too general. Changed to distinguish equal and order operators, as summarized by Ezio in items 3) and 4) of msg148760. 2. Ensured max line length of 80, in text areas affected by the patch. Andy ---------- versions: +Python 3.4 Added file: http://bugs.python.org/file35850/issue12067-expressions-py34_v5.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: It seems I still need to practice creating patches ... uploading v6 which should create a review link. No other changes. Sorry for that. Andy ---------- Added file: http://bugs.python.org/file35851/issue12067-expressions-py34_v6.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Another attempt. Really sorry... ---------- Added file: http://bugs.python.org/file35853/issue12067-expressions-py34_v7.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: In py3, *everything* is an instance of class object. This makes like simple than in 2.x. The default comparison rules are set by the rich comparison methods of object. 'By experiment' meant by experiments with instances of object, which use those default methods, rather than by inspection of the relevant .c source code. Instances of subclasses taht do not override the defaults would act the same. Here is what seem to be the default code, from object.c, do_compare. It verifies what I said (v, w are pointers, which represent identity): /* If neither object implements it, provide a sensible default for == and !=, but raise an exception for ordering. */ switch (op) { case Py_EQ: res = (v == w) ? Py_True : Py_False; break; case Py_NE: res = (v != w) ? Py_True : Py_False; break; default: /* XXX Special-case None so it doesn't show as NoneType() */ PyErr_Format(PyExc_TypeError, "unorderable types: %.100s() %s %.100s()", v->ob_type->tp_name, opstrings[op], w->ob_type->tp_name); return NULL; } Py_INCREF(res); return res; Subclasses can and ofter do override the default methods. In particular, the number subclasses compare by value, across number types. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I see. But I don't think it is a sensible default, as the source code states. The Python doc (v2 and v3) is quite consistent in stating that `==` compares the values of two objects, while `is` compares object identity. Having a default implementation on the object type that implements `==` by comparing object identity is not consistent with that. -> Can someone please elaborate what the reason for that is? -> Where is the discrepancy between the documentation of == and its default implementation on object documented? To me, a sensible default implementation for == on object would be (in Python): if v is w: return True; elif type(v) != type(w): return False else: raise ValueError("Equality cannot be determined in default implementation") Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- nosy: +benjamin.peterson, steven.daprano _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- nosy: +ethan.furman _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v8 of the patch for 3.4 and default. It reflects hopefully everything that was said in this issue thread, and on the python-dev mailing list (subject: == on object tests identity in 3.x), at least to the extent it was related to comparisons. Besides the doc changes it contained previously, it now also contains improvements for the test suite for comparisons (lib/test/test_compare.py). -> Please review both. Andy ---------- Added file: http://bugs.python.org/file35924/issue12067-expressions-py34_v8.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson added the comment: + In other words, the following expressions should have the same result: + + ``x == y`` and ``not x != y`` + + ``x < y`` and ``not x >= y`` + + ``x > y`` and ``not x <= y`` I think the second and third items here go too far: sets don't obey these rules, for example. Not all uses of comparisons need to force a total ordering. OTOH, you leave out a more fundamental relation, namely that `x < y` and `y > x` should ordinarily give the same result, as should `x <= y` and `y >= x`. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file35938/try_eq.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file35939/try_eq.out _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Mark: Both are good points! Would you add the cases from your second comment under "symmetry"? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: rhettinger -> _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v9 of the patch for 3.4 and default. It reflects Marc's comment, plus the result of the recent discussion on python-dev since v8 of th epatch, up to 2014-07-15 (subject: == on object tests identity in 3.x). -> Please review the patch. ---------- Added file: http://bugs.python.org/file35971/issue12067-expressions-py34_v9.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Berker Peksag <berker.peksag@gmail.com>: ---------- stage: needs patch -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- nosy: +vadmium _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Guido van Rossum added the comment: - This bug should discuss doc updates, not question the rules. - The rules have evolved over time and the docs stayed behind. - We should definitely update the 2.7 docs as well as the 3.4 and 3.5 (in development) docs. The 2.7 docs need to be different than the 3.x docs. - The language reference manual should clearly state the rules so that implementers can use them as guidelines for implementation. - There are several sets of relevant rules: (a) How is each operator translated into a series of lookups and method calls, etc. It's similar to other binary operators except that the reverse for __lt__ is __gt__ instead of __rlt__, and there's an extra rule that if __ne__ doesn't exist we compute __eq__ and take the opposite. (b) The default implementation (e.g. default == falls back to 'is', < raises TypeError). (c) The rules for built-in types, especially numbers (if there are still special cases that aren't explained by the __xx__ methods on the various numeric types). ---------- nosy: +gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: The point about “a != b” deferring to “not a.__eq__(b)” is not documented anywhere that I am aware of. In fact the opposite is currently documented at <https://docs.python.org/release/3.4.0/reference/datamodel.html#richcmpfuncs>, so maybe this needs to be fixed, one way or another. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Guido van Rossum added the comment: That's a pretty new feature. Someone probably forgot to clean up all the places where it was documented. On Sat, Sep 6, 2014 at 4:18 PM, Martin Panter <report@bugs.python.org> wrote:
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Just wanted to say that i will continue working on this, working in the comments made so far... Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Maybe it would be wise to split this task up and commit the bits that don’t need any more work. I think the existing patch might already solve Issue 22001. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: @Guido: Agree to all you said in your #msg226496. There is additional information about comparison in: - Tutorial (5.8. Comparing Sequences and Other Types), - Library Reference (5.3. Comparisons), - Language Reference (3.3.1. Basic customization) that needs to be reviewed in light of this patch. I'm just not sure I want to make this patch even larger as it is already, and tend to do that in a follow on issue and patch (unless directed otherwise). Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploading v10 of the patch, which addresses all review comments made on v9. There is one open question back to Martin Panter about which different types of byte sequences can be compared in Py 3.4. I also believe this patch addresses all of Issue 22001. Let me know if you find that that is not the case. If we continue to scope this patch to only the comparison chapter of the language reference, then I think we are done (see msg229229 about other places that need review and possibly updates). Please review the patch v10. ---------- Added file: http://bugs.python.org/file36896/issue12067-expressions-py34_v10.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Here is the delta between v9 and v10 of the patch, if people want to see just that. ---------- Added file: http://bugs.python.org/file36897/issue12067-expressions-py34_delta-v9-v10.di... _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: About the byte sequence comparisons, I wondered if it was misleading to say that a list(), tuple() or range() can only be compared to the same type, without mentioning that bytes() and bytearray() can be compared to each other. BTW just noticed you say range() supports lexicographical ordering, which I think is incorrect. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have addressed the comments by Jim Jewett, Martin Panter and of myself in a new version v11, which got posted. For the expression.rst doc file, this version of the patch has its diff sections in a logical order, so that the original text and the patched text are close by each other. Please review. ---------- Added file: http://bugs.python.org/file36920/issue12067-expressions-py34_v11.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I also made sure in both files that the line length of any changed or new lines is max 80. Sorry if that creates extra changes when looking at deltas between change sets. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have posted v12 of the patch, which addresses all comments since v11. This Python 3.4 patch can be applied to the "default" (3.5 dev) branch as well. I will start working on a similar patch for Python 2.7 now. ---------- Added file: http://bugs.python.org/file36978/issue12067-expressions-py34_v12.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Issue 4395 is already open for the != delegating to __eq__ issue that Guido pointed out earlier. Yet another issue that this doc patch should solve: Issue 22000. I am posting v13 of the patch that works with the current “default” (3.5) branch. Minor modifications to the documentation part: * Undo some paragraph reflowing to reduce the size of the diff hunks * Use math.nan instead of nan = float('NaN') * Tweaked the new subheadings to better differentiate “Value comparisons” (which this bug is concerned with) from “Identity comparisons” Changes I made to the added test cases: * Drop Python 3 version check * Use TestCase.subTest() so it’s easier to see everything that [is] was failing * Merge some tests for != and fix changed expectations due to Issue 21408 being fixed * Drop _inst_str() and __str__(), apparently unused It would be nice to see at least the documentation part reviewed and committed, even if the tests require more work. (After having to hack the tests to get them working, I might point out some odd bits in the code review.) Though I guess they are just tests, so it doesn’t matter so much as long as they pass. ---------- Added file: http://bugs.python.org/file38183/issue12067-expressions-py3.5_v13.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file38303/issue12067-expressions-py3.5_v14.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have posted v14 of the patch (for the 3.5 'default' branch), based on Martin's v13. v14 addresses all comments Martin made, as described in my responses to them (see patch set 10). On Issue 4395: That issue should be pursued in addition to this issue; it seems Martin's patch for it is complementary to the patch for this issue here. On Issue 22000: I agree that that issue is addressed by the patch for this issue here. All: Please review the v14 patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Ethan Furman <ethan@stoneleaf.us>: ---------- nosy: -ethan.furman _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Patch v15. No doc changes, but I refactored the test code: * Manually merged with recent changes * Separate assert_equality_only() and assert_total_order() test methods. Hopefully this is a bit simpler for people to understand and review, and avoids suggesting that partial ordering is tested. * Dropped subTest() instances with identical parameters. The basic stack trace can already distinguish these. * Eliminated is_value_comparable(); remove the ”meth” parameters from the call sites instead ---------- versions: +Python 3.6 Added file: http://bugs.python.org/file39970/issue12067-expressions-py3.5_v15.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Berker Peksag added the comment: I think we can commit documentation and tests separately. I just did a quick review of the test changes and I will add some review comments later (sorry, lack of time :)). ---------- nosy: +berker.peksag _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: I can split out a documentation-only patch if it would help get that committed. In the meantime, patch v16 includes some fixups to comments etc in the test code that I missed myself. ---------- assignee: -> docs@python components: +Tests Added file: http://bugs.python.org/file40052/issue12067-expressions-py3.6_v16.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Roundup Robot added the comment: New changeset 1fc049e5ec14 by Martin Panter in branch '3.4': Issue #12067: Rewrite Comparisons section in the language reference https://hg.python.org/cpython/rev/1fc049e5ec14 New changeset b6698c00265b by Martin Panter in branch '3.5': Issue #12067: Merge comparisons doc from 3.4 into 3.5 https://hg.python.org/cpython/rev/b6698c00265b New changeset 294b8a7957e9 by Martin Panter in branch 'default': Issue #12067: Merge comparisons doc from 3.5 https://hg.python.org/cpython/rev/294b8a7957e9 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: I committed the changes to expressions.rst for 3.4+. That still leaves the changes to test_compare.py, and possibly changes for 2.7. Andy: In msg229721 you mentioned a potential 2.7 patch. Did you get anywhere with that? Even if it is only half finished, someone else may be able to keep working on it. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- dependencies: +Wrong documentation (Language Ref) for unicode and str comparison _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Here is a port of the documentation to Python 2. Main differences: * Default rules for order comparisons are different * Not all kinds of objects inherit from object() * str(), unicode() compatibility * xrange() only seems to have default comparability * NAN, “binary sequences” and sets not listed ---------- Added file: http://bugs.python.org/file46372/expressions-py2.7.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Updated patch for 2.7, which I plan to commit soon. Corresponding Py 3 patch coming soon. ---------- Added file: http://bugs.python.org/file46398/expressions-py2.7_v17.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: Added file: http://bugs.python.org/file46399/expressions-py3.7_v17.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Roundup Robot added the comment: New changeset 8c9a86aa222e by Martin Panter in branch '3.5': Issue #12067: Recommend that hash and equality be consistent https://hg.python.org/cpython/rev/8c9a86aa222e New changeset 9702c5f08df1 by Martin Panter in branch '3.6': Issues #12067: Merge hash recommendation from 3.5 https://hg.python.org/cpython/rev/9702c5f08df1 New changeset 9dbb7bbc1449 by Martin Panter in branch 'default': Issues #12067: Merge hash recommendation from 3.6 https://hg.python.org/cpython/rev/9dbb7bbc1449 New changeset 8a9904c5cb1d by Martin Panter in branch '2.7': Issue #12067: Rewrite Comparisons section in the language reference https://hg.python.org/cpython/rev/8a9904c5cb1d ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Cheryl Sabella added the comment: It appears all the patches for this issue have been applied. Is the only open item the changes to test_compare? ---------- nosy: +csabella _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Yes I think I committed all the documentation. Someone needs to decide whether to use Andy’s tests as they are, or perhaps modify or drop some or all of them. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Guido van Rossum <guido@python.org>: ---------- nosy: -gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Cheryl Sabella <chekat2@gmail.com>: ---------- pull_requests: +3238 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Cheryl Sabella added the comment: I've created a PR for the changes to test_compare from v16 of the patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Chris Rebert <pybugs@rebertia.com>: ---------- nosy: +cvrebert _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Ezio Melotti <ezio.melotti@gmail.com> added the comment: Can you provide a patch? ---------- nosy: +ezio.melotti _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson <dickinsm@gmail.com> added the comment: [Docs] "If both are numbers, they are converted to a common type." [Terry] "In any case, I think it is only true for built-in number types," It's not even true for built-in number types. When comparing an int with a float, it's definitely *not* the case that the int is converted to a float and the floats compared. And that's for good reason: the int -> float conversion is lossy for large integers, so if int <-> float comparisons just converted the int to a float before comparing, we'd have (for example):
10**16 == 1e16 == 10**16 + 1
leading to broken transitivity of equality, and strange dict and set behaviour. So int <-> float comparisons do a complicated dance under the hood to compare the exact numerical values of the two objects and produce the correct result. I'm not sure what the intent of the original sentence was, or how to reword it. The key point is simply that it *is* possible to compare an int with a float, and that the result is sensible, based on numeric values. ---------- nosy: +mark.dickinson _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Ezio Melotti <ezio.melotti@gmail.com> added the comment: Would it be ok to state that: 1) <, >, ==, >=, <=, and != compare the values of two objects; 2) the two objects don't necessarily have to be of the same type; 3) with == and !=, objects of different types compare unequal, unless they define a specific __eq__ and/or __ne__; 4) with <, >, <=, and >=, the comparison of objects of different types raises a TypeError, unless they define specific __lt__, __gt__, __le__, and __ge__; 5) some built-in types define these operations, so it's possible to compare e.g. int and floats; This should summarize the possible behaviors. There's no reason IMHO to expose implementation details and to special case built-in types (unless their comparison is actually different and doesn't depend on __eq__, __ne__, etc.). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: In Python 3, where all classes inherit from object, the default rules are, by experiment, (which someone can verify from the code) simpler than you stated. 3. By default, == and /= compare identities. 4. By default, order comparisons raise TypeError. ob <= ob raises even though ob == ob because ob is ob. I am not sure of the method look-up rules for rich comparisons, but perhaps the following are true: 3) with == and !=, an object is equal to itself and different objects (a is not b) compare unequal, unless the class of the first define a specific __eq__ and __ne__; 4) with <, >, <=, and >=, comparison raises a TypeError, unless the class of the first object defines specific __lt__, __gt__, __le__, and __ge__, or the class of the second defines the reflected method (__ge__ reflects __lt__, etcetera); What is not clear to me is whether the reflected method is called if the first raises TypeError. The special method names doc (reference 3.3) says "A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments. ... There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection." Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section. I think point 5 should say a bit more: builtin numbers compare as expected, even when of different types; builtin sequences compare lexicographically. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Mike Hoy <mhoy09@gmail.com>: ---------- nosy: +mikehoy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- type: -> enhancement _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: http://bugs.python.org/issue15997 is this issue related to what Terry has mentioned: Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section. I am working on this patch and need confirmation as to whether or not this has to be included in my patch. I'm not clear on it so I may just pass on making a patch if it is required for this issue. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: After further thought: This section is about the syntax operators, not the special methods. The syntax operators never evaluate to NotImplemented as they (apparently) interpret its return from a special method the same as a raising of TypeError, and always raise TypeError when neither the op or its reflection is supported. So there should be no mention of NotImplemented here. Just a reference to 3.3. #15997 is related to my 'wonder' but not directly relevant to a patch for this. Please submit a draft patch when you have one. I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method, at least for a couple of cases. (And same seems true for arithmetic ops too.) class C(): def __ge__(self, other): # print("in C.__ge__", end='') return True def __add__(self, other): return 44 __radd__ = __add__ class O(): def __le__(self, other): # print ("in O.__le__") return NotImplemented def __add__(self, other): return NotImplemented c = C() o = O() ob = object() print(c >= o, o <= c, ob <= c) # True True True # print(ob <= ob) # raises TypeError print(c + o, o + c, ob + c) # 44 44 44 # print(ob + ob) # raises TypeError # print(ob >= o) # with O.__le__ print uncommented # in O.__le__ # so non-implemented reflected o <= ob *is* called # TypeError: unorderable types: object() >= O() ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: I've attempted to incorporate both Terry's and Ezio's suggestions. Here is a patch to get started with. There is a section that has been deleted. Patch uploaded. ---------- keywords: +patch Added file: http://bugs.python.org/file27256/issue12067-expressions.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Chris Jerdonek added the comment: Some minor comments: -The operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the +``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the values of two I think it reads better to start a sentence (and in this case a paragraph) with a word rather than a symbol. -values of two objects. The objects need not have the same type. If both are +objects. The two objects don't necessarily have to be of the same type. With The replacement sentence seems wordier to me. +(:meth:`__ge__()` reflects :meth:`__lt__()`, etcetera). Builtin numbers compare +as expected, even when of different types. Builtin sequences compare "Built-in" is hyphenated in the docs. See, for example, here: http://docs.python.org/dev/library/functions.html ---------- nosy: +cjerdonek _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson added the comment:
I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method
Are you sure? raise TypeError *should* result in the operation being abandoned, with the reflected operation not tried. Python 3.3.0rc2+ (default:3504cbb3e1d8, Sep 20 2012, 22:08:44) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information.
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: You are right, I did not look deep enough. I was fooled by the conversion of NotImplemented, returned from object.__le__, etc, to TypeError. Sorry for that noise. For comparison and arithmetic, the actual alternative to defining a function that returns NotImplemented seems to be to not define it at all. class C(): def __ge__(self, other): return True def __add__(self, other): return 44 __radd__ = __add__ class O(): pass # removed NotImplemented defs c = C() o = O() print(c >= o, o <= c) # True True print(c + o, o + c) # 44 44 (I looked at the codes for binary_op1 in abstract.c and do_richcompare in object.c and do not yet see any effective difference between not defined and a NotImplemented return.) I'll take a look at the patch later. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: Changed patch to include suggestions by Chris Jerdonek. http://bugs.python.org/issue12067#msg170953 ---------- hgrepos: +153 Added file: http://bugs.python.org/file27442/issue12067-expressions_v2.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Mike Hoy <mhoy09@gmail.com>: ---------- hgrepos: -153 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mike Hoy added the comment: Considering that the docs have changed does this issue still need to be open? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Hi, I'd like to revive this issue. IMHO, the changes in issue12067-expressions_v2.diff go too far. I don't think that deleting the entire section about the details of comparing objects of the same type makes sense. I agree with Terry's statement in msg170936 that the chapter is about the operators and not about the ways to customize them, so some of what the patch introduces in that area should not be introduced. So far, that means that I'm pretty much against that patch entirely... Having said that, I do believe that there are still issues: 1. both the 2.7 and 3.x sections about the comparison operators are sufficiently convoluted and could be organized better by grouping the various statements that are made, into categories like this: - comparisons involving objects of user-defined types - comparing objects of same built-in type - comparing objects of differing built-in type 2. There are still some errors, ambiguities and omissions that need to be fixed. For example, in the 3.x version: a) omission about treatment of NaN for numbers of different type (could in theory be implied from the statement "are converted to same type", but that statement is problematic as was pointed out in Mark's comment on this issue). b) Amgiguous statement "Most numeric types [of same type] can be compared with one another.". I think what is true is that all built-in numeric types (including Fraction and Decimal) compare mathematically correct in 3.x, and that the only non-support is that complex numbers are not considered orderable and an attempt to use an ordering operator raises TypeError. c) Ambiguous statement "When cross-type comparison is not supported, the comparison method returns NotImplemented.". If this is about the customization methods, it should not be here, but there. Here, it is relevant that a TypeError is raised when using the operator. d) Terminology in "Bytes objects are compared lexicographically using the numeric values of their elements.": Chapter [4.8.1. Bytes] defines bytes objects as immutable sequences of single bytes (not elements). e) Terminology in "Tuples and lists are compared lexicographically using comparison of corresponding elements.": lists and tuples contain "items" not "elements", and an item-wise comparison should not be called "lexicographically" because that makes sense only when the items are characters. f) Ambiguity in "If not equal, the sequences are ordered the same as their first differing elements.": "Are ordered" could be interpreted (e.g. by non-native speakers) to mean that the sequence is changed to achieve that ordering, which is not the case of course. g) Editorial: In the list item about sets and froze sets, the example set {2,3} is not in example font. h) Omission: Range types are not covered. i) Omission: The section "Comparison of objects of differing types..." is silent about which built-in types support comparison across types (except for numeric types where that is covered). I think that should be explicitly listed for those built-in types that are also listed explicitly in the section about comparing objects of same type. I'll try to come up with a patch for 3.x, and once that is agreed, with one for 2.x. Andy ---------- nosy: +andymaier _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- versions: +Python 3.5 -Python 3.2, Python 3.3 Added file: http://bugs.python.org/file35843/issue12067-expressions_v3.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded issue12067-expressions_v3.diff for the 3.5 tip. Please review. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: docs@python -> rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded issue12067-expressions_v4.diff to improve the unicode footnote 3, and to revert to using the term "lexicographical" for sequences (after learning that it applies there as well). Also, this version was produced using "hg diff" to make it properly reviewable. Please review. ---------- Added file: http://bugs.python.org/file35845/issue12067-expressions_v4.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: PS: The v4 patch does not address comments f) and h) from msg222204, and it seems to me they do not need to be addressed. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Terry, I'd like to comment on your statement:
3. By default, == and /= compare identities. in msg148774.
What experiment lead you to that conclusion? Here is one that contradicts it (using cpython 3.4.1):
Is it possible, that your experiment got influenced by the optimization that attempts to reuse existing objects of immutable types? Like in this:
Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v5 of the patch. Changes: 1. The statement that comparison of different built-in types (always) raises TypeError, was too general. Changed to distinguish equal and order operators, as summarized by Ezio in items 3) and 4) of msg148760. 2. Ensured max line length of 80, in text areas affected by the patch. Andy ---------- versions: +Python 3.4 Added file: http://bugs.python.org/file35850/issue12067-expressions-py34_v5.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: It seems I still need to practice creating patches ... uploading v6 which should create a review link. No other changes. Sorry for that. Andy ---------- Added file: http://bugs.python.org/file35851/issue12067-expressions-py34_v6.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Another attempt. Really sorry... ---------- Added file: http://bugs.python.org/file35853/issue12067-expressions-py34_v7.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Terry J. Reedy added the comment: In py3, *everything* is an instance of class object. This makes like simple than in 2.x. The default comparison rules are set by the rich comparison methods of object. 'By experiment' meant by experiments with instances of object, which use those default methods, rather than by inspection of the relevant .c source code. Instances of subclasses taht do not override the defaults would act the same. Here is what seem to be the default code, from object.c, do_compare. It verifies what I said (v, w are pointers, which represent identity): /* If neither object implements it, provide a sensible default for == and !=, but raise an exception for ordering. */ switch (op) { case Py_EQ: res = (v == w) ? Py_True : Py_False; break; case Py_NE: res = (v != w) ? Py_True : Py_False; break; default: /* XXX Special-case None so it doesn't show as NoneType() */ PyErr_Format(PyExc_TypeError, "unorderable types: %.100s() %s %.100s()", v->ob_type->tp_name, opstrings[op], w->ob_type->tp_name); return NULL; } Py_INCREF(res); return res; Subclasses can and ofter do override the default methods. In particular, the number subclasses compare by value, across number types. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I see. But I don't think it is a sensible default, as the source code states. The Python doc (v2 and v3) is quite consistent in stating that `==` compares the values of two objects, while `is` compares object identity. Having a default implementation on the object type that implements `==` by comparing object identity is not consistent with that. -> Can someone please elaborate what the reason for that is? -> Where is the discrepancy between the documentation of == and its default implementation on object documented? To me, a sensible default implementation for == on object would be (in Python): if v is w: return True; elif type(v) != type(w): return False else: raise ValueError("Equality cannot be determined in default implementation") Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- nosy: +benjamin.peterson, steven.daprano _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: ---------- nosy: +ethan.furman _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v8 of the patch for 3.4 and default. It reflects hopefully everything that was said in this issue thread, and on the python-dev mailing list (subject: == on object tests identity in 3.x), at least to the extent it was related to comparisons. Besides the doc changes it contained previously, it now also contains improvements for the test suite for comparisons (lib/test/test_compare.py). -> Please review both. Andy ---------- Added file: http://bugs.python.org/file35924/issue12067-expressions-py34_v8.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Mark Dickinson added the comment: + In other words, the following expressions should have the same result: + + ``x == y`` and ``not x != y`` + + ``x < y`` and ``not x >= y`` + + ``x > y`` and ``not x <= y`` I think the second and third items here go too far: sets don't obey these rules, for example. Not all uses of comparisons need to force a total ordering. OTOH, you leave out a more fundamental relation, namely that `x < y` and `y > x` should ordinarily give the same result, as should `x <= y` and `y >= x`. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file35938/try_eq.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file35939/try_eq.out _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Mark: Both are good points! Would you add the cases from your second comment under "symmetry"? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: rhettinger -> _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploaded v9 of the patch for 3.4 and default. It reflects Marc's comment, plus the result of the recent discussion on python-dev since v8 of th epatch, up to 2014-07-15 (subject: == on object tests identity in 3.x). -> Please review the patch. ---------- Added file: http://bugs.python.org/file35971/issue12067-expressions-py34_v9.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Berker Peksag <berker.peksag@gmail.com>: ---------- stage: needs patch -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- nosy: +vadmium _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Guido van Rossum added the comment: - This bug should discuss doc updates, not question the rules. - The rules have evolved over time and the docs stayed behind. - We should definitely update the 2.7 docs as well as the 3.4 and 3.5 (in development) docs. The 2.7 docs need to be different than the 3.x docs. - The language reference manual should clearly state the rules so that implementers can use them as guidelines for implementation. - There are several sets of relevant rules: (a) How is each operator translated into a series of lookups and method calls, etc. It's similar to other binary operators except that the reverse for __lt__ is __gt__ instead of __rlt__, and there's an extra rule that if __ne__ doesn't exist we compute __eq__ and take the opposite. (b) The default implementation (e.g. default == falls back to 'is', < raises TypeError). (c) The rules for built-in types, especially numbers (if there are still special cases that aren't explained by the __xx__ methods on the various numeric types). ---------- nosy: +gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: The point about “a != b” deferring to “not a.__eq__(b)” is not documented anywhere that I am aware of. In fact the opposite is currently documented at <https://docs.python.org/release/3.4.0/reference/datamodel.html#richcmpfuncs>, so maybe this needs to be fixed, one way or another. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Guido van Rossum added the comment: That's a pretty new feature. Someone probably forgot to clean up all the places where it was documented. On Sat, Sep 6, 2014 at 4:18 PM, Martin Panter <report@bugs.python.org> wrote:
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Just wanted to say that i will continue working on this, working in the comments made so far... Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Maybe it would be wise to split this task up and commit the bits that don’t need any more work. I think the existing patch might already solve Issue 22001. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: @Guido: Agree to all you said in your #msg226496. There is additional information about comparison in: - Tutorial (5.8. Comparing Sequences and Other Types), - Library Reference (5.3. Comparisons), - Language Reference (3.3.1. Basic customization) that needs to be reviewed in light of this patch. I'm just not sure I want to make this patch even larger as it is already, and tend to do that in a follow on issue and patch (unless directed otherwise). Andy ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Uploading v10 of the patch, which addresses all review comments made on v9. There is one open question back to Martin Panter about which different types of byte sequences can be compared in Py 3.4. I also believe this patch addresses all of Issue 22001. Let me know if you find that that is not the case. If we continue to scope this patch to only the comparison chapter of the language reference, then I think we are done (see msg229229 about other places that need review and possibly updates). Please review the patch v10. ---------- Added file: http://bugs.python.org/file36896/issue12067-expressions-py34_v10.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: Here is the delta between v9 and v10 of the patch, if people want to see just that. ---------- Added file: http://bugs.python.org/file36897/issue12067-expressions-py34_delta-v9-v10.di... _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: About the byte sequence comparisons, I wondered if it was misleading to say that a list(), tuple() or range() can only be compared to the same type, without mentioning that bytes() and bytearray() can be compared to each other. BTW just noticed you say range() supports lexicographical ordering, which I think is incorrect. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have addressed the comments by Jim Jewett, Martin Panter and of myself in a new version v11, which got posted. For the expression.rst doc file, this version of the patch has its diff sections in a logical order, so that the original text and the patched text are close by each other. Please review. ---------- Added file: http://bugs.python.org/file36920/issue12067-expressions-py34_v11.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I also made sure in both files that the line length of any changed or new lines is max 80. Sorry if that creates extra changes when looking at deltas between change sets. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have posted v12 of the patch, which addresses all comments since v11. This Python 3.4 patch can be applied to the "default" (3.5 dev) branch as well. I will start working on a similar patch for Python 2.7 now. ---------- Added file: http://bugs.python.org/file36978/issue12067-expressions-py34_v12.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Issue 4395 is already open for the != delegating to __eq__ issue that Guido pointed out earlier. Yet another issue that this doc patch should solve: Issue 22000. I am posting v13 of the patch that works with the current “default” (3.5) branch. Minor modifications to the documentation part: * Undo some paragraph reflowing to reduce the size of the diff hunks * Use math.nan instead of nan = float('NaN') * Tweaked the new subheadings to better differentiate “Value comparisons” (which this bug is concerned with) from “Identity comparisons” Changes I made to the added test cases: * Drop Python 3 version check * Use TestCase.subTest() so it’s easier to see everything that [is] was failing * Merge some tests for != and fix changed expectations due to Issue 21408 being fixed * Drop _inst_str() and __str__(), apparently unused It would be nice to see at least the documentation part reviewed and committed, even if the tests require more work. (After having to hack the tests to get them working, I might point out some odd bits in the code review.) Though I guess they are just tests, so it doesn’t matter so much as long as they pass. ---------- Added file: http://bugs.python.org/file38183/issue12067-expressions-py3.5_v13.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Andy Maier <andreas.r.maier@gmx.de>: Added file: http://bugs.python.org/file38303/issue12067-expressions-py3.5_v14.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Andy Maier added the comment: I have posted v14 of the patch (for the 3.5 'default' branch), based on Martin's v13. v14 addresses all comments Martin made, as described in my responses to them (see patch set 10). On Issue 4395: That issue should be pursued in addition to this issue; it seems Martin's patch for it is complementary to the patch for this issue here. On Issue 22000: I agree that that issue is addressed by the patch for this issue here. All: Please review the v14 patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Ethan Furman <ethan@stoneleaf.us>: ---------- nosy: -ethan.furman _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Patch v15. No doc changes, but I refactored the test code: * Manually merged with recent changes * Separate assert_equality_only() and assert_total_order() test methods. Hopefully this is a bit simpler for people to understand and review, and avoids suggesting that partial ordering is tested. * Dropped subTest() instances with identical parameters. The basic stack trace can already distinguish these. * Eliminated is_value_comparable(); remove the ”meth” parameters from the call sites instead ---------- versions: +Python 3.6 Added file: http://bugs.python.org/file39970/issue12067-expressions-py3.5_v15.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Berker Peksag added the comment: I think we can commit documentation and tests separately. I just did a quick review of the test changes and I will add some review comments later (sorry, lack of time :)). ---------- nosy: +berker.peksag _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: I can split out a documentation-only patch if it would help get that committed. In the meantime, patch v16 includes some fixups to comments etc in the test code that I missed myself. ---------- assignee: -> docs@python components: +Tests Added file: http://bugs.python.org/file40052/issue12067-expressions-py3.6_v16.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Roundup Robot added the comment: New changeset 1fc049e5ec14 by Martin Panter in branch '3.4': Issue #12067: Rewrite Comparisons section in the language reference https://hg.python.org/cpython/rev/1fc049e5ec14 New changeset b6698c00265b by Martin Panter in branch '3.5': Issue #12067: Merge comparisons doc from 3.4 into 3.5 https://hg.python.org/cpython/rev/b6698c00265b New changeset 294b8a7957e9 by Martin Panter in branch 'default': Issue #12067: Merge comparisons doc from 3.5 https://hg.python.org/cpython/rev/294b8a7957e9 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: I committed the changes to expressions.rst for 3.4+. That still leaves the changes to test_compare.py, and possibly changes for 2.7. Andy: In msg229721 you mentioned a potential 2.7 patch. Did you get anywhere with that? Even if it is only half finished, someone else may be able to keep working on it. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- dependencies: +Wrong documentation (Language Ref) for unicode and str comparison _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Here is a port of the documentation to Python 2. Main differences: * Default rules for order comparisons are different * Not all kinds of objects inherit from object() * str(), unicode() compatibility * xrange() only seems to have default comparability * NAN, “binary sequences” and sets not listed ---------- Added file: http://bugs.python.org/file46372/expressions-py2.7.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Updated patch for 2.7, which I plan to commit soon. Corresponding Py 3 patch coming soon. ---------- Added file: http://bugs.python.org/file46398/expressions-py2.7_v17.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: Added file: http://bugs.python.org/file46399/expressions-py3.7_v17.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Roundup Robot added the comment: New changeset 8c9a86aa222e by Martin Panter in branch '3.5': Issue #12067: Recommend that hash and equality be consistent https://hg.python.org/cpython/rev/8c9a86aa222e New changeset 9702c5f08df1 by Martin Panter in branch '3.6': Issues #12067: Merge hash recommendation from 3.5 https://hg.python.org/cpython/rev/9702c5f08df1 New changeset 9dbb7bbc1449 by Martin Panter in branch 'default': Issues #12067: Merge hash recommendation from 3.6 https://hg.python.org/cpython/rev/9dbb7bbc1449 New changeset 8a9904c5cb1d by Martin Panter in branch '2.7': Issue #12067: Rewrite Comparisons section in the language reference https://hg.python.org/cpython/rev/8a9904c5cb1d ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Cheryl Sabella added the comment: It appears all the patches for this issue have been applied. Is the only open item the changes to test_compare? ---------- nosy: +csabella _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Martin Panter added the comment: Yes I think I committed all the documentation. Someone needs to decide whether to use Andy’s tests as they are, or perhaps modify or drop some or all of them. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Guido van Rossum <guido@python.org>: ---------- nosy: -gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Changes by Cheryl Sabella <chekat2@gmail.com>: ---------- pull_requests: +3238 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
Cheryl Sabella added the comment: I've created a PR for the changes to test_compare from v16 of the patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12067> _______________________________________
participants (15)
-
Andy Maier -
Berker Peksag
-
Cheryl Sabella
-
Chris Jerdonek
-
Chris Rebert
-
Ethan Furman
-
Ezio Melotti
-
Guido van Rossum
-
Mark Dickinson
-
Martin Panter
-
Mike Hoy
-
Raymond Hettinger
-
Roundup Robot
-
Serhiy Storchaka
-
Terry J. Reedy