[issue16701] Docs missing the behavior of += (in-place add) for lists.
New submission from Ashwini Chaudhary: I think the python docs are missing the behavior of += for lists. It actually calls list.extend() but can't find that anywhere in docs expect in source code, http://hg.python.org/cpython/file/2d2d4807a3ed/Objects/listobject.c#l892. ---------- assignee: docs@python components: Documentation messages: 177627 nosy: docs@python, montysinngh priority: normal severity: normal status: open title: Docs missing the behavior of += (in-place add) for lists. type: enhancement versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: Well, it is effectively documented by the text here: http://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-st... since "a + b" is logically equivalent to a.extend(b) when a is being updated "in-place". The fact that it is in fact implemented using extend is an implementation detail. That said, it would be logical to add an entry for the augmented assignment to the table here: http://docs.python.org/3/library/stdtypes.html#mutable-sequence-types There also may be other places in that chapter where augmented assignment deserves mention. ---------- nosy: +r.david.murray versions: -Python 3.1, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- nosy: +ezio.melotti stage: -> needs patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
benrg added the comment: This is bizarre: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
x = y = [1, 2] x += [3] y [1, 2, 3] x = y = {1, 2} x -= {2} y {1}
Since when has this been standard behavior? The documentation says: "An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead." What is "when possible" supposed to mean here? I always thought it meant "when there are known to be no other references to the object". If op= is always destructive on lists and sets, then "where possible" needs to be changed to "always" and a prominent warning added, like "WARNING: X OP= EXPR DOES NOT BEHAVE EVEN REMOTELY LIKE X = X OP EXPR IN PYTHON WHEN X IS A MUTABLE OBJECT, IN STARK CONTRAST TO EVERY OTHER LANGUAGE WITH A SIMILAR SYNTAX." ---------- nosy: +benrg _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Ezio Melotti added the comment:
What is "when possible" supposed to mean here?
Generally it means "when the object is mutable":
l = [1,2,3] id(l) 3074713484 l += [4] id(l) 3074713484 t = (1,2,3) id(t) 3074704004 t += (4,) id(t) 3075304860
Tuples are not mutable, so it's not possible to modify them in place, and a new tuple needs to be created. Note that while most mutable objects in the stdlib that support += do indeed modify the object rather than creating a new one, I don't think this is strictly required. IOW that paragraph is already warning you that (with mutable objects) the object might be reused, depending on the implementation. Maybe this should be clarified? (IIRC in CPython it could be possible that in some situations an immutable object still has the same id after an augmented assignment, but, if it really happens, it is an implementation detail and shouldn't affect semantics.) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: If you really want to freak out, try this:
x = ([],) x[0] += [1] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment x ([1],)
but to answer your question, it has *always* worked that way, from the time augmented assignment was introduced (see, eg, issue 1306777, which was reported against python 2.4). Remember, Python names refer to pointers to objects, they are not variables in the sense that other languages have variables. Guido resisted augmented assignment for a long time. These confusions speak to why. As far as I know Ezio is correct, "when possible" means "when the target is mutable". The documentation should probably be clarified on that point. I'm not sure it is practical to let whether or not the target is mutated be an implementation detail. IMO the behavior must be clearly defined for each type that is built in to Python. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Ezio Melotti added the comment: To clarify, with "depends on the implementation" I meant the way a particular class is implemented (i.e. a class might decide to return a new object even if it's mutable). The behavior of built-in types is well defined and should be the same across all the Python implementations. Regarding the comment about immutable types, it's something specific to CPython (I don't remember the specific details though, so I might be wrong), and somewhat similar to:
'a'*20 is 'a'*20 True 'a'*25 is 'a'*25 False This shouldn't be a problem though, so if you e.g. do "x = y = immutableobj; y += 1", 'x' should never be affected.
---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
benrg added the comment:
As far as I know Ezio is correct, "when possible" means "when the target is mutable". The documentation should probably be clarified on that point.
Yes, it needs to be made very, very clear in the documentation. As I said, I'm not aware of any other language in which var op= expr does not mean the same thing as var = var op expr. I'm actually amazed that neither of you recognize the weirdness of this behavior (and even more amazed that GvR apparently didn't). I'm an experienced professional programmer, and I dutifully read the official documentation cover to cover when I started programming in Python, and I interpreted this paragraph wrongly, because I interpreted it in the only way that made sense given the meaning of these operators in every other language that has them. Python is designed to be unsurprising; constructs generally mean what it looks like they mean. You need to explain this unique feature of Python in terms so clear that it can't possibly be mistaken for the behavior of all of the other languages.
Remember, Python names refer to pointers to objects, they are not variables in the sense that other languages have variables.
That has nothing to do with this. Yes, in Python (and Java and Javascript and many other languages) all objects live on the heap, local variables are not first-class objects, and var = expr is a special form. That doesn't change the fact that in all of those other languages, var += expr means var = var + expr. In C++ local variables are first-class objects and var += expr means var.operator+=(expr) or operator+=(var, expr), and this normally modifies the thing on the left in a way that's visible through references. But in C++, var = var + expr also modifies the thing on the left, in the same way. In Python and Java and Javascript and ..., var = value never visibly mutates any heap object, and neither does var = var + value (in any library that defines a sane + operator), and therefore neither should var += value (again, in any sanely designed library). And it doesn't. Except in Python. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Ezio Melotti added the comment:
Python is designed to be unsurprising; constructs generally mean what it looks like they mean.
AFAIK in C "x += 1" is equivalent to "x++", and both are semantically more about incrementing (mutating) the value of x than about creating a new value that gets assigned to x. Likewise it seems to me more natural to interpret "x += y" as "add the value of y to the object x" than "add x and y together and save the result in x". Clearly if you are used to other languages with different semantics you might expect a different behavior, but you could say the same about the fact that int/int gives float on Python 3: it's surprising if you are used to other languages like C, but otherwise it's more natural.
I interpreted this paragraph wrongly, because I interpreted it in the only way that made sense given the meaning of these operators in every other language that has them.
It seems to me that the documentation doesn't leave much room for interpretation regarding the fact that the object is mutated in place; the only problem is that it doesn't specify clearly what are the objects that do this. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
benrg added the comment:
AFAIK in C "x += 1" is equivalent to "x++", and both are semantically more about incrementing (mutating) the value of x than about creating a new value that gets assigned to x. Likewise it seems to me more natural to interpret "x += y" as "add the value of y to the object x" than "add x and y together and save the result in x".
Look, it's very simple: in C, ++x and x += 1 and x = x + 1 all mean the same thing. You can argue about how to describe the thing that they do, but there's only one thing to describe. Likewise, in every other language that borrows the op= syntax from C, it is a shorthand for the expanded version with the bare operator. As far as I know, Python is the only exception. If you know of another exception please say so.
x = ([],) x[0] += [1] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment x ([1],)
I actually knew about this. It's an understandably difficult corner case, since the exception is raised after __iadd__ returns, so there's no chance for it to roll back its changes. At least, I thought it was a difficult corner case back when I thought the in-place update was a mere optimization. But if += really means .extend() on lists, this should not raise an exception at all. In fact there's no sense in having __iadd__ return a value that gets assigned anywhere, since mutable objects always mutate and return themselves and immutable objects don't define __iadd__. It looks like the interface was designed with the standard semantics in mind but the implementation did something different, leaving a vestigial assignment that's always a no-op. What a disaster. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Terry J. Reedy added the comment: Augmented assignment confuses enough people that I think we can improve the doc. In #21358 I suggest an augmented version of the previous claim, about evaluation just once. I think something here is needed perhaps even more. I have not decided what just yet. ---------- nosy: +terry.reedy versions: +Python 3.5 -Python 2.6, Python 3.2, Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Brendan Barnwell added the comment: This needs to be fixed. The documentation for the behavior of += on lists needs to be with the documentation on lists. The existing, vague documentation that += works in-place "when possible" is insufficient. A central feature of Python is that the behavior of operators like + and += is overridable on a per-type basis. Hence, the Language Reference is not the appropriate place for describing the behavior of += on a particular type. The behavior of += on lists should be documented where the behavior of lists is documented (as, for instance, the behavior of + on lists already is), not where the syntax of += is documented. Someone just asked a question on StackOverflow about this (http://stackoverflow.com/questions/32657637/python-changing-variables-vs-arr...). It is embarrassing to have to tell people, "To know what += does on a type, you need to look at the documentation for that type. . . except that the documentation for the builtin types doesn't document what some operators do." ---------- nosy: +BrenBarn _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: I suggested updating the library reference in my first reply on this issue. No one has proposed a patch yet, though. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Martin Panter added the comment: Here is a patch documenting the += and *= mutable sequence operations. Please review my wording. These operations already seem to be tested, at least on the basic mutable sequences: see /Lib/test/list_tests.py, test_array, test_collections, test_bytes (tests bytearray). The only other places that I thought might be missing augmented assignment were for sets, but there is no problem there: <https://docs.python.org/dev/library/stdtypes.html#set.update>. However, there are other operations that I think may be missing from this page of the documentation. But it might be better to handle those in a separate bug report. Some of this could build off the work in Issue 12067. * Equality comparisons (mentioned for range and dict, but apparently not tuple, set, strings, etc) * Ordering comparisons (not supported for range) * min() and max() don’t really belong; maybe substitute with iter() ---------- keywords: +patch nosy: +martin.panter stage: needs patch -> patch review versions: +Python 3.6 Added file: http://bugs.python.org/file40552/seq-inplace.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: Wording looks ok...except that technically it is not that 'n' is an integer, it's that 'n' can play the role of an integer (ie: it has an __index__ method). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Martin Panter added the comment: New patch mentioning __index__() ---------- Added file: http://bugs.python.org/file40560/seq-inplace.v2.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: Something I missed on the first review: why did you change "the same as" to "usually the same as"? When is it different? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Martin Panter added the comment: The “s += t” operation assigns the result back to s. So it could involve an extra __setattr__()/__setitem__() call, or an exception trying to modify a tuple item, etc. The extend() and slice assignment versions don’t have this extra stage. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: Ah, good point. I'd say something like "for the most part" then instead of "usually", since what you describe always happens. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Martin Panter added the comment: “For the most part” works for me. Here is the patch. ---------- Added file: http://bugs.python.org/file40637/seq-inplace.v3.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
R. David Murray added the comment: Looks good to me. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- assignee: docs@python -> martin.panter nosy: +berker.peksag stage: patch review -> commit review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Roundup Robot added the comment: New changeset ec373d762213 by Martin Panter in branch '2.7': Issue #16701: Document += and *= for mutable sequences https://hg.python.org/cpython/rev/ec373d762213 New changeset f83db23bec7f by Martin Panter in branch '3.4': Issue #16701: Document += and *= for mutable sequences https://hg.python.org/cpython/rev/f83db23bec7f New changeset 6e43a3833293 by Martin Panter in branch '3.5': Issue #16701: Merge sequence docs from 3.4 into 3.5 https://hg.python.org/cpython/rev/6e43a3833293 New changeset a92466bf16cc by Martin Panter in branch 'default': Issue #16701: Merge sequence docs from 3.5 https://hg.python.org/cpython/rev/a92466bf16cc ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- resolution: -> fixed stage: commit review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue16701> _______________________________________
participants (8)
-
Ashwini Chaudhary -
benrg
-
Brendan Barnwell
-
Ezio Melotti
-
Martin Panter
-
R. David Murray
-
Roundup Robot
-
Terry J. Reedy