inplace operators and __setitem__
Hi, a general question. Consider: class A(list): def __setitem__(self, index, item): # do something with index and item return list.__setitem__(self, index, item) lst = A([1,set()]) lst[0] |= 1 lst[1] |= set([1]) Do we want lst.__setitem__ to be called in the second inplace assignment? A case where this matters is here: http://python.org/sf/1306777 Reinhold -- Mail address is perfectly valid!
On Sep 28, 2005, at 9:12 AM, Reinhold Birkenfeld wrote:
Hi,
a general question. Consider:
class A(list): def __setitem__(self, index, item): # do something with index and item return list.__setitem__(self, index, item)
lst = A([1,set()])
lst[0] |= 1
lst[1] |= set([1])
Do we want lst.__setitem__ to be called in the second inplace assignment?
Yes. Right now, you can roughly explain the behavior by stating that, after "x=a", "x |= y" is the same as "x = x | y", except that "a"'s value is undefined (it might have changed, or it might have not).
A case where this matters is here: http://python.org/sf/1306777
This confusion between modification of immutable types and modification of mutable types is why I feel that it's often best to simply avoid the inplace operators in favor of their explicit equivalents. In this case, set.update(). James
At 03:12 PM 9/28/2005 +0200, Reinhold Birkenfeld wrote:
Hi,
a general question. Consider:
class A(list): def __setitem__(self, index, item): # do something with index and item return list.__setitem__(self, index, item)
lst = A([1,set()])
lst[0] |= 1
lst[1] |= set([1])
Do we want lst.__setitem__ to be called in the second inplace assignment?
Yes. See: http://www.python.org/2.0/new-python.html#SECTION000700000000000000000 The purpose of the augmented assignment forms is to allow for the possibility that the item's __i*__ method may or may not exist, and may or may not return the same object. In the case where there is no __i*__ form, or it does not return the same object, the lvalue *must* be re-bound to the new value, or the semantics break.
A case where this matters is here: http://python.org/sf/1306777
I've closed it as invalid; the behavior is as-defined. In principle, there *could* be an optimization to avoid rebinding the lvalue in the case where the __i*__ form did return self. But using it for the purpose of allowing augmented assignment to tuple members seems dubious at best, and likely to create confusion about the mutability or lack thereof of tuples. IMO it's better to have augmented assignment to tuple members always fail, so that the code has to be a little more specific about its intent.
Phillip J. Eby wrote:
A case where this matters is here: http://python.org/sf/1306777
I've closed it as invalid; the behavior is as-defined.
In principle, there *could* be an optimization to avoid rebinding the lvalue in the case where the __i*__ form did return self. But using it for the purpose of allowing augmented assignment to tuple members seems dubious at best, and likely to create confusion about the mutability or lack thereof of tuples. IMO it's better to have augmented assignment to tuple members always fail, so that the code has to be a little more specific about its intent.
Okay. I assume that we must accept that s = set() t = (s,) t[0] |= set([1]) changes s in spite of raising TypeError. Reinhold -- Mail address is perfectly valid!
At 05:15 PM 9/28/2005 +0200, Reinhold Birkenfeld wrote:
Okay. I assume that we must accept that
s = set() t = (s,) t[0] |= set([1])
changes s in spite of raising TypeError.
There are lots of operations that can be partially completed before raising an error, so I'm not sure why this case would be special. Sets do have an update() method, however, and it's unambiguous as to being an in-place update. The code above would be clearer using it, and produce no errors: s = set() t = (s,) t[0].update([1])
Phillip J. Eby a écrit :
At 03:12 PM 9/28/2005 +0200, Reinhold Birkenfeld wrote: [...] Yes. See:
http://www.python.org/2.0/new-python.html#SECTION000700000000000000000
The purpose of the augmented assignment forms is to allow for the possibility that the item's __i*__ method may or may not exist, and may or may not return the same object. In the case where there is no __i*__ form, or it does not return the same object, the lvalue *must* be re-bound to the new value, or the semantics break.
A case where this matters is here: http://python.org/sf/1306777
I've closed it as invalid; the behavior is as-defined.
Rather than closing this as invalid, it would be wiser to update the documentation before ! Nothing corresponds to the current behavior. I think that in this page : http://docs.python.org/ref/augassign.html The last paragraph whould be replace by : """ For targets which are attribute (or indexed) references, the initial value is retrieved with a getattr() (resp. __getitem__) and the result is assigned with a setattr() (resp. __setitem__). Notice that the two methods do not necessarily refer to the same variable. When getattr() refers to a class variable, setattr() still writes to an instance variable. For example: """ That way it will be clearly defined in the documentation. Now, one can wonder if the augmented assignment is really an improvement. Lots of errors are made because they are counter-intuitive. For example, in the standard library, I found very few uses of "+=" with a mutable object, and none would be broken if "a += b" is to be replaced by "a = a+b". At worst, there will be a performance issue that will easily be fixed by using "extend" method for lists and corresponding methods for other objects. My opinion is, redefining the augmented assignment is a problem given the assignment semantic, and perhaps we should get rid of it. Pierre -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68
At 05:40 PM 9/28/2005 +0200, Pierre Barbier de Reuille wrote:
Rather than closing this as invalid, it would be wiser to update the documentation before ! Nothing corresponds to the current behavior.
I got my information from here: http://www.python.org/2.0/new-python.html#SECTION000700000000000000000
I think that in this page : http://docs.python.org/ref/augassign.html
The last paragraph whould be replace by :
""" For targets which are attribute (or indexed) references, the initial value is retrieved with a getattr() (resp. __getitem__) and the result is assigned with a setattr() (resp. __setitem__). Notice that the two methods do not necessarily refer to the same variable. When getattr() refers to a class variable, setattr() still writes to an instance variable. For example: """
That way it will be clearly defined in the documentation.
Actually, the broken part is this sentence: """Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead.""" It is subtly misleading, and would be better stated as: """Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object the old object is modified instead. In either case, however, the assignment to the target is still performed."""
Now, one can wonder if the augmented assignment is really an improvement. Lots of errors are made because they are counter-intuitive.
Huh?
For example, in the standard library, I found very few uses of "+=" with a mutable object, and none would be broken if "a += b" is to be replaced by "a = a+b". At worst, there will be a performance issue that will easily be fixed by using "extend" method for lists and corresponding methods for other objects.
The intended use case (as I understand it) for augmented assignment is to allow you to hint that an operation should be done in place *if* it can be. It means that you are not expecting a new object to be the result, but are prepared for the possibility it *might* be a new object.
My opinion is, redefining the augmented assignment is a problem given the assignment semantic, and perhaps we should get rid of it.
How is it a problem? If the assignment semantic weren't what it is, what would it be good for? You could just write an in-place method and be done with it. The whole point is that it allows client code not to care whether it's in-place or not, and to allow implementations to decide (even at runtime) whether to return a different object or not.
Phillip J. Eby a écrit :
At 05:40 PM 9/28/2005 +0200, Pierre Barbier de Reuille wrote:
Rather than closing this as invalid, it would be wiser to update the documentation before ! Nothing corresponds to the current behavior.
I got my information from here:
http://www.python.org/2.0/new-python.html#SECTION000700000000000000000
I know ... I already read this page as you already posted it ;)
I think that in this page : http://docs.python.org/ref/augassign.html
The last paragraph whould be replace by :
""" For targets which are attribute (or indexed) references, the initial value is retrieved with a getattr() (resp. __getitem__) and the result is assigned with a setattr() (resp. __setitem__). Notice that the two methods do not necessarily refer to the same variable. When getattr() refers to a class variable, setattr() still writes to an instance variable. For example: """
That way it will be clearly defined in the documentation.
Actually, the broken part is this sentence:
"""Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead."""
It is subtly misleading, and would be better stated as:
"""Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object the old object is modified instead. In either case, however, the assignment to the target is still performed."""
Indeed ! I missed that one :-S Your proposal should be integrated inside documentation (if anyone knows how to do so ...) !!!
Now, one can wonder if the augmented assignment is really an improvement. Lots of errors are made because they are counter-intuitive.
Huh?
Regularly, you see questions about augmented assignment on Python-tutor mailing list, I often have question in my lab because of problems ... most of the time people learn to avoid these operators in the end ! And my look in the standard library confirmed my intuition about it.
For example, in the standard library, I found very few uses of "+=" with a mutable object, and none would be broken if "a += b" is to be replaced by "a = a+b". At worst, there will be a performance issue that will easily be fixed by using "extend" method for lists and corresponding methods for other objects.
The intended use case (as I understand it) for augmented assignment is to allow you to hint that an operation should be done in place *if* it can be. It means that you are not expecting a new object to be the result, but are prepared for the possibility it *might* be a new object.
My opinion is, redefining the augmented assignment is a problem given the assignment semantic, and perhaps we should get rid of it.
How is it a problem? If the assignment semantic weren't what it is, what would it be good for? You could just write an in-place method and be done with it. The whole point is that it allows client code not to care whether it's in-place or not, and to allow implementations to decide (even at runtime) whether to return a different object or not.
The problem is: this seems to be more a problem than a solution ! There is a huge difference between in-place or not, and I find it very difficult not to consider it. If you have a use-case for this "let the object decide about in-place operation or not" I'd be interested as I found none. Pierre PS: I'm not criticizing the assignment operator semantic which is exactly what is should be ;) -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68
At 06:15 PM 9/28/2005 +0200, Pierre Barbier de Reuille wrote:
Regularly, you see questions about augmented assignment on Python-tutor mailing list, I often have question in my lab because of problems ... most of the time people learn to avoid these operators in the end ! And my look in the standard library confirmed my intuition about it.
Some example of the problems would help. For the specific bug report being discussed, I don't understand why someone would use augmented assignment with an immutable lvalue, since x |= y is short for x = x | y, which is clearly invalid on the face of it if x is a tuple member!
The problem is: this seems to be more a problem than a solution ! There is a huge difference between in-place or not, and I find it very difficult not to consider it.
Consider string addition. The fact that string concatenation can be implemented with += allows a string to consider based on its refcount to return a new string or to modify itself in-place. If someone uses a = b + c, it may be assumed that they still desire a reference to b, and that therefore the operation *cannot* be done in-place. If they use a += b, then this is a *hint* that an in-place operation is desirable. So, there are two features of augmented assignment: 1. It is a shortcut for spelling out the full assignment 2. Types that override augmented assignment methods may optimize in-place operations, *without the need for client code to change*.
If you have a use-case for this "let the object decide about in-place operation or not" I'd be interested as I found none.
The whole point of it is that I don't need to *care* whether a particular use is such or not. I simply say what the code intends, and then if somebody needs to pass in something different, or the behavior of some other part of the system changes, then I get that for free. Looking for a specific use case for that is like looking for a use case for duck typing. That is, *everything* is a use case for it, because the point isn't the initial state of the system. The point is what happens when you *change* the system.
Ok, so I took a closer look at the documentation and tried a few things to understand better what you said and I have some remark ... Phillip J. Eby a ecrit :
At 06:15 PM 9/28/2005 +0200, Pierre Barbier de Reuille wrote:
Regularly, you see questions about augmented assignment on Python-tutor mailing list, I often have question in my lab because of problems ... most of the time people learn to avoid these operators in the end ! And my look in the standard library confirmed my intuition about it.
Some example of the problems would help. For the specific bug report being discussed, I don't understand why someone would use augmented assignment with an immutable lvalue, since x |= y is short for x = x | y, which is clearly invalid on the face of it if x is a tuple member!
Well, the problem is:
a = ([1,2], [3,4]) a[0] += [5,6]
... a[0] *is* mutable, so the author of the bug report did not feel like its l-value was immutable as he supposed a[0] was the l-value, however in this case, both "a" and "a[0]" are l-values ! Otherwise, a very common problem (encounter regularly with labmates):
def foo(a,b): a += b return a a = 3 b = 4 c = foo(a,b) # Works fine as intended d = [1,2] e = [3,4] f = foo(d,e) # Oops ... *d* is modified
Of course, actual code is much more complex, but the problem can be reduced to that most of the time. Also, on your sentence (which I find much more accurate than the current one): """Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object the old object is modified instead. In either case, however, the assignment to the target is still performed.""" I would add some pseudo-code equivalence like, if "__iadd__" is defined: a += b <=> a = a.__iadd__(b) which I believe is true from the look at the pseudo-code generated at compile-time. Like this, it is easier to remember and much less subject to interpretation than with human-language sentences ;)
The problem is: this seems to be more a problem than a solution ! There is a huge difference between in-place or not, and I find it very difficult not to consider it.
Consider string addition. The fact that string concatenation can be implemented with += allows a string to consider based on its refcount to return a new string or to modify itself in-place. If someone uses a = b + c, it may be assumed that they still desire a reference to b, and that therefore the operation *cannot* be done in-place. If they use a += b, then this is a *hint* that an in-place operation is desirable.
However, the implementation makes no use of the operator ! First, there is no "__iadd__" in the "str" class, second documentation says that "a+=b" and "a=a+b" are both optimized. So this does not validate for a use-case ^_^
So, there are two features of augmented assignment:
1. It is a shortcut for spelling out the full assignment 2. Types that override augmented assignment methods may optimize in-place operations, *without the need for client code to change*.
Sure, I understood ... still I do not see any need for that, while I can see a bunch of problems !
If you have a use-case for this "let the object decide about in-place operation or not" I'd be interested as I found none.
The whole point of it is that I don't need to *care* whether a particular use is such or not. I simply say what the code intends, and then if somebody needs to pass in something different, or the behavior of some other part of the system changes, then I get that for free. Looking for a specific use case for that is like looking for a use case for duck typing. That is, *everything* is a use case for it, because the point isn't the initial state of the system. The point is what happens when you *change* the system.
My point here is: if the syntax causes a problem and does not solve any, why would we keep it ? As for duck-typing use cases are plenty ! A huge part of my code use benefit of duck-typing and so does the Python library, so there *are* use-cases ! Pierre -- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68
On Thu, Sep 29, 2005, Pierre Barbier de Reuille wrote:
Ok, so I took a closer look at the documentation and tried a few things to understand better what you said and I have some remark ...
I've got some counter-remarks, but python-dev is not the place to discuss them. Please move this thread to comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything.
Done :) I summarized my point of view and I'm waiting for comments :) Pierre Aahz a écrit :
On Thu, Sep 29, 2005, Pierre Barbier de Reuille wrote:
Ok, so I took a closer look at the documentation and tried a few things to understand better what you said and I have some remark ...
I've got some counter-remarks, but python-dev is not the place to discuss them. Please move this thread to comp.lang.python.
-- Pierre Barbier de Reuille INRA - UMR Cirad/Inra/Cnrs/Univ.MontpellierII AMAP Botanique et Bio-informatique de l'Architecture des Plantes TA40/PSII, Boulevard de la Lironde 34398 MONTPELLIER CEDEX 5, France tel : (33) 4 67 61 65 77 fax : (33) 4 67 61 56 68
participants (5)
-
Aahz -
James Y Knight -
Phillip J. Eby -
Pierre Barbier de Reuille -
Reinhold Birkenfeld