Andrew Koenig wrote:

Why can't you do this? foo = log.setdefault(r,'') foo += "test %d\n" % t

You can do it, but it's useless!

...

...

d = {} foo = d.setdefault(42, "buckle") foo += " my shoe" d {42: 'buckle'}

What Mr. Leighton wanted is *impossible* when the value concerned is immutable, because by the time you get to the += operator, there's no information left about where the value came from, and thus no way to update the dict with the new value. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Andrew Koenig <ark@research.att.com>:

So the behavior of

foo = d.setdefault(r,'') foo += "test %d\n" % t

depends on what type foo has, and the OP didn't say.

I assumed that the code snippet was from his actual application, in which case he *did* want it to work on strings, in which case, even if he had the feature he wanted, it wouldn't have helped him. I think the fact that this would only work when the value was mutable is a good reason to disallow it. Too big a source of surprises, otherwise. Being forced to find another way to update the value in this case is a feature, because the absence of such a way when the value is immutable makes it clear that there's no way to do what you're trying to do! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Greg Ewing wrote:

Andrew Koenig <ark@research.att.com>:

...

So the behavior of

foo = d.setdefault(r,'') foo += "test %d\n" % t

depends on what type foo has, and the OP didn't say.

I assumed that the code snippet was from his actual application, in which case he *did* want it to work on strings, in which case, even if he had the feature he wanted, it wouldn't have helped him. [...] Being forced to find another way to update the value in this case is a feature, because the absence of such a way when the value is immutable makes it clear that there's no way to do what you're trying to do!

Mutable (or at least appendable) strings should probably be done with StringIO/cStringIO. How about adding support for __iadd__ and __str__ (and __unicode__) to both? Bye, Walter Dörwald

ark> Why can't you do this? ark> for t in range(5): ark> for r in range(10): ark> foo = log.setdefault(r,'') ark> foo += "test %d\n" % t Luke> after running this code, Luke> log = {0: '', 1: '', 2:'', 3: '' ... 9: ''} Luke> and foo equals "test 5". Then that is what foo would be if you were able to write log.setdefault(r,'') += "test %d\n" % t as you had wished. Luke> if, however, you do this: Luke> for t in range(5): Luke> for r in range(10): Luke> foo = log.setdefault(r,[]) Luke> foo.append("test %d\n" % t) Luke> then empirically i conclude that you DO end up with the Luke> expected results (but is this true all of the time?) I presume that is because you are now dealing with vectors instead of strings. In that case, you could also have written for t in range(5): for r in range(10): foo = log.setdefault(r,[]) foo += ["test %d]n" % t] with the same effect. Luke> the reason why your example, andrew, does not work, is Luke> because '' is a string - a basic type to which a pointer is Luke> NOT returned i presume that the foo += "test %d"... returns a Luke> DIFFERENT result object such that the string in the dictionary Luke> is DIFFERENT from the string result of foo being updated. Well, yes. But that is what you would have gotten had you been allowed to write log.setdefault(r,"") += <whatever> in the first place. Luke> if that makes absolutely no sense whatsoever then think of it Luke> being the difference between integers and pointers-to-integers Luke> in c. I think this analogy is pointless, as the only people who will understand it are those who didn't need it in the first place :-) Luke> can anyone tell me if there are any PARTICULAR circumstances where Luke> foo = log.setdefault(r,[]) Luke> foo.append("test %d\n" % t) Luke> will FAIL to work as expected? It will fail if your expectations are incorrect or unrealistic. Luke> andrew, sorry it took me so long to respond: i initially Luke> thought that under all circumstances for all types of foo, Luke> your example would work. But it does! At least in the sense of the original query. The original query was of the form Why can't I write an expression like f(x) += y? and my answer was, in effect, If you could, it would have the same effect as if you had written foo = f(x) foo += y and then used the value of foo. Perhaps I'm missing something, but I don't think that anything you've said contradicts this answer. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark

On Fri, May 16, 2003 at 08:07:23AM -0400, Andrew Koenig wrote:

ark> Why can't you do this?

ark> for t in range(5): ark> for r in range(10): ark> foo = log.setdefault(r,'') ark> foo += "test %d\n" % t

Luke> after running this code,

Luke> log = {0: '', 1: '', 2:'', 3: '' ... 9: ''}

Luke> and foo equals "test 5".

Then that is what foo would be if you were able to write

log.setdefault(r,'') += "test %d\n" % t

as you had wished.

hmm... ..mmmm... you're absolutely right!!!

Luke> if, however, you do this:

Luke> for t in range(5): Luke> for r in range(10): Luke> foo = log.setdefault(r,[]) Luke> foo.append("test %d\n" % t)

Luke> then empirically i conclude that you DO end up with the Luke> expected results (but is this true all of the time?)

I presume that is because you are now dealing with vectors instead of strings. In that case, you could also have written

for t in range(5): for r in range(10): foo = log.setdefault(r,[]) foo += ["test %d]n" % t]

with the same effect.

Luke> the reason why your example, andrew, does not work, is Luke> because '' is a string - a basic type to which a pointer is Luke> NOT returned i presume that the foo += "test %d"... returns a Luke> DIFFERENT result object such that the string in the dictionary Luke> is DIFFERENT from the string result of foo being updated.

Well, yes. But that is what you would have gotten had you been allowed to write

log.setdefault(r,"") += <whatever>

in the first place.

[i oversimplified in the example, leading to all the communication problems. the _actual_ usage i was expecting is based on {}.setdefault(0, []) += [1,2] rather than setdefault(0, '') += 'hh' ]

Luke> can anyone tell me if there are any PARTICULAR circumstances where

Luke> foo = log.setdefault(r,[]) Luke> foo.append("test %d\n" % t)

Luke> will FAIL to work as expected?

It will fail if your expectations are incorrect or unrealistic.

... that sounds like a philosophical or "undefined" answer rather than the technical one i was seeking ... but it is actually quite a _useful_ answer :) to put the question in a different way, or to say again, to put a different, more specific, question: can anyone tell me if there are circumstances under which the second argument from setdefault will SOMETIMES be copied instead of returned and SOMETIMES be returned as-is, such that operations of the type being attempted will SOMETIMES work as expected and SOMETIMES not?

Luke> andrew, sorry it took me so long to respond: i initially Luke> thought that under all circumstances for all types of foo, Luke> your example would work.

But it does! At least in the sense of the original query.

where the sense was mistaken, consequently the results are not, as you rightly pointed out, not as expected.

The original query was of the form

Why can't I write an expression like f(x) += y?

and my answer was, in effect,

If you could, it would have the same effect as if you had written

foo = f(x) foo += y

and then used the value of foo.

Perhaps I'm missing something, but I don't think that anything you've said contradicts this answer.

based on this clarification, my queries are two-fold: 1) what is the technical, syntactical or language-specific reason why I can't write an expression like f(x) += y ? 2) the objections that i can see as to why f(x) += y should not be _allowed_ to work are that, as andrew points out, some people's expectations of any_function() += y may be unrealistic, particularly as normally the result of a function is discarded. however, in the case of setdefault() and get() on dictionaries, the result of the function is NOT discarded: in SOME instances, a reference is returned to the dictionary item. under such circumstances, why should the objections - to disallow {}.setdefault(0, []) += [] or {}.get([]) += [] - stand? l.

hiya jeff, on radio 4 today there was a discussion about art - what makes people go "wow" instead of being shocked. seeing the byte code in front of my eyes isn't so much of a shock, more of a "wow" because i have at some point in my past actually _looked_ at the python sources stack machine, for investigating parallelising it (!!!!!) okay. how do i run the examples you list? dis.dis(f) gives an "unrecognised variablename dis". okay. let's give this a shot. Script started on Mon May 19 08:44:19 2003 lkcl@highfield:~$ python O Python 2.2.2 (#1, Jan 18 2003, 10:18:59) [GCC 3.2.2 20030109 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

import dis def g(): return f(x) ... dis.dis(g) 0 LOAD_GLOBAL 0 (f) 3 LOAD_GLOBAL 1 (x) 6 CALL_FUNCTION 1 9 RETURN_VALUE 10 LOAD_CONST 0 (None) 13 RETURN_VALUE def g(): f(x) ... dis.dis(g) 0 LOAD_GLOBAL 0 (f) 3 LOAD_GLOBAL 1 (x) 6 CALL_FUNCTION 1 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE

lkcl@highfield:~$ exit

Script done on Mon May 19 08:44:56 2003 right. the difference between these two is the POP_TOP. so, the return result is placed on the stack, from the call to f(x). so... if there's instead an f(x) += 1 instead of f(x), then the result is going to be pushed onto the top of the stack, followed by the += 1, followed at the end by a POP_TOP. if the result is used (e.g. assigned to a variable), x = f(x) += 1 then you don't do the POP_TOP. ... am i missing something? what am i missing? that it's not known what type of variable is returned, therefore you're not certain as to what type of STORE to use? Script started on Mon May 19 08:51:22 2003 lkcl@highfield:~$ python -O Python 2.2.2 (#1, Jan 18 2003, 10:18:59) [GCC 3.2.2 20030109 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

def f(): return 5 ... def g(): ... x = f() + 1 ... return x ... import dis dis.dis(g) 0 LOAD_GLOBAL 0 (f) 3 CALL_FUNCTION 0 6 LOAD_CONST 1 (1) 9 BINARY_ADD 10 STORE_FAST 0 (x) 13 LOAD_FAST 0 (x) 16 RETURN_VALUE 17 LOAD_CONST 0 (None) 20 RETURN_VALUE

lkcl@highfield:~$ Script done on Mon May 19 08:52:40 2003

okay... soo.... you get an assignment into a variable... ... okay, i think i see what the problem is. because the return result _may_ not be used, you don't know what type of STORE to use? or, because there are optimisations added, it's not always possible to "pass down" the right kind of STORE_xxx to the previous stack level? i believe you may be thinking that this is more complex than it is. that's very patronising of me. scratch that. i believe this should not be complex :) "+=" itself is a function call with two arguments and a return result, where the return result is the first argument. it just _happens_ that that function call has been drastically optimised - with its RETURN_VALUE removed; STORE_xxx removed. more thought needed. i'll go look at some code. l. p.s. 10 and 13 in the 8:52:40am typescript above look like they could be optimised / removed. p.p.s. yes i _have_ written a stack-machine optimiser before. On Sat, May 17, 2003 at 10:21:39AM -0500, Jeff Epler wrote:

I think that looking at the generated bytecode is useful.

# Running with 'python -O'

...

...

...

def f(x): x += 1 dis.dis(f) 0 LOAD_FAST 0 (x) 3 LOAD_CONST 1 (1) 6 INPLACE_ADD 7 STORE_FAST 0 (x) *** 10 LOAD_CONST 0 (None) 13 RETURN_VALUE def g(x): x[0] += 1 dis.dis(g) 0 LOAD_GLOBAL 0 (x) 3 LOAD_CONST 1 (0) 6 DUP_TOPX 2 9 BINARY_SUBSCR 10 LOAD_CONST 2 (1) 13 INPLACE_ADD 14 ROT_THREE 15 STORE_SUBSCR *** 16 LOAD_CONST 0 (None) 19 RETURN_VALUE def h(x): x.a += 1 dis.dis(h) 0 LOAD_GLOBAL 0 (x) 3 DUP_TOP 4 LOAD_ATTR 1 (a) 7 LOAD_CONST 1 (1) 10 INPLACE_ADD 11 ROT_TWO 12 STORE_ATTR 1 (a) *** 15 LOAD_CONST 0 (None) 18 RETURN_VALUE

In each case, there's a STORE step to the inplace statement. In the case of the proposed def j(x): x() += 1 what STORE instruction would you use?

...

...

...

[opname for opname in dis.opname if opname.startswith("STORE")] ['STORE_SLICE+0', 'STORE_SLICE+1', 'STORE_SLICE+2', 'STORE_SLICE+3', 'STORE_SUBSCR', 'STORE_NAME', 'STORE_ATTR', 'STORE_GLOBAL', 'STORE_FAST', 'STORE_DEREF']

If you don't want one from the list, then you're looking at substantial changes to Python.. (and STORE_DEREF probably doesn't do anything that's relevant to this situation, though the name sure sounds promising, doesn't it)

Jeff

-- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about...

At 12:53 19.05.2003 +0000, Luke Kenneth Casson Leighton wrote:

jeff,

beat bolli's code example:

count[word] = count.get(word, 0) + 1

i think best illustrates what issue you are trying to raise.

okay, we know there are two issues so let's give an example that removes one of those issues:

count = {}

count[word] = count.get(word, []) + ['hello']

the issue is that the difference between the above 'hello' example and this:

count.get(word, []) += ['hello']

is that you don't know what STORE to use after the use of get() in the second example, but you do in the first example because it's explicity set out.

so, does this help illustrate what might be done?

if it's possible to return a result and know what should be done with it, then surely it should be possible to return a result from a += "function" and know what should be done with it?

l.

...

...

def refiadd(r,v): # r+=v, r is a reference, not a an lvalue ... if hasattr(r.__class__,'__iadd__'): ... r.__class__.__iadd__(r,v) ... else: ... raise ValueError,"non-sense" ... greetings={} refiadd(greetings.setdefault('susy',[]),['hello']) # greetings.setdefault('s usy',[]) += ['hello'] refiadd(greetings.setdefault('susy',[]),['!']) # greetings.setdefault('susy' ,[]) += ['!'] greetings {'susy': ['hello', '!']} refiadd(greetings.setdefault('betty',1),1) # greetings.setdefault('susy',1) += 1 Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 5, in refiadd ValueError: non-sense

regards.

"Luke Kenneth Casson Leighton" <lkcl@samba-tng.org> wrote in message > 1) what is the technical, syntactical or language-specific reason why

I can't write an expression like f(x) += y ?

In general, ignoring repetition of side-effects, this translates to f(x) = f(x) + y. Abstractly, the assignment pattern is target(s) = object(s), whereas above is object = object. As some of have tried to point out to the cbv (call-by-value) proponents on a clp thread, targets are not objects. so object = object is not proper Python. The reason inplace op syntax is possible is that syntax that defines a target on the left instead denotes an object when on the right (of '='), so that syntax to the left of op= does double duty. As Jeff Eppler pointed out in his response, the compiler uses that syntax to determine the type of target and thence the appropriate store instruction. But function call syntax only denotes an object and does not define a target and hence cannot do double duty. The exception to all this is listob += seq, which translates to listob.extend(seq). So if f returns a list, f(x) += y could be executed, but only with runtime selection of the apropriate byte code. However, if you know that f is going to return a list, so that f(x)+=y seem sensible, you can directly write f(x).extend(y) directly (or f(x).append(y) if that is what you actually want). However, since this does not bind the result to anything, even this is pointless unless all f does is to select from lists that you can already access otherwise. (Example: f(lista,listb,bool_exp).extend(y).) Terry J. Reedy