At 09:02 AM 1/27/04 +1100, Delaney, Timothy C (Timothy) wrote:
From: Guido van Rossum
I'm happy to see progress on the generator expressions
implementation,
but I think the specification of namespaces, which is just a sketch, might be simplified.
Ouch! Where were you when this PEP was discussed on python-dev? I was originally strongly in your camp, but Tim and several others convinced me that in every single case where a generator expression has a free variable, you want early binding, not late.
I think I agree with Jeremy. I was originally in the early-binding camp, but I think we're better off trusting the programmer.
The intent is that listcomps should be "safely" replaceable with genexprs anywhere that an iterator is acceptable in place of a list.
How often is a generator expression not going to be evaluated almost immediately? I guess, when they're passed to a function. But even in that case, how often are the bindings going to change? Except in pathological cases, they won't.
A trivial example is:
iterators = [] for i in range(5): iterators.append(x*2 for x in range(i))
print map(list,iterators)
If a listcomp is used, you get:
[[],[0,],[0,2],[0,2,4],[0,2,4,6],[0,2,4,6,8]]
If genexprs do late binding, you get:
[[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8]]
which is a significant difference in semantics. To make this code work with a late binding genexpr, it becomes necessary to create a function definition, thus negating the benefit of using the genexpr in the first place.
As to how common this is, ask how often you use a list comprehension inside of another loop, and then ask how often you'd use a genexpr instead if it was available. Finally, ask yourself whether you're likely to remember that you need to totally rewrite the structure if you decide to use a genexpr instead. :)
(Of course, I also often forget that free variables don't bind early in nested functions! It always seems strange to me that I have to write a function that returns a function in order to just define a function with bound variables. Indeed, it's hard to think of a time where I've *ever* wanted late binding of variables in a nested function.)
On Mon, 2004-01-26 at 17:36, Phillip J. Eby wrote:
The intent is that listcomps should be "safely" replaceable with genexprs anywhere that an iterator is acceptable in place of a list.
How often is a generator expression not going to be evaluated almost immediately? I guess, when they're passed to a function. But even in that case, how often are the bindings going to change? Except in pathological cases, they won't.
A trivial example is:
Are there non-trivial examples? The PEP suggests that they exist, but doesn't provide any.
iterators = [] for i in range(5): iterators.append(x*2 for x in range(i))
print map(list,iterators)
If a listcomp is used, you get:
[[],[0,],[0,2],[0,2,4],[0,2,4,6],[0,2,4,6,8]]
If genexprs do late binding, you get:
[[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8]]
Note that Armin Rigo suggested a small change here a few weeks ago that makes the list comp and the gen expr behave the same way. The range(i) part of the gen expr is evaluated at the point of definition. The gen expr, thus, translates to:
def f(it): for x in it: yield x * 2
iterators.append(f(range(i))
As a result of this change, the only new scope is for the body of the target expression. I don't know how that effects the earlier examples.
BTW is there good terminology for describing the parts of a list comp or gen expr? How about the iterator, the conditions, and the expression?
Jeremy
At 07:51 PM 1/26/04 -0500, Jeremy Hylton wrote:
On Mon, 2004-01-26 at 17:36, Phillip J. Eby wrote:
The intent is that listcomps should be "safely" replaceable with genexprs anywhere that an iterator is acceptable in place of a list.
How often is a generator expression not going to be evaluated almost immediately? I guess, when they're passed to a function. But even in that case, how often are the bindings going to change? Except in pathological cases, they won't.
A trivial example is:
Are there non-trivial examples? The PEP suggests that they exist, but doesn't provide any.
http://mail.python.org/pipermail/python-dev/2003-October/039323.html
I'd have posted a link to this first, but it took a while to track it down.
iterators = [] for i in range(5): iterators.append(x*2 for x in range(i))
print map(list,iterators)
If a listcomp is used, you get:
[[],[0,],[0,2],[0,2,4],[0,2,4,6],[0,2,4,6,8]]
If genexprs do late binding, you get:
[[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8]]
Note that Armin Rigo suggested a small change here a few weeks ago that makes the list comp and the gen expr behave the same way.
It's a start, but it doesn't address Tim's example.
BTW is there good terminology for describing the parts of a list comp or gen expr? How about the iterator, the conditions, and the expression?
Sounds good to me. My contrived example demonstrated early binding for the iterator. Tim's shows early binding for the conditions. I think it's also straightforward to see that early binding for the expression is similarly desirable.
If you're looking for more arguments, I'd suggest looking at all of Tim Peters' posts for October under the subject "accumulator display syntax". :)
On Mon, 2004-01-26 at 21:10, Phillip J. Eby wrote:
If you're looking for more arguments, I'd suggest looking at all of Tim Peters' posts for October under the subject "accumulator display syntax". :)
I'll try, but I was hoping the PEP author would help out here. An important part of the PEP process is documenting the rationale for a decision. I'm not opposed to reading through all the messages, but it's going to take a lot more time than reading a PEP.
Jeremy
[Jeremy]
Are there non-trivial examples? The PEP suggests that they exist, but doesn't provide any.
Well, this example *should* be both easy and natural -- but it turns out to be a disaster on two distinct counts if late binding is used:
http://mail.python.org/pipermail/python-dev/2003-October/039323.html
I'll repeat one bit from
http://mail.python.org/pipermail/python-dev/2003-October/039328.html
too:
Whenever I've written a list-of-generators, or in the recent example a generator pipeline, I have found it semantically necessary, without exception so far, to capture the bindings of the variables whose bindings wouldn't otherwise be invariant across the life of the generator. [If] it turns out that this is always, or nearly almost always, the case, across future examples too, then it would just be goofy not to implement generator expressions that way ("well, yes, the implementation does do a wrong thing in every example we had, but what you're not seeing is that the explanation would have been a line longer had the implementation done a useful thing instead" <wink>).
Note that Armin Rigo suggested a small change here a few weeks ago that makes the list comp and the gen expr behave the same way. The range(i) part of the gen expr is evaluated at the point of definition. The gen expr, thus, translates to:
def f(it): for x in it: yield x * 2
iterators.append(f(range(i))
As a result of this change, the only new scope is for the body of the target expression. I don't know how that effects the earlier examples.
The example in the first link above is:
pipe = source for p in predicates: # add a filter over the current pipe, and call that the new pipe pipe = e for e in pipe if p(e)
"p" and "pipe" both vary, and disaster ensues if the bindings (for both) in effect at the time of each binding (to "pipe") aren't used when the last generator in the chain (the final binding of "pipe") is provoked into delivering results. If Armin's suggestion transforms that to
pipe = source for p in predicates: def g(it): for e in it: if p(e): yield e pipe = g(pipe)
then the binding of "p" is left bound to predicates[-1] in all the intermediate generators.
BTW is there good terminology for describing the parts of a list comp or gen expr? How about the iterator, the conditions, and the expression?
Probably more descriptive than body, mind and soul <wink>. Fine by me!
A trivial example is:
Are there non-trivial examples? The PEP suggests that they exist, but doesn't provide any.
iterators = [] for i in range(5): iterators.append(x*2 for x in range(i))
print map(list,iterators)
If a listcomp is used, you get:
[[],[0,],[0,2],[0,2,4],[0,2,4,6],[0,2,4,6,8]]
If genexprs do late binding, you get:
[[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8],[0,2,4,6,8]]
Note that Armin Rigo suggested a small change here a few weeks ago that makes the list comp and the gen expr behave the same way.
I may be mistaken, but I had always assumed it should be the way Armin suggested.
The range(i) part of the gen expr is evaluated at the point of definition. The gen expr, thus, translates to:
def f(it): for x in it: yield x * 2
iterators.append(f(range(i))
As a result of this change, the only new scope is for the body of the target expression. I don't know how that effects the earlier examples.
There were definitely examples along the lines of using 'i' in the target expression.
BTW is there good terminology for describing the parts of a list comp or gen expr? How about the iterator, the conditions, and the expression?
Expression is too generic; I suggest target expression.
--Guido van Rossum (home page: http://www.python.org/~guido/)
At 17:36 26.01.2004 -0500, Phillip J. Eby wrote:
(Of course, I also often forget that free variables don't bind early in nested functions! It always seems strange to me that I have to write a function that returns a function in order to just define a function with bound variables. Indeed, it's hard to think of a time where I've *ever* wanted late binding of variables in a nested function.)
0) It should be noted that late binding makes likely the most sense when rebinding is allowed (a feature I still don't know whether it ought to be added, but that's beside the point). At the moment what it allows is mutually referring or self-recursive nested definitions:
def h(): def f() # f can refer to g def g() ...
[ not that is totally unimagibable a semantics where:
def f(): x = 1 def g(): return x x = 2 return g()
def f(): def g(): return x x = 1 x = 2 return g()
both return 1 ... ]
1) we suffer a bit more the late binding semantics because creating a new scope is not a lightweight operation in Python, consider this Common Lisp code:
(defun f ()
(do ((x 0 (+ x 1)) (lst () (append lst (list (lambda () x))))) ((= x 4) lst)))
(mapcar #'funcall (f))
(4 4 4 4)
kind of like: def f(): x = 0 lst = [] while not (x == 4): lst.append(lambda : x) x += 1 return lst
[ el() for el in f() ]
[4, 4, 4, 4]
an inline 'let' makes it do the what's maybe expected:
(defun f ()
(do ((x 0 (+ x 1)) (lst () (append lst (list (let ((x x)) (lambda () x)))))) ((= x 4) lst)))
(mapcar #'funcall (f))
(0 1 2 3)
In Scheme both 'do' and named-let directly introduce a fresh scope per iteration:
(define (f) (do ((x 0 (+ x 1)) (lst () (append lst (list (lambda () x))))) ((= x 4) lst) ))
(map (lambda (func) (func)) (f))
(0 1 2 3)
regards.
-
Guido van Rossum -
Jeremy Hylton -
Phillip J. Eby -
Samuele Pedroni -
Tim Peters