Re: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward

Hello Guido,
I did a quick review of the stdlib, including the tests, to see which list comprehensions could be replaced with generator expressions. Thought I admit I am biased towards early binding, I ended up looking for cases with the following properties: [...]
Even if we switched to early binding, won't these issues still bite you with mutable values? E.g.: some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict['z'] = 5 iter2 = (some_dict.get(v, 7) for v in input2) It seems like it would be surprising (to me, anyway) if this gave a different result than: some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict = {'x': 1, 'y': 2, 'z': 5} iter2 = (some_dict.get(v, 7) for v in input2) -Edward

Even if we switched to early binding, won't these issues still bite you with mutable values? E.g.:
some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict['z'] = 5 iter2 = (some_dict.get(v, 7) for v in input2)
It seems like it would be surprising (to me, anyway) if this gave a different result than:
some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict = {'x': 1, 'y': 2, 'z': 5} iter2 = (some_dict.get(v, 7) for v in input2)
Semantically, in the first case you are modifying your dictionary, which has consequences to anything that has a reference to the object. Mutable modification during runtime should be something we are used to by now. In the second case, you are rebinding the name some_dict to something else. Your earlier binding in the generator expression to some_dict (if we had early-binding for generator expressions) doesn't change, so the fact that the two iterators in the second case return different results, to me (and others I'm sure), is expected. Name rebinding should also be something we are used to by now. I think that it would be very convenient if generator expressions had the exact same behavior as list comprehensions (people already know how to do those). As such, early binding is essentially what list comprehensions do, though they don't suffer from the "I modified my dictionary before I used my generator expression" issue. In that case, the only way we can preserve behavior is through a copy of all the objects that are bound in early binding, which is obviously a no-no, and doesn't necessarily solve our problem. I'm happy with early binding, even given the mutable object modification issue shown above, and I think it would be relatively easy to document with a few minor examples. What does everyone else think? - Josiah

Josiah Carlson <jcarlson@uci.edu>:
Semantically, in the first case you are modifying your dictionary, which has consequences to anything that has a reference to the object. Mutable modification during runtime should be something we are used to by now.
The point is that anyone who's going to be confused by the difference between early and late binding behaviour is likely to be equally confused by this. The issue is really about when expressions are evaluated, not when names are bound. Patching up just one part of it (binding of free variables) doesn't really fix the problem. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

At 12:29 30.04.2004 -0400, Edward Loper wrote:
Hello Guido, I did a quick review of the stdlib, including the tests, to see which list comprehensions could be replaced with generator expressions. Thought I admit I am biased towards early binding, I ended up looking for cases with the following properties: [...]
Even if we switched to early binding, won't these issues still bite you with mutable values? E.g.:
some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict['z'] = 5 iter2 = (some_dict.get(v, 7) for v in input2)
It seems like it would be surprising (to me, anyway) if this gave a different result than:
some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict = {'x': 1, 'y': 2, 'z': 5} iter2 = (some_dict.get(v, 7) for v in input2)
well, the deep problem here is that lazinesss and side-effects (or mutability, if one wants, inclusive of bindings) do not mix well. There is so much that we can do about that. It's really a matter of, depending on our choices, how often the casual user will be bitten by this and what kind of questions we want to get. Otherwise we would have to give up about the whole idea.
participants (4)
-
Edward Loper
-
Greg Ewing
-
Josiah Carlson
-
Samuele Pedroni