Seeming unintended difference between list comprehensions and generator expressions...
Recently I found the need to generate some constants inside a class body. What I discovered was a bit unintuitive, and may not be intended... In 2.5 and 2.6:
class foo: ... x = {} ... x.update((i, x.get(i, None)) for i in xrange(10)) ... Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in foo File "<stdin>", line 3, in <genexpr> NameError: global name 'x' is not defined class foo: ... x = {} ... x.update([(i, x.get(i, None)) for i in xrange(10)]) ...
In 3.0:
class foo(): ... x = {} ... x.update((i, x.get(i, None)) for i in range(10)) ... Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in foo File "<stdin>", line 3, in <genexpr> NameError: global name 'x' is not defined class foo(): ... x = {} ... x.update([(i, x.get(i, None)) for i in range(10)]) ... Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in foo File "<stdin>", line 3, in <listcomp> NameError: global name 'x' is not defined
The behavior of 3.0 WRT list comprehensions behaving the same way as generator expressions is expected, but why generator expressions (generally) don't keep a reference to the class scope during execution seems to be unintended. - Josiah
Josiah Carlson wrote:
The behavior of 3.0 WRT list comprehensions behaving the same way as generator expressions is expected, but why generator expressions (generally) don't keep a reference to the class scope during execution seems to be unintended.
It's intended. While arguably not ideal, it would require some pretty major changes to the lexical scoping rules to make them behave any differently. The translation of (i*i for i in x) is conceptually along the lines of: def _ge(arg): for i in arg: yield i*i <expr_value> = _ge(x) Similarly, a 3.x list comprehension [i*i for i in x] is very roughly translated as: def _lc(arg): result = [] for i in arg: result.append(i*i) return result <expr_value> = _lc(x) Like any function scope inside a class namespace, the body of a genexp (and, in 3.x, comprehension) doesn't have direct access to the class namespace because classes don't play any part in the lexical scoping rules. Basically, if a generator or 3.x comprehension needs access to a value from a containing class scope anywhere other than the outermost iterator, then it needs to be put into a temporary function and given the extra value as an argument: .>> class C: ... x = {} ... def _init_constants(d, itr): ... d.update((i, d.get(i, None)) for i in itr) ... _init_constants(x, range(10)) ... del _init_constants ... .>> C.x {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None} (in this toy case, of course, it would be simpler to make the temporary function create and return the constants dictionary, but the above approach with multiple arguments being passed in applies more generally when you need to access multiple existing values from the class scope) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Fri, Feb 20, 2009 at 3:14 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Josiah Carlson wrote:
The behavior of 3.0 WRT list comprehensions behaving the same way as generator expressions is expected, but why generator expressions (generally) don't keep a reference to the class scope during execution seems to be unintended.
It's intended. While arguably not ideal, it would require some pretty major changes to the lexical scoping rules to make them behave any differently.
The translation of (i*i for i in x) is conceptually along the lines of:
def _ge(arg): for i in arg: yield i*i
<expr_value> = _ge(x)
Similarly, a 3.x list comprehension [i*i for i in x] is very roughly translated as:
def _lc(arg): result = [] for i in arg: result.append(i*i) return result
<expr_value> = _lc(x)
I was under the impression that in 3.x, it was equivalent to list(<genexp>) . Which would explain the difference between 2.6 and 3.0.
Like any function scope inside a class namespace, the body of a genexp (and, in 3.x, comprehension) doesn't have direct access to the class namespace because classes don't play any part in the lexical scoping rules.
Indeed, though I had thought (if only briefly ;) ) that when executing non-definitions in the class body, it would behave similar to the a more or less equivalent function-based class factory
def make_class(*bases): ... def make_class(fcn): ... dict = fcn() ... return type(object)(fcn.__name__, bases, dict) ... return make_class ... @make_class(object) ... def foo(): ... x = {} ... x.update((i, x.get(i, None)) for i in xrange(10)) ... return locals() ... foo <class '__main__.foo'> foo.x {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None}
But I was wrong ;) Thank you for the help :) - Josiah
Basically, if a generator or 3.x comprehension needs access to a value from a containing class scope anywhere other than the outermost iterator, then it needs to be put into a temporary function and given the extra value as an argument:
.>> class C: ... x = {} ... def _init_constants(d, itr): ... d.update((i, d.get(i, None)) for i in itr) ... _init_constants(x, range(10)) ... del _init_constants ... .>> C.x {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None}
(in this toy case, of course, it would be simpler to make the temporary function create and return the constants dictionary, but the above approach with multiple arguments being passed in applies more generally when you need to access multiple existing values from the class scope)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Josiah Carlson wrote:
Similarly, a 3.x list comprehension [i*i for i in x] is very roughly translated as:
def _lc(arg): result = [] for i in arg: result.append(i*i) return result
<expr_value> = _lc(x)
I was under the impression that in 3.x, it was equivalent to list(<genexp>) . Which would explain the difference between 2.6 and 3.0.
Semantically, the two translations end up being the same. Actually *implementing* it that way turned out to be unnecessarily slow: the comprehensions instead create a normal function that builds the desired type of object (list/set/dict) incrementally rather than building a generator function and invoking it. The speed increase comes from being able to skip the generator resume/yield overhead on each iteration.
Like any function scope inside a class namespace, the body of a genexp (and, in 3.x, comprehension) doesn't have direct access to the class namespace because classes don't play any part in the lexical scoping rules.
Indeed, though I had thought (if only briefly ;) ) that when executing non-definitions in the class body, it would behave similar to the a more or less equivalent function-based class factory
I think there may be a section of the language reference that describes it in those terms. If there is, it should probably be changed, since it gives misleading ideas about how the scoping works. Execution of a class body is actually a bit more like using an exec statement than it is like an implicit function definition. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Josiah Carlson wrote:
Similarly, a 3.x list comprehension [i*i for i in x] is very roughly translated as:
def _lc(arg): result = [] for i in arg: result.append(i*i) return result
<expr_value> = _lc(x)
I was under the impression that in 3.x, it was equivalent to list(<genexp>) . Which would explain the difference between 2.6 and 3.0.
Semantically, the two translations end up being the same.
Actually, they're not quite the same. list(<genexp>) will swallow any StopIteration exceptions that end up getting thrown inside the generator, but even in Python 3, list comprehensions will not catch StopIteration. So for explanatory purposes, thinking of a list comprehension as a function that returns a list is more useful than thinking of it as list(<genexp>). -- Carl
participants (3)
-
Carl Johnson -
Josiah Carlson -
Nick Coghlan