[Python-3000] Is this a bug with list comprehensions or not?
ncoghlan at gmail.com
Fri Jul 11 14:59:04 CEST 2008
Raymond Hettinger wrote:
> From: "Nick Coghlan" <ncoghlan at gmail.com>
>> Georg and I tried doing it that way and had major problems trying to
>> get it to work - the hard part is that the body of the list comp
>> (which may include nested list comps, lambda functions and generator
>> expressions) needs to be able to see the iteration variables exactly
>> as they are, but the surrounding code shouldn't be able to see them at
> Did you try just saving and restoring the variable so that hiding
> wouldn't be necessary.
> [for x in s]
> save(x) if it exists
> do the listcomp
> restore(x) if it was saved
And watch any generator expressions or lambda functions returned from
the body of the list comprehension that reference the iteration variable
fail inexplicably. Either we revert to 2.x behaviour and leak the
iteration variable or we keep the extra scope that is added by the
current implementation. Otherwise it just doesn't work.
> It's a bummer that the genexp implementation was borrowed
> for this purpose. IMO, what we had before was cleaner,
> faster, obvious, and simple.
Then argue for a reversion to the 2.x behaviour: if we want to continue
to explain list comprehensions in terms of an equivalent for loop, then
we need to leak the iteration variables just like a for loop would.
A version which hid the iteration variable from the containing scope but
didn't share the other scoping characteristics of generator expressions
would just be strange.
> BTW, the performance cost isn't "possible", it's a guaranteed,
> non-trivial regression. It becomes a reason to not upgrade to 3.0.
Time it and then say that. For module and class scope code, the
performance gain from using function locals for the iteration variables
inside the body of the list comp quickly outstrips the overhead from the
single function call needed to invoke the implicitly created function.
For list comprehensions that are already at function scope (and hence
already benefiting from the function local optimisations), I agree the
new implementation is a guaranteed regression (the overhead of a single
> All I'm asking Guido is whether he is open to an alternate patch
> that doesn't create an inner function.
Create the patch and have it pass all the current Py3k list
comprehension tests and I'll support this. Without code in hand, I
believe what you're suggesting simply isn't practical to implement (i,e,
you would have to rewrite large chunks of the compiler to make it work
> P.S. I know this group doesn't care about Psyco, but it was
> nice that psyco could handle listcomps just like it could with
> regular for-loops. Turning it into a genexp stops psyco in its tracks.
> Likewise, Cython won't be able to handle the semantics.
> ISTM, we've made something hard out of something that
> used to be simple. What was gained?
Exactly what was asked for: the iteration variable doesn't leak into the
surrounding scope anymore. The behaviour you're currently objecting to
is the natural consequence of hiding list comprehension iteration
variables from the local scope given Python's lexical scoping rules.
So if you want to get "simple" back, then I firmly believe you need to
argue in favour of continuing to leak the iteration variables from list
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
More information about the Python-3000