[Python-3000] Is this a bug with list comprehensions or not?

[Nick Coghlan]

Raymond Hettinger wrote:
> From: "Nick Coghlan" <ncoghlan at gmail.com>
>> Georg and I tried doing it that way and had major problems trying to 
>> get it to work - the hard part is that the body of the list comp 
>> (which may include nested list comps, lambda functions and generator 
>> expressions) needs to be able to see the iteration variables exactly 
>> as they are, but the surrounding code shouldn't be able to see them at 
>> all.
> 
> Did you try just saving and restoring the variable so that hiding 
> wouldn't be necessary.
> 
> [for x in s]
>   -->
>   save(x) if it exists
>   do the listcomp
>   restore(x) if it was saved

And watch any generator expressions or lambda functions returned from 
the body of the list comprehension that reference the iteration variable 
fail inexplicably. Either we revert to 2.x behaviour and leak the 
iteration variable or we keep the extra scope that is added by the 
current implementation. Otherwise it just doesn't work.

> It's a bummer that the genexp implementation was borrowed
> for this purpose.  IMO, what we had before was cleaner,
> faster, obvious, and simple.

Then argue for a reversion to the 2.x behaviour: if we want to continue 
to explain list comprehensions in terms of an equivalent for loop, then 
we need to leak the iteration variables just like a for loop would.

A version which hid the iteration variable from the containing scope but 
didn't share the other scoping characteristics of generator expressions 
would just be strange.

> BTW, the performance cost isn't "possible", it's a guaranteed,
> non-trivial regression.  It becomes a reason to not upgrade to 3.0.

Time it and then say that. For module and class scope code, the 
performance gain from using function locals for the iteration variables 
inside the body of the list comp quickly outstrips the overhead from the 
single function call needed to invoke the implicitly created function.

For list comprehensions that are already at function scope (and hence 
already benefiting from the function local optimisations), I agree the 
new implementation is a guaranteed regression (the overhead of a single 
function call).

> All I'm asking Guido is whether he is open to an alternate patch
> that doesn't create an inner function.

Create the patch and have it pass all the current Py3k list 
comprehension tests and I'll support this. Without code in hand, I 
believe what you're suggesting simply isn't practical to implement (i,e, 
you would have to rewrite large chunks of the compiler to make it work 
properly).

> P.S.  I know this group doesn't care about Psyco, but it was
> nice that psyco could handle listcomps just like it could with
> regular for-loops.  Turning it into a genexp stops psyco in its tracks.
> Likewise, Cython won't be able to handle the semantics.
> ISTM, we've made something hard out of something that
> used to be simple.  What was gained?

Exactly what was asked for: the iteration variable doesn't leak into the 
surrounding scope anymore. The behaviour you're currently objecting to 
is the natural consequence of hiding list comprehension iteration 
variables from the local scope given Python's lexical scoping rules.

So if you want to get "simple" back, then I firmly believe you need to 
argue in favour of continuing to leak the iteration variables from list 
comprehensions.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org