[Python-3000] Is this a bug with list comprehensions or not?

[Raymond Hettinger]

From: "Nick Coghlan" <ncoghlan at gmail.com>
> Georg and I tried doing it that way and had major problems trying to get it to work - the hard part is that the body of the list 
> comp (which may include nested list comps, lambda functions and generator expressions) needs to be able to see the iteration 
> variables exactly as they are, but the surrounding code shouldn't be able to see them at all.

Did you try just saving and restoring the variable so that hiding wouldn't be necessary.

[for x in s]
   -->
   save(x) if it exists
   do the listcomp
   restore(x) if it was saved


> As for whether or not this is worth fixing, I think getting rid of the current subtle differences between the scoping rules for 
> list comprehensions and generator expressions is a very worthwhile outcome, even at the possible performance cost.

Difference of opinion here.
All 2-to-3 conversions don't automatically work
and when they don't, the reason is subtle, surprising, and weird.
With genexps, you expect function scoping rules to apply
(we explain genexps in terms of an equivalent generator).
With set/list/dict comps, that is unnecessary, slow, and unexpected
(we should be able to explain those in terms of an equivalent
in-lined for-loop with the loop variable restricted to the scope of the loop).
The current code is also surprising and difficult to work with when you run in through pdb.
Genexps are fundamentally different from the other three comps.
It's a bummer that the genexp implementation was borrowed
for this purpose.  IMO, what we had before was cleaner,
faster, obvious, and simple.

BTW, the performance cost isn't "possible", it's a guaranteed,
non-trivial regression.  It becomes a reason to not upgrade to 3.0.

All I'm asking Guido is whether he is open to an alternate patch
that doesn't create an inner function.


Raymond

P.S.  I know this group doesn't care about Psyco, but it was
nice that psyco could handle listcomps just like it could with
regular for-loops.  Turning it into a genexp stops psyco in its tracks.
Likewise, Cython won't be able to handle the semantics.
ISTM, we've made something hard out of something that
used to be simple.  What was gained?






>
> For example, why should this work:
>
> class A(object):
>    v = 8
>    x = [v for i in range(6)]
>
> And these fail:
>
> class A(object):
>    v = 8
>    x = list(v for i in range(6))
>
> class A(object):
>    v = 8
>    x = tuple(v for i in range(6))
>
>> Not sure if the current implementation was really a winning trade. It may have been better to have the minor but easy to explain 
>> nuisance
>> of exposing the induction variable instead of having the harder-to-explain
>> subtleties associated with a slower underlying genexp.
>
> Finding ways to speed up all generator expression invocations would strike me as a more worthwhile use of sprint time rather than 
> creating complicated special cases in the compiler that would benefit only list and set comprehensions.
>
> Cheers,
> Nick.
>
> [1] Guido's original OK of the class variable semantic change:
> http://mail.python.org/pipermail/python-3000/2007-March/006077.html
>
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>             http://www.boredomandlaziness.org