[Python-3000] Change to class construction?

[Nick Coghlan]

Georg Brandl wrote:
> Greg Ewing schrieb:
>> Phillip J. Eby wrote:
>>> This looks like a bug to me.  A list comprehension's local scope 
>>> should be the locals of the enclosing code, even if its loop indexes 
>>> aren't exposed to that scope.
>> It sounds like list comprehensions are being implemented
>> using genexps behind the scenes now.
> 
> That's not true, but the implementation is somewhat similar in that
> the code is executed in its own function context.

Georg is correct. A list comprehension like:

[(x * y) for x in seq1 for y in seq2]

expands to the following in 2.x (% prefixes the compiler's hidden 
variables):

   %n = []
   for x in seq1:
     for y in seq2:
       %n.append(x*y) # Special opcode, not a normal call

In py3k it expands to:

   def <anon>(outermost):
     %0 = []
     for x in outermost:
       for y in seq2:
         %0.append(x*y) # Special opcode, not a normal call
     return %0
   %n = <anon>(seq1)

Python's scoping rules are somewhat tricky - doing it this way means we 
know they are being applied the same way in list and set comprehensions 
as they are applied in generator expressions, even if it isn't quite as 
fast as the 2.x approach to comprehensions.

Another significant benefit from a maintainability point of view is that 
the 3 kinds of comprehension (list, set, genexp) now follow the same 
code path through the compiler, with only minor variations in the 
setup/cleanup code and the statement inside the innermost loop.

>> Is this wise? In a recent thread, I suggested that one
>> of the reasons for keeping the LC syntax was that it
>> could be faster than list(genexp). Has anyone investigated
>> whether any speed is being lost by making them equivalent?
> 
> I don't remember the details, but IIRC the new LC implementation
> was not slower than the 2.x one. Nick should know more about that.

Inside a function, Py3k is slower by a constant amount relative to 2.x 
(the cost of creating and calling a function object) regardless of the 
length of the resulting list/set. At module level, Py3k will typically 
be faster, as the fixed cost from the anonymous function object will be 
overtaken by the speedup from the iteration variables becoming function 
locals instead of module globals.

The Py3k comprehensions are still significantly faster than the 
equivalent generator expressions, as they still avoid suspending and 
resuming a generator for each value in the resulting sequence.

The bit that makes all of this tricky isn't really hiding the iteration 
variables from the containing scope - it's making sure that the body of 
the comprehension can still see them after you have done so 
(particularly challenging if the comprehension itself contains a lambda 
expression, or another comprehension/genexp).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org