[Python-ideas] free variables in generator expressions

[Arnaud Delobelle]

On 12 Dec 2007, at 21:56, Georg Brandl wrote:

> Brett Cannon schrieb:
[...]
>
>>
>> Consider what your genexp, ``(x for x in P if x % p)``, really is::
>>
>>  def _genexp():
>>      for x in P:
>>        if x % p:
>>          yield x
>
> Actually it is
>
> def _genexp(P):
>    for x in P:
>        if x % p:
>            yield x
>
> IOW, the outmost iterator is not a free variable, but passed to the
> invisible function object.


I see. 'P' gets frozen but not 'p', so I should be able to write:

def gen2_sieve(n):
     "Generate all primes less than n"
     P =  xrange(2, n)
     while True:
         for p in P:
             yield p
             P = (lambda p: (x for x in P if x % p))(p)
             break
         else:
             return

 >>> list(gen2_sieve(20))
[2, 3, 5, 7, 11, 13, 17, 19]

It seems to work.  Ok then in the same vein, I imagine that

     (x + y for x in A for y in B)

becomes:

def _genexp(A):
     for x in A:
         for y in B:
             yield x + y

Let's test this (python 2.5):

 >>> A = '12'
 >>> B = 'ab'
 >>> gen = (x + y for x in A for y in B)
 >>> A = '34'
 >>> B = 'cd'
 >>> list(gen)
['1c', '1d', '2c', '2d']

So in the generator expression, A is remains bound to the string '12'
but B gets rebound to 'cd'.  This may make the implementation of
generator expressions more straighforward, but from the point of view
of a user of the language it seems rather arbitrary. What makes A so
special as opposed to B?  Ok it belongs to the outermost loop, but
conceptually in the example above there is no outermost loop.

At the moment I still think it makes more sense for the generator
expressions to generate as much as possible a sequence which is the
same as what the corresponding list comprehension would have been,
i.e.:

l = [f(x, y) for x in A for y in B(x) if g(x, y)]
g = [f(x, y) for x in A for y in B(x) if g(x, y)]
<code, maybe binding A, B, f, g to new objects>
assert list(g) == l

to work as much as possible.

Perhaps I should go and see how generator expressions are generated in
the python source code.

-- 
Arnaud