[Patches] [ python-Patches-872326 ] generator expression implementation

[SourceForge.net]

Patches item #872326, was opened at 2004-01-07 21:10
Message generated for change (Comment added) made by jiwon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470

Category: Parser/Compiler
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Jiwon Seo (jiwon)
Assigned to: Hye-Shik Chang (perky)
Summary: generator expression implementation

Initial Comment:
Since I was interested in pep 289(generator
expression), I dabbled with it, and implemented a
working version of it. I'm not sure if I did it right,
but maybe someone who knows better can fix it right.

1. Grammar has two changes, which is
 a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ...

     changes to 

    atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ...

     where testgenexpr defines like this.
 
    testgenexpr: test ( gen_for | (',' test)* [','] )

 b. argument: [test '='] test
 
     changes to

    argument: [test '='] test [gen_for]

 (gen_for, gen_if, gen_iter is similar to list_for,
list_if, list_iter respectively.)

2. Instead of changing rule of arglist in Grammar to
accept generator expression, I changed argument rule
like 1.b. This means Grammar accepts generator
expression without parenthesis in function call even
there are several arguments, like

reduce(operator.add, (x for x in range(10))) 

This is against what pep requires, so I made
parsermodule.c and compile.c to catch that and throw
error message when there's more than one argument in a
function. The reason I did it that way is to go around
a problem of ambiguity in the grammar by adding
generator expression to arglist.


3. I implemented generator expression as equivalent to
a call to a function which has variable capturing code
and generator code. For example,

x = 1
g = (x for i in range(10))

is equivalent to

x = 1
def __gen_wrapper():
    _[x] = x # variable capture here
    def __gen():
        for i in range(10):
            yield _[x]
    return __gen()

g = __gen_wrapper()

4. Since I implemented generator expression equivalent
to a function call, symbol table should enter new scope
when it meets generator expression. So I ended up with
adding following code to symtable.c

PyObject *
PySymtableEntry_New(struct symtable *st, char *name,
int type, int lineno)
{

...
        switch (type) {
        case funcdef:
        case lambdef:
!       case testgenexpr: /* generator expression */
!       case argument:    /* generator expression */
                ste->ste_type = TYPE_FUNCTION;
                break;
...

so that generator expression can be dealt as function
type. This is kind of stupid, but I couldn't find other
easy for it, so I just left it like that.

5. this patch does not include diff of
src/Lib/symbol.py, so you must run python Lib/symbol.py
to get it right.

----------------------------------------------------------------------

>Comment By: Jiwon Seo (jiwon)
Date: 2004-01-08 13:50

Message:
Logged In: YES 
user_id=595483

I added diff of Lib/symbol.py, so no need to run python
Lib/symbol.py now.

----------------------------------------------------------------------

Comment By: Hye-Shik Chang (perky)
Date: 2004-01-07 21:25

Message:
Logged In: YES 
user_id=55188

Assigned to me.
The originator is my friend and I have much interest on
this. :-)


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470