[Patches] [ python-Patches-872326 ] generator expression
implementation
SourceForge.net
noreply at sourceforge.net
Fri Jan 30 00:09:09 EST 2004
Patches item #872326, was opened at 2004-01-07 21:10
Message generated for change (Comment added) made by perky
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470
Category: Parser/Compiler
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Jiwon Seo (jiwon)
Assigned to: Hye-Shik Chang (perky)
Summary: generator expression implementation
Initial Comment:
Since I was interested in pep 289(generator
expression), I dabbled with it, and implemented a
working version of it. I'm not sure if I did it right,
but maybe someone who knows better can fix it right.
1. Grammar has two changes, which is
a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ...
changes to
atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ...
where testgenexpr defines like this.
testgenexpr: test ( gen_for | (',' test)* [','] )
b. argument: [test '='] test
changes to
argument: [test '='] test [gen_for]
(gen_for, gen_if, gen_iter is similar to list_for,
list_if, list_iter respectively.)
2. Instead of changing rule of arglist in Grammar to
accept generator expression, I changed argument rule
like 1.b. This means Grammar accepts generator
expression without parenthesis in function call even
there are several arguments, like
reduce(operator.add, (x for x in range(10)))
This is against what pep requires, so I made
parsermodule.c and compile.c to catch that and throw
error message when there's more than one argument in a
function. The reason I did it that way is to go around
a problem of ambiguity in the grammar by adding
generator expression to arglist.
3. I implemented generator expression as equivalent to
a call to a function which has variable capturing code
and generator code. For example,
x = 1
g = (x for i in range(10))
is equivalent to
x = 1
def __gen_wrapper():
_[x] = x # variable capture here
def __gen():
for i in range(10):
yield _[x]
return __gen()
g = __gen_wrapper()
4. Since I implemented generator expression equivalent
to a function call, symbol table should enter new scope
when it meets generator expression. So I ended up with
adding following code to symtable.c
PyObject *
PySymtableEntry_New(struct symtable *st, char *name,
int type, int lineno)
{
...
switch (type) {
case funcdef:
case lambdef:
! case testgenexpr: /* generator expression */
! case argument: /* generator expression */
ste->ste_type = TYPE_FUNCTION;
break;
...
so that generator expression can be dealt as function
type. This is kind of stupid, but I couldn't find other
easy for it, so I just left it like that.
5. this patch does not include diff of
src/Lib/symbol.py, so you must run python Lib/symbol.py
to get it right.
----------------------------------------------------------------------
>Comment By: Hye-Shik Chang (perky)
Date: 2004-01-30 14:09
Message:
Logged In: YES
user_id=55188
Yet another Fatal Python error;
>>> (a for a in (b for b in (a for a in 'xxx' if True) if
False) if True)
lookup 'True' in ? 3 -1
freevars of <generator expression>: ('True',)
Fatal Python error: com_make_closure()
zsh: abort (core dumped) ./python
# applied patch as of 2004-01-30
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-30 12:01
Message:
Logged In: YES
user_id=595483
ok, I've fixed the bug quiver commented, and added related
test code to test_grammar.py.
----------------------------------------------------------------------
Comment By: George Yoshida (quiver)
Date: 2004-01-29 20:48
Message:
Logged In: YES
user_id=671362
yet another Fatal Python error;
>>> (a for a in (b for b in (0,1)))
<generator object at 0x40170f8c>
>>> (a for a in (b for b in range(2)))
Fatal Python error: unknown scope for range in ?(0) in <stdin>
symbols: {'range': 8}
locals: {}
globals: {}
Aborted
# applied patch as of 2004-01-27
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-27 16:44
Message:
Logged In: YES
user_id=595483
I fixed the patch for the bug that arigo mentioned, and for
what perky mentioned - PyList_GetSlice error handling - .
now, I'll tackle python compiler package :)
----------------------------------------------------------------------
Comment By: Armin Rigo (arigo)
Date: 2004-01-27 00:15
Message:
Logged In: YES
user_id=4771
>>> (a for d in a)
Fatal Python error: unknown scope for a in <generator
expression>(1) in <stdin>
symbols: {'a': 2048, '_[1]': 4, 'd': 2}
locals: {'_[1]': 0, 'd': 1}
globals: {}
----------------------------------------------------------------------
Comment By: Hye-Shik Chang (perky)
Date: 2004-01-26 15:17
Message:
Logged In: YES
user_id=55188
Please ignore the previous comment.
It was a result from an old revision of patches. It works on
the recent.
----------------------------------------------------------------------
Comment By: Hye-Shik Chang (perky)
Date: 2004-01-26 08:44
Message:
Logged In: YES
user_id=55188
BTW, does unavaliability of yield (genexpr) and return
(genexpr) an intended behavior?
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2004-01-22 03:13
Message:
Logged In: YES
user_id=6656
Documentation would be nice!
----------------------------------------------------------------------
Comment By: Hye-Shik Chang (perky)
Date: 2004-01-21 23:14
Message:
Logged In: YES
user_id=55188
Okay. My review is done. The revised patch "gexp.diff" looks fine
for me.
There're few minor tweaks still needed to get into CVS.
- Some error exit cases are ignored. You need to provide safe exit
paths for MemoryError. (eg. PyList_GetSlice usage of Python/
compiler.c)
- A patch for 'compiler' pure python package. (there's an old
implementation on SF #795947)
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-19 20:10
Message:
Logged In: YES
user_id=595483
ok. I modified the patch so that it evaluates iterator expr in
generator expression creation time. That means,
g = (x for x in range(10))
is equivalent to
def __gen(iter1):
for x in iter1:
yield x
g = __gen(range(10))
so, evalution of range(10) is not deferred anymore.
I also changed testgenexpr to testlist_gexp in
Grammar/Grammar, since Hye-Shik Chang advised as such.
I'm willing to take any advice about this patch, so please do.
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-12 12:26
Message:
Logged In: YES
user_id=595483
ok. I've implemented capturing variables part as arigo
suggested. File genexpr-capture-vars-in-args.diff is that.
If you look at the code, you'll find that the code for
generator expression is much shorter, and there's lesser
modification in the existing code.
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-09 10:49
Message:
Logged In: YES
user_id=595483
What arigo wrote sounds reasonable, and not very difficult
to implement. I'll try to do that way.
----------------------------------------------------------------------
Comment By: Armin Rigo (arigo)
Date: 2004-01-09 03:50
Message:
Logged In: YES
user_id=4771
We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes:
x = 1
def __gen(x):
for i in range(10):
yield x
g = __gen(x)
I don't know though if this is easy to implement in compile.c. Alternatively:
x = 1
def __gen(x=x):
for i in range(10):
yield x
g = __gen()
----------------------------------------------------------------------
Comment By: Hye-Shik Chang (perky)
Date: 2004-01-08 14:44
Message:
Logged In: YES
user_id=55188
Okay. I verified that basic characteristics mentioned on PEP
are working.
I started to review the implementation.
----------------------------------------------------------------------
Comment By: Jiwon Seo (jiwon)
Date: 2004-01-08 13:50
Message:
Logged In: YES
user_id=595483
I added diff of Lib/symbol.py, so no need to run python
Lib/symbol.py now.
----------------------------------------------------------------------
Comment By: Hye-Shik Chang (perky)
Date: 2004-01-07 21:25
Message:
Logged In: YES
user_id=55188
Assigned to me.
The originator is my friend and I have much interest on
this. :-)
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470
More information about the Patches
mailing list