[872326] generator expression implementation
Hello, Sorry to bring this out again, I didn't find a reference about the following issue in generator expressions: (for x in expr) When should 'expr' be evaluated? A priori, one would expect it to be evaluated immediately, but it is not what the suggested implementation does: def __gen(): for x in expr: yield x __gen() This only computes expr when the first element of the generator is needed (so it may even never be computed). Is it done purposefully? It may also be surprising to see the anonymous generator in tracebacks if an exception is raised within expr. A bientot, Armin.
On Mon, 2004-01-12 at 05:58, Armin Rigo wrote:
Sorry to bring this out again, I didn't find a reference about the following issue in generator expressions:
(for x in expr)
When should 'expr' be evaluated? A priori, one would expect it to be evaluated immediately, but it is not what the suggested implementation does:
def __gen(): for x in expr: yield x __gen()
First a quick clarification: Is the "suggested implementation" SF patch 872326 or is it that implementation sketch in PEP 289? The latter certainly suggests it, but it's hard to tell even then if it's intentional. Perhaps we should work out the formal specification part of the PEP in parallel with the implementation. The PEP currently says that the reference manual should contain a specification, but that spec is supposed to be part of the PEP, too. I agree, at any rate, that the expression should be evaluated immediately. That's easy enough to implement: def __gen(it): for x in it: yield x __gen() Jeremy
Jeremy Hylton <jeremy@alum.mit.edu> writes:
On Mon, 2004-01-12 at 05:58, Armin Rigo wrote:
Sorry to bring this out again, I didn't find a reference about the following issue in generator expressions:
(for x in expr)
When should 'expr' be evaluated? A priori, one would expect it to be evaluated immediately, but it is not what the suggested implementation does:
def __gen(): for x in expr: yield x __gen()
First a quick clarification: Is the "suggested implementation" SF patch 872326 or is it that implementation sketch in PEP 289? The latter certainly suggests it, but it's hard to tell even then if it's intentional. Perhaps we should work out the formal specification part of the PEP in parallel with the implementation. The PEP currently says that the reference manual should contain a specification, but that spec is supposed to be part of the PEP, too.
I agree, at any rate, that the expression should be evaluated immediately. That's easy enough to implement:
def __gen(it): for x in it: yield x __gen()
Should be __gen(expr), right? Cheers, mwh -- For their next act, they'll no doubt be buying a firewall running under NT, which makes about as much sense as building a prison out of meringue. -- -:Tanuki:- -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html
I'm the originator of the patch (SF patch 872326 - generator expression). If "immediate evaluation of expr in generator expression" means that generator expression: (x for x in range()) should be equivalent to: def __foo(_range=range()): for x in _range: yield x it seems to me that generator expression has almost no merit over list comprehension. I think the merit of generator is that it does not need to allocate all the memory that it needs in the initial time, right? Or, am I confusing something? :) Regards, Jiwon. ----- Original Message ----- From: "Michael Hudson" <mwh@python.net> To: <python-dev@python.org> Sent: Tuesday, January 13, 2004 12:55 AM Subject: Re: [Python-Dev] [872326] generator expression implementation
Jeremy Hylton <jeremy@alum.mit.edu> writes:
On Mon, 2004-01-12 at 05:58, Armin Rigo wrote:
Sorry to bring this out again, I didn't find a reference about the following issue in generator expressions:
(for x in expr)
When should 'expr' be evaluated? A priori, one would expect it to be evaluated immediately, but it is not what the suggested implementation does:
def __gen(): for x in expr: yield x __gen()
First a quick clarification: Is the "suggested implementation" SF patch 872326 or is it that implementation sketch in PEP 289? The latter certainly suggests it, but it's hard to tell even then if it's intentional. Perhaps we should work out the formal specification part of the PEP in parallel with the implementation. The PEP currently says that the reference manual should contain a specification, but that spec is supposed to be part of the PEP, too.
I agree, at any rate, that the expression should be evaluated immediately. That's easy enough to implement:
def __gen(it): for x in it: yield x __gen()
Should be __gen(expr), right?
Cheers, mwh
-- For their next act, they'll no doubt be buying a firewall running under NT, which makes about as much sense as building a prison out of meringue. -- -:Tanuki:- -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html
On Mon, 2004-01-12 at 21:53, jiwon wrote:
I'm the originator of the patch (SF patch 872326 - generator expression). If "immediate evaluation of expr in generator expression" means that generator expression:
(x for x in range())
should be equivalent to:
def __foo(_range=range()): for x in _range: yield x
it seems to me that generator expression has almost no merit over list comprehension. I think the merit of generator is that it does not need to allocate all the memory that it needs in the initial time, right? Or, am I confusing something? :)
The range() case isn't all the interesting, because you wouldn't delay the memory allocation very long. The first time an element was consumed from the g.expr it would materialize the whole list. So you would only delay for the likely brief time between creating the generator and consuming its first element. There are two more common cases. One is that the expression is already a fully materialized sequence. Most of the examples in PEP 289 fall into this category: You've got a list of objects and you want to sum their spam_score attributes. The other case is that the expression is itself a generator, e.g. dotproduct = sum(x*y for x,y in itertools.izip(x_vector, y_vector)) For all these cases, the benefit is that the result list is never materialized, not that the input list is never materialized. Jeremy
On Mon, 2004-01-12 at 21:53, jiwon wrote:
I'm the originator of the patch (SF patch 872326 - generator expression). If "immediate evaluation of expr in generator expression" means that generator expression:
(x for x in range())
should be equivalent to:
def __foo(_range=range()): for x in _range: yield x
it seems to me that generator expression has almost no merit over list comprehension. I think the merit of generator is that it does not need to allocate all
ok. I was confused, sorry. :) (I thought range() is evaluated somewhat like xrange() if it's in "for ... in range()" ) I'll fix the patch so that expr is evaluated when generator expression is created. Regards, Jiwon ----- Original Message ----- From: "Jeremy Hylton" <jeremy@alum.mit.edu> To: "jiwon" <jiwon@softwise.co.kr> Cc: "Michael Hudson" <mwh@python.net>; <python-dev@python.org> Sent: Tuesday, January 13, 2004 12:01 PM Subject: Re: [Python-Dev] [872326] generator expression implementation the
memory that it needs in the initial time, right? Or, am I confusing something? :)
The range() case isn't all the interesting, because you wouldn't delay the memory allocation very long. The first time an element was consumed from the g.expr it would materialize the whole list. So you would only delay for the likely brief time between creating the generator and consuming its first element.
There are two more common cases. One is that the expression is already a fully materialized sequence. Most of the examples in PEP 289 fall into this category: You've got a list of objects and you want to sum their spam_score attributes. The other case is that the expression is itself a generator, e.g.
dotproduct = sum(x*y for x,y in itertools.izip(x_vector, y_vector))
For all these cases, the benefit is that the result list is never materialized, not that the input list is never materialized.
Jeremy
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jiwon%40softwise.co.kr
Hello Jeremy, On Mon, Jan 12, 2004 at 10:41:48AM -0500, Jeremy Hylton wrote:
First a quick clarification: Is the "suggested implementation" SF patch 872326 or is it that implementation sketch in PEP 289?
That's both the implementation in PEP 289 and in the latest version of the SF patch.
I agree, at any rate, that the expression should be evaluated immediately.
If nobody disagrees I will change this in the PEP and post a note on the SF patch page. Armin
Sorry to bring this out again, I didn't find a reference about the following issue in generator expressions:
(for x in expr)
When should 'expr' be evaluated? A priori, one would expect it to be evaluated immediately, but it is not what the suggested implementation does:
def __gen(): for x in expr: yield x __gen()
This only computes expr when the first element of the generator is needed (so it may even never be computed). Is it done purposefully? It may also be surprising to see the anonymous generator in tracebacks if an exception is raised within expr.
Looks like a bug to me; I'd like to hear arguments to the contrary. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (6)
-
Armin Rigo -
Guido van Rossum -
Jeremy Hylton -
Jeremy Hylton -
jiwon -
Michael Hudson