generator expressions: performance anomaly?
steve at holdenweb.com
Tue Jan 18 07:12:18 EST 2005
Antoon Pardon wrote:
> Op 2005-01-18, Nick Coghlan schreef <ncoghlan at iinet.net.au>:
>>Raymond Hettinger wrote:
>>>[Delaney, Timothy C]
>>>>Nick's other suggestion - that genexps propagate __len__ - might
>>>>still be interesting. Of course, it would only be applicable for
>>>>unconditional genexps(i.e. no if clause).
>>>Length transparency for iterators is not as general as one would expect. I once
>>>spent a good deal of effort exploring where it made sense, and I was surprised
>>>to find that it only rarely works out. Length transparency is an unexpectedly
>>>thorny subject with many dead-ends which precludes a fully general solution such
>>>as that proposed by Nick.
>>>For a recap of my research, see the docstring for Lib/test/test_iterlen.py .
>>"""The situation slightly more involved whenever an object allows length
>>mutation during iteration. """
>>Ouch. Nice understatement.
>>It's rather unfortunate that we can't make use of the length information even
>>when the source *doesn't* mutate, though. I'll have to think some more to see if
>>I can come up with any concrete ideas for you to shoot down :)
> Something else I was thinking about. I think it would be nice if the
> python compilor could figure out whether a genexp in a list or tuple
> expression always generates the same list or tuple and then instead
> of generating code would generate the list or tuple in place.
Since it doesn't yet optimize 2+5 to a constant-folded 7 you should
realize that you are suggesting a large increase in the compiler's
I agree it would be nice under certain circumstances, but don't forget
that unlike list comprehensions (for which it would be even nicer) the
whole point of generator expressions is often to defer the generation of
the individual items until they are required and thereby relieve stress
As an edge case to demonstrate the point, what about a constant but
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
More information about the Python-list