need help on need help on generator...

[Nick Coghlan]

Francis Girard wrote:
> In particular, I don't know what Python constructs does generate a generator. 
> I know this is now the case for reading lines in a file or with the new 
> "iterator" package. But what else ? Does Craig Ringer answer mean that list 
> comprehensions are lazy ? Where can I find a comprehensive list of all the 
> lazy constructions built in Python ? (I think that to easily distinguish lazy 
> from strict constructs is an absolute programmer need -- otherwise you always 
> end up wondering when is it that code is actually executed like in Haskell).

I don't think there is a *list* as such, but there are some rules of thumb for 
when lazy evaluation will take place (hopefully others will note any cases that 
I missed):

1. Iterators (classes with a next() method, and an __iter__ method that returns 
'self') are lazily evaluated, as itr.next() is called to retrieve each value. I 
think you will find it is this method, rather than __new__ which is relevant to 
creating class-based generators. Note that "for x in itr" automatically calls 
itr.next() in order to obtain each new value of the loop variable.
   This iterator protocol is the basis of lazy evaluation in Python, and is 
described here:
   http://www.python.org/dev/doc/devel/lib/typeiter.html

2. Iterables (classes with an __iter__ method) will return a lazy iterator via 
iter(obj). Actual iterators return themselves from __iter__, so iter(obj) is a 
good way to make sure you have an iterator.

3. Generators (functions that use 'yield' instead of 'return') and generator 
expressions (like list comprehensions, but without the square brackets) are 
simply concise ways of creating iterators.

4. The utility functions in the itertools module generally return iterators 
rather than lists (this shouldn't suprise anyone!)

5. Several builtin functions return iterators rather than lists, specifically 
xrange(), enumerate() and reversed(). Other builtins that yield sequences 
(range(), sorted(), zip()) return lists.

However, be aware that some things which accept any iterable don't take 
advantage of the lazy evaluation, and still cause the whole thing to be created 
in memory at once - "".join(itr) is currently one such operation.

The sequence vs iterator distinction is somewhat unfortunate (since it breaks 
with TOOWTDI), but completing the transition to an iterator based approach isn't 
going to be possible until Python 3.0, when things that currently return lists 
can be converted to return iterators (i.e. it has been suggested that the 
fundamental construct in Python 3.x should be an iterator just as a list is the 
fundamental construct in Python 2.x)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net