PEP 255: Simple Generators

Michael Robin me at mikerobin.com
Tue Jun 26 13:57:31 EDT 2001


It seems to me the following are reasons to use generators:

1.	The routine doing the generation is complex, and possibly
pre-existing. This makes it hard to keep state explicitly between
calls.
2.	Keeping state explicitly (by stuffing & reloading object
attributes, for example) is bound to be slower than yield()'s
implementation of frame-freezing (And if the generator fn is not part
of a class, there may be no convenient place to hold state anyway, but
I guess it has to, so one can override __iter__...)
3.	The number of generated objects is large, so consuming generated
elements avoids the allocation of a large collection to hold all
pre-generated values.
4.	The amount of time to create each new object is relatively large.
Consuming each element as needed "spreads out" computation time, which
could be beneficial in real-time applications. The generator's memory
allocation is only the worst-case for generating the next element.
5.	Every element will be processed in order of generation. If
processing is done out of order, returning a collection may be more
efficient. As the iterator object only supports .next(), skipping
generation through arbitrary indexing or slicing is not supported.

Is all this right? 

A question about canceling generators:
Is the following a correct metaphor for canceling a generator:

def __init__(self):
        self.keepGenerating = 1
def __iter__(self):
        return inorder(self)
def cancel_inorder(self, node):
        self.keepGenerating = 0
def inorder(self, node):
        while self.keepGenerating:
                ...your cool code here
                yield result
        return 
       
This seems to have several problems, among which is loosing many of
the niceties we gained by using yield(). Could this be handled better
by a method/attr on the iterator object itself? (Then again, we don't
normally have an explicit reference to it...) Of course, we can't tell
the difference between a "cancellation" and normal exhaustion of
elements. Is "raise StopIteration(reason)" a reasonable option?
Another option is to cancel the consumer instead. I guess the current
PEP defines narrow class of generator usage and there probably is no
"correct metaphor" for canceling a generator. This assumes another
thread is stopping the generation, so I guess we could be using the
heavyweight thread-based generator module instead in this case.

Async cancelation aside then, if a generator is in-progress and the
consumer exits before all elements are consumed causing the
iterator-object to go out of scope, I assume all the right things
happen and the frames go away with all the references they hold? I'm
sure it does, but I don't think I saw reference to this case in the
PEP.

I look forward to using the new generators when appropriate, and also
hope a more general uthread or continuation-based model also makes
main-stream one day.

thanks,
m

============================================================
Greg Ewing <greg at cosc.canterbury.ac.nz> wrote in message news:<3B37E7B4.AA2A1095 at cosc.canterbury.ac.nz>...
> Andrew Dalke wrote:
> > 
> > Chemist: A what?  What's a generator?
> > Me: It's a function that returns a special kind of object.
> 
> You shouldn't have said "function", you should
> have said "callable object".
> 
> > Me: Well, you can think of a class as a function that returns
> >   a special kind of object.
> 
> You're going way too fast for this guy, given that
> he doesn't know about OO yet. He needs to become
> comfortable with OO concepts first, then you can
> explain iterator objects, *then* you can explain
> what generators do.
> 
> Until then, generators are going to seem like
> black magic no matter what keyword they're defined
> with.



More information about the Python-list mailing list