[Python-ideas] should there be a difference between generators and iterators?

[Guido van Rossum]

The proposal to extend all iterators with an additional protocol goes
directly against one of the design goals for iterators, i.e., that it
should be easy to implement a new iterator, without subclassing
something. So any proposal that wants to add new methods to *all*
iterators (let alone all iterables, which is a much larger set -- read
abc.py in the 3.0 stdlib for the difference) is doomed. That said,
having *optional* additions to the protocol might still be reasonably
debated. On the 3rd hand, I'd like to understand more about your use
case.

On Thu, Sep 4, 2008 at 7:58 AM, Bruce Frederiksen <dangyogi at gmail.com> wrote:
> There are several points here that might or might not be adopted
> individually.
>
> The specific prompt for this message is that I've run into a snag using
> generators with 'itertools.chain.from_iterable'.  I need to be able to
> control when the generators called by 'chain' get closed (so that their
> 'finally' clauses are run).
>
> Unfortunately, chain does not propagate 'close' (or 'throw' or 'send') back
> to the generators that it's using.
>
> Nor, for that matter do any of the other tools in itertools, or the builtin
> map function (to my knowledge).
>
> I propose the following.  Each of these is independent of the others, but
> all related to cleaning up how this works:
>
> 1.  All of the itertools and map (and I've no doubt left some others out
> here) be extended to propagate the extra generators methods: close, throw
> and send.
>
> 2.  That the 'for' loop be extended so that if an exception occurs within
> its body, it calls 'throw' on its iterable (if it has a throw method).
>
> 3.  That the 'for' loop be extended to call the 'close' method on its
> iterable (if it has a close method) when the loop terminates (either
> normally, with break, or with an exception).
>
> 4.  That a 'not_closing' builtin function be added that takes an iterable
> and shields it from a 'close' call.
>
> 5.  That 'close' and 'throw' be added to all iterables.
>
>
> Motivation:
>
> The motivation for each proposal is (by their number):
>
> 1.  This is the one that I'm specifically stuck on.  I was relying on
> garbage collection to do this, but this doesn't work in jython and
> ironpython...  Since the chain function is dealing with two iterables (an
> inner and outer iterable), I think that it makes sense for it to check
> whether each of these have the extra methods or not.  For example, the inner
> iterable may have a 'close', but not the outer iterable (or vise versa).
>  This shouldn't cause an error if 'close' is called on the chain.
>
> This one, specifically, would be helpful for me to move to Python 3K; so the
> sooner the better!  (Please!)
>
> 2.  There has been some discussion here about extending 'for' loops that has
> touched on non-local continue/break capability.  If step 2 is provided, this
> capability could be provided as follows:
>
>   class funky:
>       ...
>       def continue_(self): raise ContinueError(self)
>       def break_(self): raise BreakError(self)
>       def throw(self, type, value, tb):
>           if issubclass(type, ContinueError) and value.who is self:
>               return next(self)
>           if issubclass(type, BreakError) and value.who is self:
>               raise StopIteration
>
>   top = funky(iterable1)
>   for x in top:
>       middle = funky(iterable2)
>       for y in middle:
>           bottom = funky(iterable3)
>           for z in bottom:
>               ...
>               middle.continue_()
>
> 3.  But this is a problem with:
>
>    for line in filex:
>        if test1(line): break
>    for line in filex:
>        ...
>
>    which brings us to:
>
> 4.  A solution to the above:
>
>    for line in not_closing(filex):
>        if test1(line): break
>    for line in filex:
>        ...
>
> 5.  I thought that I may as well throw this in for discussion...  This might
> cause some consternation to those who has written their own iterables...
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)