[Python-Dev] The iterator story

[Oren Tirosh]

On Fri, Jul 19, 2002 at 04:22:26PM -0700, Neil Schemenauer wrote:
> Neal Norwitz wrote:
> > In what context?  Were you iterating over a file or something else?
> > I'm wondering if this is a problem, perhaps pychecker could generate
> > a warning?
> 
> I was switching between implementing something as a generator and
> returning a list.  I was curious why I was getting different behavior
> until I realized I was iterating over the result twice.  I don't
> think pychecker could warn about such a bug.

That's the scenario that bit me too.  For me it was a little more difficult
to find because it was wrapped in a few layers of chained transformations.
I can't tell by the last element in the chain whether the first one is 
re-iterable or not. 

One approach to solve this is Ka-Ping Yee's proposal to specify in advance
whether you are expecting an iterator or a re-iterable container using
either 'for x in y' or 'for x from y'.  I don't think this will work.
There's already too much code that uses for x in y where y is an iterator.
Another problem is that a transformation shouldn't care whether its upstream 
source is an iterator or an iterable - it's a generic reusable building 
block.

My suggestion (which was rejected by Guido) was to raise an error when an
iterator's .next() method is called afer it raises StopIteration.  This
way, if I try to iterate over the result again at least I'll get and error
like "IteratorExhaustedError" instead something that is indistinguishable
from an iterator of an empty container. I hate silent errors.

This shouldn't be required from all iterator implementers but if all
built-in iterators supported this (especially generators) it would help a
lot to find such errors.

	Oren

P.S. My definition of a transformation is a function taking one iterable
argument and returning an iterator.  It is usually implemented as a
generator function.