[Python-Dev] Please reconsider PEP 479.

[Mark Shannon]

Hi,

I have serious concerns about this PEP, and would ask you to reconsider it.

[ Very short summary:
     Generators are not the problem. It is the naive use of next() in an 
iterator that is the problem. (Note that all the examples involve calls 
to next()).
     Change next() rather than fiddling with generators.
]

I have five main concerns with PEP 479.
1. Is the problem, as stated by the PEP, really the problem at all?
2. The proposed solution does not address the underlying problem.
3. It breaks a fundamental aspect of generators, that they are iterators.
4. This will be a hindrance to porting code from Python 2 to Python 3.
5. The behaviour of next() is not considered, even though it is the real 
cause of the problem (if there is a problem).

1. The PEP states that "The interaction of generators and StopIteration 
is currently somewhat surprising, and can conceal obscure bugs."
I don't believe that to be the case; if someone knows what StopIteration 
is and how it is used, then the interaction is entirely as expected.

I believe the naive use of next() in an iterator to be the underlying 
problem.
The interaction of generators and next() is just a special case of this.

StopIteration is not a normal exception, indicating a problem, rather it 
exists to signal exhaustion of an iterator.
However, next() raises StopIteration for an exhausted iterator, which 
really is an error.
Any iterator code (generator or __next__ method) that calls next() 
treats the StopIteration as a normal exception and propogates it.
The controlling loop then interprets StopIteration as a signal to stop 
and thus stops.
*The problem is the implicit shift from signal to error and back to signal.*

2. The proposed solution does not address this issue at all, but rather 
legislates against generators raising StopIteration.

3. Generators and the iterator protocol were introduced in Python 2.2, 
13 years ago.
For all of that time the iterator protocol has been defined by the 
__iter__(), next()/__next__() methods and the use of StopIteration to 
terminate iteration.

Generators are a way to write iterators without the clunkiness of 
explicit __iter__() and next()/__next__() methods, but have always 
obeyed the same protocol as all other iterators. This has allowed code 
to rewritten from one form to the other whenever desired.

Do not forget that despite the addition of the send() and throw() 
methods and their secondary role as coroutines, generators have 
primarily always been a clean and elegant way of writing iterators.

4. Porting from Python 2 to Python 3 seems to be hard enough already.

5. I think I've already covered this in the other points, but to 
reiterate (excuse the pun):
Calling next() on an exhausted iterator is, I would suggest, a logical 
error.
However, next() raises StopIteration which is really a signal to the 
controlling loop.
The fault is with next() raising StopIteration.
Generators raising StopIteration is not the problem.

It also worth noting that calling next() is the only place a 
StopIteration exception is likely to occur outside of the iterator protocol.

An example
----------

Consider a function to return the value from a set with a single member.
def value_from_singleton(s):
     if len(s) < 2:  #Intentional error here (should be len(s) == 1)
        return next(iter(s))
     raise ValueError("Not a singleton")

Now suppose we pass an empty set to value_from_singleton(s), then we get 
a StopIteration exception, which is a bit weird, but not too bad.

However it is when we use it in a generator (or in the __next__ method 
of an iterator) that we get a serious problem.
Currently the iterator appears to be exhausted early, which is wrong.
However, with the proposed change we get RuntimeError("generator raised 
StopIteration") raised, which is also wrong, just in a different way.

Solutions
---------
My preferred "solution" is to do nothing except improving the 
documentation of next(). Explain that it can raise StopIteration which, 
if allowed to propogate can cause premature exhaustion of an iterator.

If something must be done then I would suggest changing the behaviour of 
next() for an exhausted iterator.
Rather than raise StopIteration it should raise ValueError (or IndexError?).

Also, it might be worth considering making StopIteration inherit from 
BaseException, rather than Exception.


Cheers,
Mark.

P.S. 5 days seems a rather short time to respond to a PEP.
Could we make it at least a couple of weeks in the future,
or better still specify a closing date for comments.