[Python-ideas] Generators are iterators

[Oscar Benjamin]

On 14 December 2014 at 15:02, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 13 December 2014 at 09:15, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>> I'm just saying that it's a mistake that can easily be made. Since I
>> made it I'm now super-suspicious of next() so I probably wouldn't make
>> it again. This mistake is the root of all of the problems that PEP 479
>> attempts to solve. If you don't believe it is a problem that someone
>> might make this mistake then it follows that PEP 479 is a massive
>> mistake.
>
> Oscar, StopIteration unexpectedly escaping from __next__ methods is
> *not* the problem PEP 479 is trying to solve, and that
> misunderstanding appears to be the fundamental error causing your
> concern. PEP 479 is aiming to solve a specific problem with *writing
> generator functions*, not the more general problem with invoking
> iterators that bothers you. There is nothing to be done about
> StopIteration escaping from __next__ methods in general - it's an
> inherent limitation of the iterator protocol. StopIteration escaping
> from generator frames is a different story, as the generator-iterator
> __next__ method implementation has the chance to intercept them and
> convert them to something else.

Fair enough. I understand your point that the intention of the PEP is
only to solve this for generators. I also understand that these
threads are not working towards anything so this is my last post on
the subject.

I see nothing wrong with the rationale that "it's easier to to address
this problem for generators" but that's very different from the "this
problem is unique to generators" argument. Since the start of this
thread the PEP is now modified so that it no longer explicitly makes
the latter argument (although it still does not explicitly state the
former).

> So let me try again from the pedagogical angle in a post PEP 380 Python world.
>
> If you are teaching someone to write explicit __next__ methods, here
> are the things you *absolutely* need to teach them:
>
> 1. Values returned from the method are produced as values in the iterator
> 2. Raising StopIteration terminates the iterator
> 3. If invoked via "yield from", the argument to StopIteration is the
> result of the "yield from" expression (default: None)
> 4. If delegating to a function via a function call, points 2 & 3 also
> apply to that subfunction (this includes the builtin next() function)
>
> If you are instead teaching someone to write generator functions, here
> are the things you *absolutely* need to teach them:
>
> 1. Values passed to yield expressions are produced as values in the iterator
> 2. Returning from the frame terminates the iterator
> 3. If invoked via "yield from", the return value of the generator
> frame is the result of the "yield from" expression (default: None)
> 4. If delegating to a subgenerator (rather than to an arbitrary
> iterator) via a "yield from" expression, then next(), send() and
> throw() are passed to that subgenerator until it terminates, at which
> point execution resumes in the parent generator frame (either
> producing a value from the yield from expression, or raising an
> exception if the subgenerator failed)
>
> With just those 8 points (and their existing knowledge regarding how
> to *use* iterators and generators), a developer is now equipped to
> write full featured __next__ method and generator function
> implementations, including delegation to subiterators and
> subgenerators via "yield from" in the latter case.
>
> Unfortunately, in both Python 3.3 and 3.4, there's a problem with
> these instructions: they leave something out. The thing they leave out
> is the fact that points 2-4 from the section on __next__ methods
> *also* apply to writing generator functions. Even though it's
> completely unnecessary, and usually not the right approach if only
> supporting modern versions of Python, new users still need to be
> taught it, because it can have unexpected side effects on the way
> their generators work.

I'm not sure why you single out 3.3 and 3.4 as it happens without
yield from. In 2.7 (and anything since 2.2):

>>> def f():
...     yield 1
...     yield 2
...     raise StopIteration
...     yield 3
...
>>> list(f())
[1, 2]

I will leave this subject now.


Oscar