On Tue, Nov 25, 2014 at 12:31 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum <guido@python.org> wrote:
On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker@noaa.gov> wrote:

[...] I like how it support my goal of keeping the behavior of iterator classes and generators consistent.

This is a great summary of the general confusion I am trying to clear up. The behavior of all types of iterators (including generators) from the *caller's* perspective is not in question and is not changing. It is very simple: you call next(it) (or it.__next__(), and it returns either the next value or raises StopIteration (and any other exception is, indeed, an exception).

From the POV of *implementor*, iterator classes and generators have never been at all alike -- one uses *return* to produce a new value, the other uses *yield*. Returning without a value from a __next__() method is seen as producing a None value by the caller; returning from a generator is translated into a StopIteration which will be interpreted by the caller as the end of series.

Once you start nesting these things, the distinction between "implementor" and "caller" gets mingled.

Hm. An implementer of one protocol is likely the caller of many other protocols. It's not clear that calling something that implements the *same* protocol should deserve special status. For example I could be implementing an iterator processing the lines of a CSV file. Inside this iterator I may be using another iterator that loops over the fields of the current line. (This doesn't sound too far-fetched.) But if I run out of fields in a line, why should that translate into terminating the outer iterator? And the outer iterator may itself be called by another, more outer iterator that iterators over a list of files.
And I think this is all about how nested generators behave, yes?

The inner iterator doesn't have to be a generator (apart from send() and throw(), they have the same interface).

And the point of the PEP is that an exhausted inner iterator shouldn't be taken to automatically terminate the outer one.

(You could point out that I don't do anything about the similar problem when the outer iterator is implemented as a class with a __next__() method. If I could, I would -- but that case is different because there you *must* raise StopIteration to terminate the iteration, so it becomes more similar to an accidental KeyError being masked when it occurs inside a __getitem__() method.)
If I am implementing a iterator of some sort (generator function or iterator class), and I call next() inside my code, then I am both a implementor and caller. And if I'm also writing helper functions, then I need to know about how StopIteration will be handled, and it will be handled a bit differently by generators and iterator classes.

A helper function also defines an interface. If you are writing a helper function for a generator (and the helper function is participating in the same iteration as the outer generator, i.e. not in the CSV fies / lines / fields example), the best way to do it is probably to write it as a helper generator, and use "yield from" in the outer generator.

But not a big deal, agreed, probably a much smaller deal that all the other stuff you'd better understand to write this kind of code anyway.

Which I'm sorry to see is much less widely understood than I had assumed.

--Guido van Rossum (python.org/~guido)