[Python-ideas] Generators are iterators

[Andrew Barnert]

On Dec 12, 2014, at 9:16, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>> Okay well this is not clear from the PEP (and bear in mind that I did
>> take the time to read through all of the lengthy threads on this
>> subject before posting here). I haven't read the most recent changes
>> but when I did read the PEP the justification appeared to stem from
>> claims that were confusing at best and plain false at worst e.g.
>> "generators are not iterators".
> 
> I don't think there is much technical confusion here, but there seems to be, so, so that highlights that it's very important to write vary carefully about all this. I"m going to try this:
> 
> 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now.

When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class".

This kind of shorthand is used regularly in discussing types in every OO language, not just Python. You don't see people writing about "an instance of a Type Implementing IEnumerable [or NSFastEnumeration or random_access_iterator]). You just see "an IEnumerable". And to discuss the type itself, you just say "an IEnumerable type". The fact that IEnumerable is itself defined as a type is rarely a source of confusion--and when it is, you say "an IEnumerable subtype".

In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. Especially since it doesn't even help to solve the problem--as we've just seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator". (In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?)

The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people. They want to call it one of the things that has a name. Renaming any of those things that it's not doesn't help to give the thing that it is a name.

> 2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc.
> 
> 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP:
>   1) write a class that has a __iter__ and __next__ methods that "do the right thing"
>   2a) write a generator function
>   2b) write a generator expression
> 
> The latter create a generator type, which is a specific TIIP that come built in to the standard library. Generator functions are a particularly nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is nothing you can do with a generator function that you can't do by writing a custom TIIP class. But generator functions provide really nifty syntax that "does a lot of the bookeeping for you"
> 
> This is pretty much how I've taught this stuff in the past. But what this PEP does is change generators a bit, so that they do even more of the book keeping for you -- that sure seems reasonable enough when phrased that way.
> 
> So I agree with Oscar (and others) here the issue of what happens to a StopIteration that is raised inside a TIIP be another call is an issue that effects not only generators, but also custom TIIP classes. I think is wold be nice if this were addressed in both cases the same way, but that has been proposed and rejected for what I, at least, think are sound technical reasons.
> 
> And the PEP has been accepted so the only thing left to bike-shed about is how it discussed.
> 
> So why it it worth changing the behavior of generators?
> 
> 1) we can -- custom classes are, well, custom, so are free to do what author wants them to do. Generators are built-in classes and can therefor be controlled  by python itself.
> 
> 2) generator function have a different, cleaner, more compact way to express intent. while they create generators that conform to the iteration protocol -- i.e. raise a StopIteration when done, the author does not have to explicitly do that -- you can write a generator function without any awareness of StopIteration ( I might argue that you won't get far if you don't understand all that, but...) So having generators automatically "do the right thing" when they call another TIIP inside themselves conforms well to the spirit of generator functions: clean compact code that does the book keeping for you.
> 
> 3) when you write a custom TIIP class, you have complete control over what it does, and have to be thinking about when StopIteration is raise. So you are more likely to think about how you want to handle a StopIteration that may be raised by another call to a TIP inside your code. You also want to be able to control that depending on circumstances, so probably wouldn't want magic Exception mangling anyway.
> 
> Not sure if this helps the PEP writing, but I've got some good notes for the next time I teach this...
> 
> -Chris
>  
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141212/03bbebb0/attachment-0001.html>