+1 On Wed, Dec 10, 2014 at 8:06 AM, Oscar Benjamin wrote:

Somehow during the discussion of PEP 479 it was concluded that generators will become distinct from iterators. A few people have pointed out that this is a harmful distinction (complicates the language hard to explain etc). I think it is simply an invalid distinction and should be abolished.

PEP 479 proposes to change the way that generators handle StopIteration fall-through by default. However the iterator protocol has never formally specified that an iterator should allow StopIteration to fall through to a grandparent consumer. I considered the fact that StopIteration fall-through was possible to be a convenient design feature but it was never a required part of the definition of an iterator.

Many iterators perform other kinds of exception handling without allowing that to fall through and many iterators catch StopIteration from child iterators without reraising it. This has never lead anyone to suggest that such iterators are not true iterators in any way.

Under PEP 479 generators will handle a StopIteration arising from the executing frame differently. The generator will still be an object with __iter__ and __next__ that exposes StopIteration at the appropriate time to its parent iterator-consumer. In other words it will still satisfy the definition of an iterator.

The PEP says:

""" Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """"

This is just plain false. Under the proposal generators will still be iterators. Mixing generators and iterators is nothing like mixing text and bytes and never has been. Mixing iterators and iterables is a bit like mixing text and the bytes in the sense that it can seem to work but then sometimes silently do the wrong thing. Mixing generators and iterators is like mixing sets and containers: there is no mixing since it is simply a false distinction.

AFAICT from reading through the discussions this idea has (implicitly) followed from the following fallacious argument:

1) Bare next creates a problem for generators. 2) Therefore we fix it by changing generators. 3) Therefore generators are not iterators any more.

Anyone with experience in propositional logic can see multiple fallacies in that argument but actually the biggest mistake is simply in the opening premise: bare next is a problem for all iterators. Generators are affected precisely because they are iterators.

Generators were introduced as "a kind of Python iterator, but of an especially powerful kind" (PEP 255). The coroutine idea has since turned generators into something of a Frankenstein concept but the fundamental fact that they are iterators remains unchanged. And regardless of how much discussion coroutines generate the fact remains that 99% of generators are used purely for iteration.

I propose to abolish this notion that generators are not iterators and to amend the text of the PEP to unambiguously state that generators are iterators regardless of any changes to the way they propagate StopIteration from the executing frame.

Oscar _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Juancarlo *Añez*

On 11 December 2014 at 00:18, Steven D'Aprano wrote:

If generators are to cease to be iterators, what will they be?

What the PEP is trying to say is that generators are not the same thing as __next__ method implementations (and I agree that shortening the latter to "iterators" is incorrect). __next__ method paths for leaving the frame: * "return value" = produce next value * "raise StopIteration" = end of iteration Current generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = end of iteration PEP 479 generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = RuntimeError The only change is to eliminate the redundant method of leaving the frame, as that behaviour means that adding a "yield" to a function *implicitly* suppresses StopIteration exceptions raised elsewhere in that frame. That implicit side effect on exception handling behaviour doesn't exist for __next__ method implementations, or for ordinary functions used in iteration operations that accept arbitrary callables, as ordinary functions won't include any yield expressions (by definition). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 10 Dec 2014 16:38, "Chris Angelico" wrote:

Would it be better to clarify that as "generator functions"? Maybe something like this:

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies. -n

Everyone (including Chris), hold your horses. I will come up with better words for the PEP. I apologize for the confusion. On Wed, Dec 10, 2014 at 9:37 AM, Nathaniel Smith wrote:

On 10 Dec 2014 16:38, "Chris Angelico" wrote:

...

Would it be better to clarify that as "generator functions"? Maybe something like this:

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies.

-n

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

I have replaced the confusing paragraph with this: """ The proposal does not change the relationship between generators and iterators: a generator object is still an iterator, and not all iterators are generators. Generators have additional methods that iterators don't have, like ``send`` and ``throw``. All this is unchanged. Nothing changes for generator users -- only authors of generator functions may have to learn something new. """ Let us know if there is still confusion about what the PEP means. --Guido On Wed, Dec 10, 2014 at 10:13 AM, Guido van Rossum wrote:

Everyone (including Chris), hold your horses. I will come up with better words for the PEP. I apologize for the confusion.

On Wed, Dec 10, 2014 at 9:37 AM, Nathaniel Smith wrote:

...

On 10 Dec 2014 16:38, "Chris Angelico" wrote:

...

Would it be better to clarify that as "generator functions"? Maybe something like this:

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies.

-n

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

-- --Guido van Rossum (python.org/~guido)

Chris Angelico wrote:

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

No, that's still too confused -- all of that was true before as well. Generator functions have never been iterators themselves. I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method. -- Greg

On Thu, Dec 11, 2014 at 8:34 AM, Greg Ewing wrote:

I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method.

Hang on. When would code inside a generator function implement __next__? A generator function is a valid way of implementing __iter__, not __next__. ChrisA

On Wed, Dec 10, 2014 at 4:34 PM, Greg Ewing wrote:

Chris Angelico wrote:

...

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

No, that's still too confused

I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function. It is a callable that returns a generator. To our excuse, this abuse of language is not uncommon. People often say "integer function" when they mean a function with integer domain and range (or even just range).

On Dec 10, 2014, at 14:22, Chris Angelico wrote:

On Thu, Dec 11, 2014 at 9:11 AM, Alexander Belopolsky wrote:

...

I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function. It is a callable that returns a generator. To our excuse, this abuse of language is not uncommon. People often say "integer function" when they mean a function with integer domain and range (or even just range).

Interesting. Are you saying the 'def' statement may produce a thing that isn't a function, even though it calls itself a function?

...

...

...

def gen(): ... yield 1 ... type(gen) type(gen())

It's a function that returns an iterator, same as this is:

...

...

...

def nongen(): ... return iter((1,2,3)) ... type(nongen) type(nongen())

A generator function is a function that returns a generator object. Is this problematic?

def whatisthis(): return gen() That's not a generator function, but it is a function that returns a generator. (That's a bit of a silly case, but there's plenty of real-life code like this. In fact, IIRC, a couple of the functions in itertools are generator functions in the pure Python implementation but regular functions that return iterators in the C implementation.)

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 12/10/2014 4:37 PM, Guido van Rossum wrote:

On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing mailto:greg.ewing@canterbury.ac.nz> wrote:

Chris Angelico wrote:

""" Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """

No, that's still too confused -- all of that was true before as well. Generator functions have never been iterators themselves.

I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method.

That's hardly news though. *Except* for the possibility of raising StopIteration to end the iteration, the inside of a generator has always been completely different from the inside of a __next__ method implementation.

Perhaps some Python equivalent of the C code will make it clearer what will and (mostly) will not change. Correct me if I am wrong, but I believe the function __call__ method must do something more or less like the following. class function: ... def __call__(self, *arg, **kwargs): "Setup execution frame and either run it or pass to generator." frame = execution_frame(self, args, kwargs) # several attributes of self are used to setup the frame if no_yield(self.code): run(frame) # until it stops return top(frame.stack) else: return generator(self.code, frame) The generator must look something like the following. class generator: def __iter__(self): return self def __init__(self, code, frame): self.gi_code = code self.gi_frame = frame self.gi_running = False def __next__(self): "Adapt gen-func yield, return to iterator return, raise S.I." self.gi_running = True run(self.gi_frame) if yielded(): return top(self.gi_frame.stack) else: # returned raise StopIteration() self.gi_running = False ... (close, send, and throw methods, not relevant here) (I probably have some details of gi_running wrong, and ignored how exhausted generator calls are handled.) Ignoring gi_code, which I believe is only there for introspection (such as gi_code.__name__, which I use for error-reporting), the link between the generator function and the generator's pre-written __next__ method is the suspended generator-function frame, stored as gi_frame. This is what make one generator instance different from another. Currently, a StopIteration raised during 'run(self.gi_frame)' is passed on like any other exception, but treated specially by the generator caller. The change is to wrap the frame execution or do the equivalent thereof. try: run(self.gi_frame) except StopIteration() raise RuntimeError All else remains the same. Do I have this right? The rationale for the change is the bulk of the PEP. However, the PEP rationale and its acceptance are based on the existing relationship between the execution frame of the gf body and the generator.__next__ method that translates the 'gf protocol' to the iterator protocol. Guido, Nick, and Greg have all pointed to this, but for me, thinking in terms of Python equivalents makes it clearer. -- Terry Jan Reedy

On 10 December 2014 at 20:46, Guido van Rossum wrote:

I have replaced the confusing paragraph with this:

""" The proposal does not change the relationship between generators and iterators: a generator object is still an iterator, and not all iterators are generators. Generators have additional methods that iterators don't have, like ``send`` and ``throw``. All this is unchanged. Nothing changes for generator users -- only authors of generator functions may have to learn something new. """

Generators are affected and this therefore affects authors of both generator functions and generator expressions. In the case of a generator function you can achieve the old behaviour post-PEP 479 simply by wrapping the entire body of the function with: try: # body except StopIteration as e: return e.value The only difference is that the generated StopIteration has a different traceback attached so you can't see where it was originally "raised". Generator expressions would need to be converted to generator functions to achieve the same effect.

Let us know if there is still confusion about what the PEP means.

The PEP is still very confused. Primarily the fact is that the PEP is attempting to address an issue which affects all iterators but proposing a solution which is only about generators. The PEP seems to suggest that there is something special about generators in this respect when there really isn't. For example: """ The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration can be absorbed by the generator construct. """ There is no interaction between generators and StopIteration. The issue isn't about generators it is about the iterator protocol. StopIteration cannot be absorbed by the generator construct. I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special.

On 12/11/2014 10:14 AM, Oscar Benjamin wrote:

On 10 December 2014 at 20:46, Guido van Rossum wrote:

The PEP is still very confused. Primarily the fact is that the PEP is attempting to address an issue which affects all iterators but proposing a solution which is only about generators. The PEP seems to suggest that there is something special about generators in this respect when there really isn't. For example:

""" The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration can be absorbed by the generator construct. """

There is no interaction between generators and StopIteration. The issue isn't about generators it is about the iterator protocol. StopIteration cannot be absorbed by the generator construct.

What is very confusing is the ambiguous use of 'generator' to mean either 'generator function' or 'generator class instance'. Since I do not know what either of you mean by 'generator' in each instance above, I do not know exactly what either of you meant. If the PEP and discussants consistency used 'generator' to mean "instance of the internal generator class, initialized with a suspended instance of a *user-written* generator function body", then I think some of the confusion would dissipate. The PEP is about the conjunction of the following facts (and a few more, but I think these are the most salient): a) generator function bodies follow a different protocol than iterator __next__ methods, one that does not require them to *ever* raise StopIteration; b) if a g.f. body does raise StopIteration, it might be a substitute for 'return', but it might be a bug -- and apparently bugs do happen in real code; and c) generator.__next__ currently trusts that *user-written* g.f.s are never buggy. The PEP proposal is, either literally or in effect, that generator.__next should stop trusting StopIteration from a g.f., thereby disallowing the sometimes convenient but optional substitution. The justification is that bugs should not pass silently, and now they do.

I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special.

Guido today add the following, apparently in response to the above. +When implementing a regular ``__next__()`` method, the only way to +indicate the end of the iteration is to raise ``StopIteration``. So +catching ``StopIteration`` here and converting it to ``RuntimeError`` +would defeat the purpose. This is a reminder of the special status of +generator functions: in a generator function, raising +``StopIteration`` is redundant since the iteration can be terminated +by a simple ``return``. I would put this answer slightly differently. A __next__ method is supposed to raise StopIteration if *and only if* there are no more items to return. The __next__ method author is responsible for fulfilling the contract. Core developers are responsible for generator.__next__; we are not responsible for iterators that others write. Anyone who writes an iterator class whose instances are initialized with 3rd party non-iterator code that is executed in .__next__, should think about what to do if that code raises StopIteration. It would be possible for the PEP to recommend that other .__next__ methods executing external code might do something similar to what generator.__next__ will do. try: execute(self.external_code) except StopIteration as e: raise RuntimeError(<whatever>) -- Terry Jan Reedy

On 12 December 2014 at 10:34, Nick Coghlan wrote:

On 12 December 2014 at 01:14, Oscar Benjamin wrote:

...

I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special.

I believe you're misunderstanding the problem being solved.

And I believe you are too. :)

The specific problem deemed worthy of being fixed is that the presence of "yield" in a function body can implicitly suppress StopIteration exceptions raised elsewhere in that function body (or in functions it calls).

The yield causes the function to become a generator function. The frame for a generator function (like for any other function) will allow uncaught exceptions to propagate to the frame above. The difference between generator functions and other functions is that the code in the body of a generator function is executed when someone calls the generator's __next__ method. Since the caller (the iterator consumer) is expecting StopIteration it is treated as the signalling the end of iteration. The yield suppresses nothing; it is the iterator consumer e.g. the for-loop or the list() function etc. which catches the StopIteration and treats it as termination.

The difference in behaviour between comprehensions and generator expressions when it comes to embedded function calls that trigger StopIteration is a special case of that more general difference.

I don't know what you mean by this.

This is a problem unique to generators, it does not affect any other iterator (since explicit __next__ method implementations do not use yield).

Incorrect. The problem is not unique to generators and the yield is irrelevant. The problem (insofar as it is) is a problem for all iterators since all iterators interact with iterator-consumers and it is the iterator-consumer that catches the StopIteration. Here is an example using map:

...

...

def func(x): ... if x < 0: ... raise StopIteration ... return x ** 2 ... it = map(func, [1, 2, 3, -1, 2]) list(it) [1, 4, 9] list(it) # map continues to yield values... [4]

Ordinarily map should terminate when the underlying iterator raises StopIteration. Since it simply allows a StopIteration raised anywhere to bubble up it will also allow the StopIteration raised by func to bubble up. Implicitly the effect of that is that map agrees to cooperatively terminate iteration when any part of the code that it executes raises StopIteration. Note that it is the list() function (the iterator-consumer) that actually catches the StopIteration and terminates just as it would be if map were a generator:

...

...

def map(func, iterable): ... for item in iterable: ... yield func(item) # func could raise StopIteration ... it = map(func, [1, 2, 3, -1, 2]) list(it) [1, 4, 9] list(it) # The exception terminates the frame in this case []

In principle that could be a useful feature in some cases. In practise it is more likely to be something that masks a bug manifesting in a stray StopIteration e.g.:

...

...

def first_line(filename): ... with open(filename) as fin: ... return next(fin) # What happens if the file is empty ... for line in map(first_line, filenames): ... print(line) ...

In this case it is the for-loop (the iterator-consumer) that catches the StopIteration. Whether or not map is a generator or some other iterator is irrelevant.

The change in the PEP is to change that side effect such that those exceptions are converted to RuntimeError rather than silently suppressed - making generator function bodies behave less like __next__ method implementations.

Fine but generator function bodies are __next__ method implementations. You can claim that the generator function implements __iter__ but its body implements the __next__ method of the generator. It is precisely because it is a __next__ method that it finds itself being called by an iterator consumer and so finds that a loose StopIteration is suppressed by the consumer. This is the same for all iterators and there is nothing special about generators in this regard (although the PEP will change that a little). To summarise succinctly: Loose StopIteration can have an unintended interaction with the iterator protocol and more specifically with iterator-consumers. The effect is as if the iterator had been stopped since the consumer will stop iteration. Because generators are iterators they (like all iterators) are affected by this. Oscar

On 12 December 2014 at 13:14, Nick Coghlan wrote:

On 12 December 2014 at 22:42, Oscar Benjamin wrote:

...

On 12 December 2014 at 10:34, Nick Coghlan wrote:

...

The specific problem deemed worthy of being fixed is that the presence of "yield" in a function body can implicitly suppress StopIteration exceptions raised elsewhere in that function body (or in functions it calls).

The yield causes the function to become a generator function. The frame for a generator function (like for any other function) will allow uncaught exceptions to propagate to the frame above. The difference between generator functions and other functions is that the code in the body of a generator function is executed when someone calls the generator's __next__ method. Since the caller (the iterator consumer) is expecting StopIteration it is treated as the signalling the end of iteration. The yield suppresses nothing; it is the iterator consumer e.g. the for-loop or the list() function etc. which catches the StopIteration and treats it as termination.

Oscar, this is where you're *just plain wrong* from a technical perspective.

Which of the above sentences is technically incorrect?

The conversion to a generator function also introduces an additional layer of indirection at iteration time through the generator-iterator's __next__ method implementation.

That indirection layer is the one that converts an ordinary return from the generator frame into a StopIteration exception. Historically, it also allowed StopIteration from within the frame to propagate out, make it impossible to tell from outside the generator iterator whether the termination was due to the generator returning normally, or due to StopIteraton being raised.

I understand all of this.

The introduction of "yield from" in Python 3.3, and its use in the coroutine design for the asyncio library and similar contexts *makes that distinction more important than it used to be*. It even managed to introduce obscure bugs around the use of generators to write context managers, where factoring out part of the context manager to a subgenerator may implicitly suppress a StopIteration exception raised in the body of the with statement.

The problem you think PEP 479 is trying to solve *is* the one where the discussion started, but it is *not* the one where it ended when Guido accepted PEP 479. The problems PEP 479 solves are generator specific - they have nothing to do with the iterator protocol in general.

Okay well this is not clear from the PEP (and bear in mind that I did take the time to read through all of the lengthy threads on this subject before posting here). I haven't read the most recent changes but when I did read the PEP the justification appeared to stem from claims that were confusing at best and plain false at worst e.g. "generators are not iterators". Oscar

On 12/12/2014 09:16 AM, Chris Barker wrote:

3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: 1) write a class that has a __iter__ and __next__ methods that "do the right thing" 2a) write a generator function 2b) write a generator expression

3) write a class with __getitem__ that supports integer indexing -- ~Ethan~

On Fri, Dec 12, 2014 at 11:41 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

On Fri, Dec 12, 2014 at 12:16 PM, Chris Barker wrote:

...

2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc.

The range object is not an iterator

indeed -- sorry about that -- the "rangeiterator" object, that is. The point here is that there are a bunch of builtin types that are TIIPs

Please don't confuse iterators and iterables:

indeed, that is a distinction that should be kept clear. Thanks -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, Dec 12, 2014 at 2:28 PM, Ethan Furman wrote:

...

3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: 1) write a class that has a __iter__ and __next__ methods that "do the right thing" 2a) write a generator function 2b) write a generator expression

3) write a class with __getitem__ that supports integer indexing

This gets you a custom iterable. The corresponding iterator will be the buitin iterator type:

...

...

class X: ... __getitem__ = None ... iter(X())

On 12/12/2014 12:06 PM, Alexander Belopolsky wrote:

On Fri, Dec 12, 2014 at 2:28 PM, Ethan Furman wrote:

...

3) write a class with __getitem__ that supports integer indexing

This gets you a custom iterable. The corresponding iterator will be the buitin iterator type:

...

...

...

class X: ... __getitem__ = None ... iter(X())

Good catch, thanks! -- ~Ethan~

On Fri, Dec 12, 2014 at 12:14 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

This gets you a custom iterable. The corresponding iterator will be the

...

buitin iterator type:

...

...

...

class X: ... __getitem__ = None ... iter(X())

The same logic applies to generators and I think this misconception is at the root of the confusion in this thread. When you are writing a generator function, you are not creating a custom TIIP. You are writing a code object that Python knows how to convert into an instance of a built-in TIIP (the generator type).

The point was supposed to be that you want a TIIP with particular behavior that is not built-in (hence custom). Yes, in terms of type, a generator function, when called, will return the a built-in type, but its behavior is customized. Indeed the same as writing a class with __getitem__ and no __iter__ or __next__ will give you an instance a built-in iterator type with customized behavior. I guess my take is that the user shouldn't have to know exactly what the type of TIIP is produced is -- only what its behavior will be. But maybe that IS the source of much of this confusion -- the PEP authors make the point that generators are something special. I want tend to think of them as an implementation detail. This PEP makes them more special, and that's really the point of the contentious part of the PEP. So maybe that's the the point we are trying to make -- while we can argue about whether generators ARE iterators, or whether they have been "conflated" -- generators are special, and might as well be more special ;-) -Chris

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 12 December 2014 at 22:54, Terry Reedy wrote:

On 12/12/2014 4:10 PM, Oscar Benjamin wrote:

...

The function was just supposed to generate the effect. More plausible examples can be made. I tripped over this myself once when doing something with iterators that required me to extract the first item before processing the rest. A simple demonstration:

def sum(iterable): iterator = iter(iterable) total = next(iterator) for item in iterator: total += item return total

Examples like this has been posted on python-list, probably more than once. The answer has been to catch StopIteration (which should only come from next/__next__) and either return a default value or raise ValueError('cannot sum an empty iterable').

I know what the solution is. For my purposes simply raising an exception (any exception that wasn't StopIteration) would have been sufficient. I'm just saying that it's a mistake that can easily be made. Since I made it I'm now super-suspicious of next() so I probably wouldn't make it again. This mistake is the root of all of the problems that PEP 479 attempts to solve. If you don't believe it is a problem that someone might make this mistake then it follows that PEP 479 is a massive mistake.

On Fri, Dec 12, 2014 at 3:22 PM, Andrew Barnert wrote:

1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now.

When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class".

this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... and: In [12]: g = (i for i in range(10)) In [13]: isinstance(it, collections.abc.Iterator) Out[13]: True and In [15]: def gf(): ....: yield 1 ....: yield 2 In [16]: gen = gf() In [17]: isinstance(gen, collections.abc.Iterator) Out[17]: True So generators ARE iterators, but his definition and yet the contrast was being made....

This kind of shorthand is used regularly in discussing types in every OO language, not just Python.

sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer...

In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity.

actually, the confusion between iterables and iterators want the issue I as trying to address -- but the confusion with a sentence like "Under this proposal, generators and iterators would be distinct, but related, concepts" That has been removed from the PEP -- I think it's more clean now in any case. But yes, I could have written all that and stuck with plain old "iterator" instead of making up "TIIP" Especially since it doesn't even help to solve the problem--as we've just

seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator".

My bad -- typing too fast! And this was never about the itterable vs iterator distinction anyway.

(In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?)

I was not trying to mingle iterable and iterator -- really I wan't :-)

The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.

now I am confused: In [18]: g = ( i for i in range(10) ) In [19]: type(g) Out[19]: generator how is that not a "generator"? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 12/12/2014 05:22 PM, Andrew Barnert wrote:

On Dec 12, 2014, at 16:40, Chris Barker wrote:

...

this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them...

But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a subtype of Iterator.

Um, what? -- ~Ethan~

On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote:

In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. [...]

+1 to this. But:

The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people.

I don't know what thing you are referring to. If I write this: py> def gen(): ... yield 1 ... py> it = gen() then `gen` is a function. It's a specific kind of function that uses "yield", and the name for that is a generator function. Informally, sometimes people call it a "generator", which is a bad habit due to the possibility of confusing `gen` with `it`, but since it is quite rare to be in a position where such confusion can occur (PEP 479 and related discussions not withstanding), we can often get away with such sloppy terminology. Also, `it` is an iterator. It's also a generator: py> type(it) The same applies to generator expressions: py> type(c for c in "aaa") So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example? -- Steven

On 12/13/2014 01:37 PM, Andrew Barnert wrote:

...

...

I don't know what thing you are referring to. If I write this:

py> def gen(): ... yield 1 ... py> it = gen()

then `gen` is a function. Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/.

Of course g is an instance of generator (and therefore also an iterator).

But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name.

(I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.)

...

...

def gen(): ... yield 1 ... type(gen) gen

It's a function with a flag set.

...

...

gen.__code__.co_flags 99

...

...

dis(gen) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE

Where a function's flags would be this...

...

...

def foo(): ... return 1 ... foo.__code__.co_flags 67

So, it the answer is yes and no. It's a function object, with an altered behaviour. It seems like the flag may be checked at call time, and then a new function generator instance is returned..

...

...

gi = gen() type(gi) gi

And the original function body is moved to it's gi_code attribute.

...

...

dis(gi.gi_code) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE

I think the send() method, a builtin C routine, handles all the particulars of running gi_code with gi_frame. (I'd have to look to be sure.) I think if function definitons with yield in them created generator_instance_factory objects instead of function objects, things would be easier to explain. What it comes down to, is they aren't normal functions and it's a bit confusing to think of them that way. Cheers, Ron

On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote:

On Dec 13, 2014, at 5:36, Steven D'Aprano wrote:

...

On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote:

...

In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. [...]

+1 to this.

But:

...

The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people.

I don't know what thing you are referring to. If I write this:

py> def gen(): ... yield 1 ... py> it = gen()

then `gen` is a function.

Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/.

I read your reply to Chris last night, and it didn't make sense to me then and it still doesn't make sense to me now :-( Hence my request for a concrete example. I am still unclear as to what this mystery unnamed thing is. It's not a generator function, because that is called a generator function. It's not the thing you get when you call a generator function, because that is called a generator or an iterator, depending on context. Surely it's not the generator type, because that has a name: "generator". We always use the same name for the class and its instances, e.g. we talk about "int" the class as well as saying that 123 is an int. The class "int" is not itself an int instance, but we still call it "int". And likewise for "generator".

Of course g is an instance of generator (and therefore also an iterator).

Agreed. But giving examples of things which aren't the mystery entity doesn't really help explain which mystery entity you are talking about :-)

But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name.

Sure the thing you are defining when you write "def gen() ..." is a function? I thought we were in agreement about that. It's a function with a flag set, a.k.a. a generator function. If you're referring to something else, a concrete example will help. Because as we stand now, I don't know whether we're talking about something important or not, or if we are in agreement, disagreement, or merely confusion :-) -- Steven

On Sun, Dec 14, 2014 at 12:33 PM, Ron Adam wrote:

The term generator_function was use quite a lot in the PEP discussion in order to distinguish it from a regular function, and also from the generator returned from it. So it seems to me, that maybe there is something to that.

Many things can be distinguished that are not distinct types. A generator function is a function that has a 'yield' in it. A factory function is a function that constructs and returns a new object. A recursive function is a function which calls itself. All three are functions in every way. ChrisA

On Dec 13, 2014, at 17:17, Steven D'Aprano wrote:

On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote:

...

On Dec 13, 2014, at 5:36, Steven D'Aprano wrote:

...

On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote:

...

In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. [...]

+1 to this.

But:

...

The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people.

I don't know what thing you are referring to. If I write this:

py> def gen(): ... yield 1 ... py> it = gen()

then `gen` is a function.

Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/.

I read your reply to Chris last night, and it didn't make sense to me then and it still doesn't make sense to me now :-( Hence my request for a concrete example.

You didn't ask for a concrete example, you provided the exact same answer as Chris but with different names.

I am still unclear as to what this mystery unnamed thing is. It's not a generator function, because that is called a generator function. It's not the thing you get when you call a generator function, because that is called a generator or an iterator, depending on context.

Again, it's the thing used by the generator instance's __next__ method: the frame and code objects.

...

But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name.

Sure the thing you are defining when you write "def gen() ..." is a function? I thought we were in agreement about that. It's a function with a flag set, a.k.a. a generator function.

If you're referring to something else, a concrete example will help.

I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object. So, going back to the original point: Generator instances are not different from iterators (except that it's a subtype, of course). The builtin generator type is likewise no different than a custom iterator class. The generator body is a normal function body. The only thing that isn't like an iterator is the frame. Which is not an important distinction. I think Chris was making a different distinction, claiming that generators are not an Iterator type (which they aren't, but only because they're _instances_ of an Iterator type, the builtin type generator), and I was pointing out that this can't be the distinction Guido was trying to make. But, again, since Guido has already said that sentence was mistaken, this really isn't important.

Because as we stand now, I don't know whether we're talking about something important or not, or if we are in agreement, disagreement, or merely confusion :-)

I already said from the start that this is almost certainly not important.