On 16 November 2014 00:37, Chris Angelico <rosuav@gmail.com> wrote:

...

...

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

There's an additional subtlety with this idea: if we add a new GeneratorReturn exception as a subclass of StopIteration, then generator iterators would likely also have to change to replace GeneratorReturn with a regular StopIteration (chaining appropriately via __cause__, and copying

On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: the

...

return value across).

Would have to do so automatically, meaning this is no simpler than the current proposal? Or would have to be always explicitly written to handle it?

When GeneratorReturn escaped a generator frame, the interpreter would automatically convert it into an ordinary StopIteration instance. It's still simpler because it won't need the __future__ dance (as it doesn't involve any backwards incompatible changes).

...

I definitely see value in adding a GeneratorReturn subclass to be able to tell the "returned" vs "raised StopIteration" cases apart from outside the generator (the current dance in contextlib only works because we have existing knowledge of the exact exception that was thrown in). I'm substantially less convinced of the benefit of changing generators to no longer suppress StopIteration. Yes, it's currently a rather odd corner case, but changing it *will* break code (at the very least, anyone using an old version of contextlib2, or who are otherwise relying on their own copy of contextlib rather than standard library one).

This is why it's proposed to use __future__ to protect it.

Using __future__ still imposes a large cost on the community - docs need updating, code that relies on the existing behaviour has to be changed, developers need to adjust their mental models of how the language works. There needs to be a practical payoff for those costs - and at the moment, it's looking like we can actually get a reasonably large fraction of the gain without most of the pain by instead pursuing Guido's idea of a separate StopIteration subclass to distinguish returning from the outermost generator frame from raising StopIteration elsewhere in the generator.

If anyone's still using an old version of contextlib2 once 3.7 comes along, it'll break; but is there any reason to use Python 3.7 with a contextlib from elsewhere than its standard library?

Same reason folks use it now: consistent behaviour and features across a range of Python versions. However, that's not the key point - the key point is that working through the exact changes that would need to be made in contextlib persuaded me that I was wrong when I concluded that contextlib wouldn't be negatively affected. It's not much more complicated, but if we can find a fully supported example like that in the standard library, what other things might folks be doing with generators that *don't* fall into the category of "overly clever code that we don't mind breaking"?

(I'm not familiar with contextlib2 or what it offers.)

contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially created it as a proving ground for the idea that eventually become contextlib.ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16 November 2014 01:56, Chris Angelico <rosuav@gmail.com> wrote:

On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

On 16 November 2014 00:37, Chris Angelico <rosuav@gmail.com> wrote:

...

On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan@gmail.com>

wrote:

...

...

...

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

There's an additional subtlety with this idea: if we add a new GeneratorReturn exception as a subclass of StopIteration, then generator iterators would likely also have to change to replace GeneratorReturn with a regular StopIteration (chaining appropriately via __cause__, and copying the return value across).

Would have to do so automatically, meaning this is no simpler than the current proposal? Or would have to be always explicitly written to handle it?

When GeneratorReturn escaped a generator frame, the interpreter would automatically convert it into an ordinary StopIteration instance.

Okay, let me see if I have this straight. When a 'return' statement (including an implicit one at end-of-function) is encountered in any function which contains a 'yield' statement, it is implemented as "raise GeneratorReturn(value)" rather than as "raise StopIteration(value)" which is the current behaviour. However, if any GeneratorReturn would be raised in any way other than the 'return' statement, it would magically become a StopIteration instead. Is that correct?

That's not quite how generators work. While the "returning from a generator is equivalent to raise StopIteration" model is close enough that it's functionally equivalent to the actual behaviour in most cases (with the main difference being in how try/except blocks and context managers inside the generator react), this particular PEP is a situation where it's important to have a clear picture of the underlying details. When you have a generator iterator (the thing you get back when calling a generator function), there are two key components: * the generator iterator object itself * the generator frame where the code is running When you call next(gi), you're invoking the __next__ method on the *generator iterator*. It's that method which restarts evaluation of the generator frame at the point where it last left off, and interprets any results. Now, there are three things that can happen as a result of that frame evaluation: 1. It hits a yield point. In that case, gi.__next__ returns the yielded value. 2. It can return from the frame. In that case. gi.__next__ creates a *new* StopIteration instance (with an appropriate return value set) and raises it 3. It can throw an exception. In that case, gi.__next__ just allows it to propagate out (including if it's StopIteration) The following example illustrates the difference between cases 2 and 3 (in both cases, there's a StopIteration that terminates the hidden loop inside the list() call, the difference is in where that StopIteration is raised):

...

...

def genreturn(): ... yield ... try: ... return ... except: ... print("No exception") ... raise ... list(genreturn()) [None]

...

...

def genraise(): ... yield ... try: ... raise StopIteration ... except: ... print("Exception!") ... raise ... list(genraise()) Exception! [None]

(The possible outcomes of gi.send() and gi.throw() are the same as those of next(gi). gi.throw() has the novel variant where the exception thrown in may propagate back out) The two change proposals being discussed are as follows: Current PEP (backwards incompatible): Change outcome 3 to convert StopIteration to RuntimeError (or a new exception type). Nothing else changes. Alternative (backwards compatible): Change outcome 2 to raise GeneratorReturn instead of StopIteration and outcome 3 to convert GeneratorReturn to StopIteration. The alternative *doesn't* do anything about the odd discrepancy between comprehensions and generator expressions that started the previous thread. It just adds a new capability where code that knows it's specifically dealing with a generator (like contextlib or asyncio) can more easily tell the difference between outcomes 2 and 3.

This does sound simpler. All the magic is in the boundary of the generator itself, nothing more. If a __next__ method raises either StopIteration or GeneratorReturn, or if any other function raises them, there's no special handling.

All the magic is actually at the generator boundary regardless. The key differences between the two proposals are the decision to keep StopIteration as a common parent exception, and allow it to continue propagating out of generator frames unmodified.

Question: How does it "become" StopIteration? Is a new instance of StopIteration formed which copies in the other's ``value``? Is the type of this exception magically altered? Or is it a brand new exception with the __cause__ or __context__ set to carry the original?

I'd suggest used the exception chaining machinery and creating a new exception with __cause__ and the generator return value set appropriately.

...

(I'm not familiar with

...

contextlib2 or what it offers.)

contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially created it as a proving ground for the idea that eventually become contextlib.ExitStack).

Thanks, I figured it'd be like that. Since contextlib exists in 2.7, is contextlib2 meant to be legacy support only?

contextlib has actually been around since 2.5, but some features (most notably ExitStack) weren't added until much later. Like unittest2, contextlib2 allows access to newer stdlib features on older versions (I haven't used it as a testing ground for new ideas since ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 16 November 2014 01:56, Chris Angelico <rosuav@gmail.com> wrote:

...

Okay, let me see if I have this straight. When a 'return' statement (including an implicit one at end-of-function) is encountered in any function which contains a 'yield' statement, it is implemented as "raise GeneratorReturn(value)" rather than as "raise StopIteration(value)" which is the current behaviour. However, if any GeneratorReturn would be raised in any way other than the 'return' statement, it would magically become a StopIteration instead. Is that correct?

When you call next(gi), you're invoking the __next__ method on the *generator iterator*. It's that method which restarts evaluation of the generator frame at the point where it last left off, and interprets any results.

Now, there are three things that can happen as a result of that frame evaluation:

1. It hits a yield point. In that case, gi.__next__ returns the yielded value.

2. It can return from the frame. In that case. gi.__next__ creates a *new* StopIteration instance (with an appropriate return value set) and raises it

3. It can throw an exception. In that case, gi.__next__ just allows it to propagate out (including if it's StopIteration)

Thank you for explaining. -- Cameron In case others were also oversimplifying in their heads, I've summarized the above into the PEP.

(The possible outcomes of gi.send() and gi.throw() are the same as those of next(gi). gi.throw() has the novel variant where the exception thrown in may propagate back out)

Should that variant affect this proposal? What should happen if you throw StopIteration or GeneratorReturn into a generator?

The two change proposals being discussed are as follows:

Current PEP (backwards incompatible): Change outcome 3 to convert StopIteration to RuntimeError (or a new exception type). Nothing else changes.

Alternative (backwards compatible): Change outcome 2 to raise GeneratorReturn instead of StopIteration and outcome 3 to convert GeneratorReturn to StopIteration.

The alternative *doesn't* do anything about the odd discrepancy between comprehensions and generator expressions that started the previous thread. It just adds a new capability where code that knows it's specifically dealing with a generator (like contextlib or asyncio) can more easily tell the difference between outcomes 2 and 3.

Text along these lines added to PEP, thanks!

...

Question: How does it "become" StopIteration? Is a new instance of StopIteration formed which copies in the other's ``value``? Is the type of this exception magically altered? Or is it a brand new exception with the __cause__ or __context__ set to carry the original?

I'd suggest used the exception chaining machinery and creating a new exception with __cause__ and the generator return value set appropriately.

Makes sense. If the __cause__ is noticed at all (ie this doesn't just quietly stop a loop), it wants to be very noisy.

...

Thanks, I figured it'd be like that. Since contextlib exists in 2.7, is contextlib2 meant to be legacy support only?

contextlib has actually been around since 2.5, but some features (most notably ExitStack) weren't added until much later. Like unittest2, contextlib2 allows access to newer stdlib features on older versions (I haven't used it as a testing ground for new ideas since ExitStack).

If there is breakage from this, it would simply mean "older versions of contextlib2 are not compatible with Python 3.7, please upgrade your contextlib2" - several of the variants make it perfectly possible to write cross-version-compatible code. I would hope that this remains the case. Latest version of PEP text incorporating the above changes: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt (My apologies if this email has gone through more than once. I'm having major issues with my internet connection at the moment, and delivery is failing and being retried. Hopefully it really *is* failing, and not just saying so.) ChrisA

Exactly! The current behavior is not only likely undesirable, but it is also undocumented. Even if parts of stdlib rely on the current behavior, there's no need for a deprecation (read __future__) period. Undocumented features may change any time, because are mostly about implementation quirks (Isn't that rule documented somewhere in the Python docs?). In short: -1 deprecation (__future__); no need, because nothing documented gets broken +1 fix it now (3.5); the fix may be a change in the docs to validate the current behavior, and deprecate it (Yuk!) +1 Nick's design, which kind of leaves it the same and kind of fixes it p.s. What about 2.7? This fix is *not* a new feature. Cheers, -- Juanca On Sat Nov 15 2014 at 1:50:06 PM Steve Dower <Steve.Dower@microsoft.com> wrote:

Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.)

Cheers, Steve

Top-posted from my Windows Phone ------------------------------ From: Chris Angelico <rosuav@gmail.com> Sent: ‎11/‎15/‎2014 1:30 To: python-ideas <python-ideas@python.org> Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators

PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014

Abstract ========

This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat.

Rationale =========

The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct.

Proposal ========

If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_

Consequences to existing code =============================

This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail.

(Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""")

As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7.

Alternate proposals ===================

Supplying a specific exception to raise on return -------------------------------------------------

Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed.

Making return-triggered StopIterations obvious ----------------------------------------------

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

Criticism =========

Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain.

References ==========

.. [1] Initial mailing list comment ( https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

.. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby)

.. [3] Proposal by GvR ( https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

.. [4] Response by Steven D'Aprano ( https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)

Copyright =========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Nov 16, 2014 at 5:20 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:

Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.)

The behaviour selection would have to be based on the generator's definition. This proposal, in all its variants, is about what happens as the generator terminates; if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct. ChrisA

On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:

Chris Angelico wrote:

...

if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct.

This suggests that the use of a __future__ directive is not really appropriate, since it can affect code outside of the module with the __future__ directive in it.

I don't see how that is different from any other __future__ directive. They are all per-module, and if you gain access to an object from another module, it will behave as specified in the module that created it, not the module that imported it. How is this different? -- Steven

(I'm catching up on this thread from the end.) On Sun, Nov 16, 2014 at 5:29 PM, Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:

...

Chris Angelico wrote:

...

if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct.

This suggests that the use of a __future__ directive is not really appropriate, since it can affect code outside of the module with the __future__ directive in it.

I don't see how that is different from any other __future__ directive. They are all per-module, and if you gain access to an object from another module, it will behave as specified in the module that created it, not the module that imported it. How is this different?

Well, let's see. For feature in sorted(__future__.all_feature_names):

absolute_import: Affects implementation of a keyword barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) division: Changes the meaning of one operator. generators: Introduces a keyword nested_scopes: Alters the compilation of source to byte-code(?) print_function: Removes a keyword unicode_literals: Alters the type used for literals with_statement: Introduces a keyword

Apart from the joke, it seems that every __future__ directive is there to affect the compilation, not execution, of its module: that is, once a module has been compiled to .pyc, it shouldn't matter whether it used __future__ or not. Regardless of unicode_literals, you can create bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not entirely sure about division (can you call on true-division without the future directive?), but in any case, it's all done at compilation time, as can be seen interactively:

Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

def division1(x,y): ... return x/y, x//y, x%y ... from __future__ import division def division2(x,y): ... return x/y, x//y, x%y ... import dis dis.dis(division1) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE dis.dis(division2) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_TRUE_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE

So to make this consistent with all other __future__ directives, there would need to be some kind of safe way to define this: perhaps an attribute on the generator object. Something like this:

...

...

...

def gen(x): ... yield x ... raise StopIteration(42) ... g=gen(123) list(g) [123] gen.__distinguish_returns__ = True g=gen(123) list(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in gen StopIteration: 42

The attribute on the function would be what affects behaviour; the __future__ directive applies that attribute to all generator functions in its module (including genexprs). Once the __future__ directive becomes automatic, the attribute can and will be dropped - any code which interrogates it MUST be prepared to stop interrogating it once the feature applies to all modules.

Does that sound reasonable? Should it be added to the PEP?

I agree with you and Steven that this is a fine use of __future__. What a generator does with a StopIteration that is about to bubble out of its frame is up to that generator. I don't think it needs to be a flag on the *function* though -- IMO it should be a flag on the code object. (And the flag should somehow be transferred to the stack frame when the function is executed, so the right action can be taken when an exception is about to bubble out of that frame.) One other small point: let's change the PEP to just propose RuntimeError, and move the "some other exception" to the "rejected ideas" section. -- --Guido van Rossum (python.org/~guido)

On 11/17/2014 03:11 AM, Chris Angelico wrote:

On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum <guido@python.org> wrote:

...

I agree with you and Steven that this is a fine use of __future__. What a generator does with a StopIteration that is about to bubble out of its frame is up to that generator. I don't think it needs to be a flag on the *function* though -- IMO it should be a flag on the code object. (And the flag should somehow be transferred to the stack frame when the function is executed, so the right action can be taken when an exception is about to bubble out of that frame.)

One other small point: let's change the PEP to just propose RuntimeError, and move the "some other exception" to the "rejected ideas" section.

Changes incorporated, thanks! I'm not familiar with the details of stack frame handling, so I've taken the cop-out approach and just quoted you directly into the PEP.

PEP draft: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

I fixed a typo and syntax and made a content clarification ("generator constructed" -> "generator function constructed"; the generator object itself is constructed when the function is called) and committed the changes. Georg

On Mon, Nov 17, 2014 at 2:27 AM, Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl <g.brandl@gmx.net> wrote:

...

I fixed a typo and syntax and made a content clarification ("generator constructed" -> "generator function constructed"; the generator object itself is constructed when the function is called) and committed the changes.

Thanks Georg! This means today's version is now visible here:

http://legacy.python.org/dev/peps/pep-0479/

Off-topic: the new python.org site now supports PEPs, so please switch to URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't like the formatting send a pull request to https://github.com/python/pythondotorg). -- --Guido van Rossum (python.org/~guido)

On 17/11/2014 2:29 p.m., Chris Angelico wrote:

barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)

It determines whether "not equal" is spelled "!=" or "<>", so it fits the pattern of being compile-time-only. -- Greg

On 17 November 2014 13:34, Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Nov 17, 2014 at 2:22 PM, Greg <greg.ewing@canterbury.ac.nz> wrote:

...

On 17/11/2014 2:29 p.m., Chris Angelico wrote:

...

barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)

It determines whether "not equal" is spelled "!=" or "<>", so it fits the pattern of being compile-time-only.

Right. So, are there any __future__ directives that have any effect on byte-code? I'm not seeing any, though that may just mean I didn't recognize one.

True division (in Python 2) is a nice simple one to look at, since it just swaps one bytecode for another (BINARY_DIVIDE -> BINARY_TRUE_DIVIDE)

...

...

def floor_div(x, y): ... return x / y ... from __future__ import division def true_div(x, y): ... return x / y ... import dis dis.dis(floor_div) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_DIVIDE 7 RETURN_VALUE dis.dis(true_div) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_TRUE_DIVIDE 7 RETURN_VALUE

The compiler actually stores a whole pile of useful info on code objects that doesn't show up in the disassembly output (switching to Python 3 for more up to date dis module goodness):

...

...

import dis def true_div(x, y): ... return x / y ... dis.show_code(true_div) Name: true_div Filename: <stdin> Argument count: 2 Kw-only arguments: 0 Number of locals: 2 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS, NOFREE Constants: 0: None Variable names: 0: x 1: y

So conveying to the generator iterator whether or not "from __future__ import generator_return" was in effect would just be a matter of the compiler setting a new flag on the generator code object. For *affected generators* (i.e. those defined in a module where the new future statement was in effect), StopIteration escaping would be considered a RuntimeError. For almost all code, such RuntimeErrors would look like any other RuntimError raised by a broken generator implementation. The only code which would *have* to change immediately as a "Porting to Python 3.5" requirement is code like that in contextlib, which throws StopIteration into generators, and currently expects to get it back out unmodified. Such code will need to be updated to also handle RuntimError instances where the direct cause is the StopIteration exception that was thrown in. Other affected code (such as the "next() bubbling up" groupby example) would keep working unless the __future__ statement was in effect. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I don't want to contemplate a new __next__ protocol. The existing protocol was carefully designed and tuned to have minimal memory overhead (even to the point where the exception instance returned may be reused). Wapping each result would just result in an extra allocation + deallocation per iteration, unless you can play games with reference counts or do something else to complicate the semantics). Introducing __nextx__ would require thousands of libraries implementing this to incur churn as they feel the pressure to switch to the new protocol, and the compatibility issue would be felt everywhere. The problem we're trying to fix is unique to generators (thereby also implicating generator expressions). On Mon, Nov 17, 2014 at 6:04 AM, Chris Angelico <rosuav@gmail.com> wrote:

On Tue, Nov 18, 2014 at 12:50 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:

...

As another alternative, how about a new iterator protocol that is defined without this ambiguity? Code at the bottom of my post to help explain: define a new method __nextx__ which doesn't use StopIteration for any signalling, instead, it returns None if there are no values to return, and returns a special value Some(v) if it wants to return a value v. Both next(it) and nextx(it) are made to work for any iterator that is defined using either protocol, but for loops and Python builtins all use nextx internally. Generators define __next__ unless you from __future__ import iterators, in which case they define __nextx__ instead.

I had actually contemplated adding a "what if __next__ returned a sentinel instead of raising an exception" possibility to the PEP, if only for completeness. Since someone else has suggested it too now, it may be worth doing.

Rather than a wrapper around every returned value, what I'd be inclined toward is a special sentinel that otherwise cannot be returned. This could be a dedicated, arbitrary object, or something globally unique, or something locally unique. One option that comes to mind is to have the generator return _itself_ to signal that it's returned.

I don't think this option will be better than the current front runners, but would you like me to add it for completeness? The biggest downside is that it might give a false positive; you can't, for instance, have an iterator "all_objects()" which returns, like the name says, every object currently known to Python. (I don't know that CPython is capable of implementing that, but there's no reason another Python couldn't, and it might be useful.) I expect that's why the exception system was used instead; can anyone confirm that?

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

On Mon, Nov 17, 2014 at 9:53 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote: [Guido]

...

The problem we're trying to fix is unique to generators (thereby also implicating generator expressions).

I suppose since you're only fixing generators, then that is literally the only problem you are trying to fix, but it is more general than that. I have encountered this sort of problem writing __next__ by hand in Python -- that is, that bugs inside code I call result in silent control flow changes rather than a visible exception.

I assume this is something where the __next__() method on your iterator class calls next() on some other iterator and accidentally doesn't catch the StopIteration coming out of it (or, more likely, this happens several calls deep, making it more interesting to debug). That particular problem is not unique to __next__ and StopIteration -- the same thing can (and does!) happen with __getitem__ and KeyError or IndexError, and with __getattr[ibute]__ and AttributeError. In all these cases I think there isn't much we can do apart from adding lint rules. If you are writing __next__ as a method on an iterator class, one way or another you are going to have to raise StopIteration when there isn't another element, and similarly __getitem__ has to raise KeyError or IndexError, etc. In the generator case, we have a better way to signal the end -- a return statement (or falling off the end). And that's why we can even contemplate doing something different when StopIteration is raised in the generator. -- --Guido van Rossum (python.org/~guido)

On Tue, Nov 18, 2014 at 4:40 AM, Guido van Rossum <guido@python.org> wrote:

Wapping each result would just result in an extra allocation + deallocation per iteration...

Which is why I would be more inclined to use a sentinel of some sort... but that has its own problems. There's no perfect solution, so status quo wins unless a really compelling case can be made. I could toss something into the Alternate Proposals section, but I wouldn't be personally supporting it. ChrisA

On Tue, Nov 18, 2014 at 5:17 AM, Guido van Rossum <guido@python.org> wrote:

Trust me, we went down this rabbit hole when we designed the iterator protocol. The problem is that that sentinel object must have a name (otherwise how would you know when you had seen the sentinel), which means that there is at least one dict (the namespace defining that name, probably the builtins module) that has the sentinel as one of its values, which means that iterating over that particular dict's values would see the sentinel as a legitimate value (and terminate prematurely). You could fix this by allocating a unique sentinel for every iteration, but there would be some additional overhead for that too (the caller and the iterator both need to hold on to the sentinel object).

That's a much more solid argument against it than I had. (Also amusing to try to contemplate.) The iterator protocol absolutely demands a completely out-of-band means of signalling "I have nothing to return now".

In any case, as I tried to say before, redesigning the iterator protocol is most definitely out of scope here (you could write a separate PEP and I'd reject it instantly).

That's a PEP I'll leave for someone else to write. :) ChrisA

FWIW, I spent some time this morning close-reading the PEP, and made a somewhat significant set of updates -- adding more specifics about the proposed new __futue__ statement and the new code object flag, tracking down a few more examples of code that would be affected, and some other minor edits. Here's the diff: https://hg.python.org/peps/rev/8de949863677 Hopefully the new version will soon be here: https://www.python.org/dev/peps/pep-0479 Note that I am definitely not yet deciding on this PEP. I would love it if people sent in examples of code using generator expressions that would be affected by this change (either by highlighting a bug in the code or by breaking what currently works). If this PEP gets rejected, we could resurrect the GeneratorExit proposal currently listed as an alternative -- although the more I think about that the less I think it's worth it, except for the very specific case of asyncio (thinking of which, I should add something to the PEP about that too). -- --Guido van Rossum (python.org/~guido)

Thanks, I added the survey and a summary to the PEP. If you hear about any examples please do forward them here! On Mon, Nov 17, 2014 at 3:56 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On Mon, Nov 17, 2014 at 11:38:27AM -0800, Guido van Rossum wrote:

...

Hopefully the new version will soon be here:

https://www.python.org/dev/peps/pep-0479

Note that I am definitely not yet deciding on this PEP. I would love it if people sent in examples of code using generator expressions that would be affected by this change (either by highlighting a bug in the code or by breaking what currently works).

Over a week ago I raised this issue on python-list mailing list. I expected a storm of bike-shedding, because that's the sort of place p-l is :-) but got just two people commenting.

The thread, for anyone interested:

https://mail.python.org/pipermail/python-list/2014-November/680757.html

One response suggested that it is not generators which do the wrong thing, but comprehensions, and that comprehensions should be changed to behave like generators:

https://mail.python.org/pipermail/python-list/2014-November/680758.html

That should probably be put in the PEP, even if it is not an option being considered, it at least evidence that "some people" find the behaviour of generators more natural than that of comprehensions.

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

On 11/18/2014 05:50 AM, Guido van Rossum wrote:

Separate from this special case, I am also worried about backward compatibility, and I have yet to get a good idea for how widespread code is that depends on StopIteration bubbling out from generators. I also don't have a good idea how often this issue bites users, but I have a feeling it does bite. E.g. this quote from c.l.py <http://c.l.py> (https://mail.python.org/pipermail/python-list/2014-November/680775.html):

""" I did find it annoying occasionally that raising StopIteration inside a generator expression conveys a different behavior than elsewhere. It did take me quite a while to understand why that is so, but after that it did not cause me much of a headache anymore. """

I just remembered one use of the current behavior. Two years ago or so, I was suggesting on this list a possibility for early termination of comprehensions when a particular value is encountered. In other words, an equivalent to: l = [] for x in seq: if x == y: break l.append(x) At the time, somebody suggested (roughly): def stop(): raise StopIteration l = list(x for x in seq if x!=y or stop()) which, for the very reasons discussed in this thread, works only as a generator expression and not in comprehension form. I used this solution in some not particularly important piece of code so I wouldn't despair if it wasn't compatible with the next release of the language. Also, I have a feeling that some of you may consider this sort of a hack in the first place. Just thought I'd mention it here for completeness. Wolfgang

On Wed, Nov 19, 2014 at 9:54 AM, Terry Reedy <tjreedy@udel.edu> wrote:

On 11/18/2014 12:40 PM, Wolfgang Maier wrote:

...

I just remembered one use of the current behavior. Two years ago or so, I was suggesting on this list a possibility for early termination of comprehensions when a particular value is encountered. In other words, an equivalent to:

l = [] for x in seq: if x == y: break l.append(x)

I believe I thought then that one should write the explicit loop rather than overload the 'comprehension' concept.

I'm not sure about that. Comprehensions can already be filtered; is it such a jump from there to a "filter" that aborts on a certain condition? It may not be language-supported, but I don't see that it's illogical; and any use of a loop that appends to a list is rightly considered code smell. ChrisA

On Wed, Nov 19, 2014 at 12:15 PM, Steven D'Aprano <steve@pearwood.info> wrote:

...

and any use of a loop that appends to a list is rightly considered code smell.

I'm afraid I don't understand that comment. Why is appending to a list inside a loop a code smell? That's exactly what list comps do.

That's precisely why. If I write code like this: l = [] for i in something: l.append(func(i)) then I should rework it into a comprehension. Having a filter doesn't change that: l = [] for i in something: if i: l.append(func(i)) That's still possible with a list comp, and should be rewritten as one. But having a break in there *does* change it, because there's no way in the language to do that. The question is: Is it better to abuse StopIteration or to turn the list comp back into an explicit loop? And if anyone chose the former, their code will break. ChrisA

On Wed, Nov 19, 2014 at 1:56 PM, Guido van Rossum <guido@python.org> wrote:

The only thing standing in the way of fixing this is the recognition that there may be a fair amount of code out there that depends on this hack, and which will have to be rewritten.

Has anyone come across any more non-trivial examples? We have contextlib (in the standard library) and contextlib2 (third-party), plus a number of StackOverflow posts and such. Are there any other known pieces of code that would be seriously hampered by this change? ChrisA

On 20 November 2014 15:54, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico <rosuav@gmail.com> wrote:

...

I poked around a bit in the code and managed to come up with this. It doesn't chain the previous exception, so the traceback is a little scanty, but it does turn a StopIteration into a RuntimeError. (It might also leak the original StopIteration. I'm not sure.) Prior to this patch, I had 377 of 390 tests passing flawlessly and no failures (just skips and warnings); with this applied, six failures.

With the attached demo patch, all tests pass except test_generators, which explicitly tests stuff about the correlation between return and StopIteration. There's the contextlib changes, a couple of places that were raising StopIteration and should be returning, and a couple that were letting StopIteration bubble and now need to catch it and return. I've deliberately not followed PEP 8 here, in the interests of minimizing diff size; in several cases, blocks of code ought to be indented a level, but I cheated and created a half-indentation to show how little actually changes.

Thanks Chris - could you make a tracker issue and attach this there? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 11/17/2014 08:50 PM, Guido van Rossum wrote:

Separate from this special case, I am also worried about backward compatibility, and I have yet to get a good idea for how widespread code is that depends on StopIteration bubbling out from generators. I also don't have a good idea how often this issue bites users, but I have a feeling it does bite.

One argument for making the change*: When we're writing __next__, or __getattr__, etc., it is obvious that we are playing with internals and have to be extra careful of what other exceptions might be raised in that code. Contrariwise, the only indication of something special about a generator is the presence of the yield keyword -- for ordinary use (such as in for loops) it doesn't matter whether the called function returns a list, tuple, iterator, generator, or whatever, as long as it can be iterated over, and so when writing a generator, or converting an iterable-returning function into a generator, there's nothing obvious saying, "Hey! Watch out for a StopIteration somewhere else in this block of code!" * I make no statement as to how strong this argument is, but there you have it. :) -- ~Ethan~

On 18.11.14 06:50, Guido van Rossum wrote:

Separate from this special case, I am also worried about backward compatibility, and I have yet to get a good idea for how widespread code is that depends on StopIteration bubbling out from generators. I also don't have a good idea how often this issue bites users, but I have a feeling it does bite.

E.g. current difflib implementation depends on this behavior. ... while True: ... while(found_diff is False): from_line, to_line, found_diff = next(line_pair_iterator) ...

On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

OTOH, I'm also not sure the status quo is sufficiently problematic to be worth changing. Yes, it's a little weird, but is it *that* much weirder than the unavoidable issues with exceptions thrown in __next__, __getitem__, __getattr__ and other special methods where a particular type of exception is handled directly by the interpreter?

If you write __next__, you write in a "raise StopIteration" when it's done. If you write __getattr__, you write in "raise AttributeError" if the attribute shouldn't exist. Those are sufficiently explicit that it should be reasonably clear that the exception is the key. But when you write a generator, you don't explicitly raise: def gen(): yield 1 yield 2 yield 3 return 4 The distinction in __next__ is between returning something and raising something. The distinction in a generator is between "yield" and "return". Why should a generator author have to be concerned about one particular exception having magical meaning? Imagine this scenario: def producer(): """Return user input, or raise KeyboardInterrupt""" return input("Enter the next string: ") def consumer(): """Process the user's input""" while True: try: command = producer() except KeyboardInterrupt: break dispatch(command) Okay, now let's make a mock producer: strings = ["do stuff","do more stuff","blah blah"] def mock_producer() if strings: return strings.pop(0) raise KeyboardInterrupt That's how __next__ works, only with a different exception, and I think people would agree that this is NOT a good use of KeyboardInterrupt. If you put a few extra layers in between the producer and consumer, you'd be extremely surprised that an unexpected KeyboardInterrupt just quietly terminated a loop. Yet this is exactly what the generator-and-for-loop model creates: a situation in which StopIteration, despite not being seen at either end of the code, now has magical properties. Without the generator, *only* __next__ has this effect, and that's exactly where it's documented to be. Does that make for more justification? Unexpected exceptions bubbling up is better than unexpected exceptions quietly terminating loops? ChrisA

On Thu, Nov 20, 2014 at 02:45:27AM +1000, Nick Coghlan wrote:

The part I found most compelling was when you pointed out that in the special method implementations, the normal return path was always spelled with "return", while the "value missing" result was indicated with a special kind of exception (StopIteration, AttributeError, IndexError or KeyError), and then any other exception was consider unexpected.

Generators add the third notion of being able to suspend execution via "yield", which then left them with two different ways of spelling termination inside the frame: "return" OR "raise StopIteration". The second spelling ("raise StopIteration") is then inherently surprising, as it's entirely redundant, *except* in that it allows you to effectively have a "hidden return" in a generator frame that can't be done anywhere else.

I'm not sure that many people outside of this and the python-dev mailing lists would find the use of "raise StopIteration" surprising. Rather, I expect that they will find the use of an explicit "return" inside a generator surprising. People are informally taught that generators use yield *instead of* return, so seeing both in the same function is a surprise. (Most generators quitely fall out the bottom with no explicit end.) I don't claim that doing so is Pythonic or even good practice, but I am sure that there are a lot of people who believe that raising StopIteration to exit a generator is (1) supported and (2) preferred. Examples of code in the wild using StopIteration to exit: http://code.openhub.net/file?fid=ezlejSoT2q7PWrhgNkpdU55MWOA&cid=jVcYOxnQhvU&s=raise%20StopIteration&fp=301369&mp&projSelected=true#L0 http://code.openhub.net/file?fid=M0gWWCpn-avqHO_jnsYcG2T81lg&cid=VKn_M0_GgKM&s=raise%20StopIteration&fp=301283&mp&projSelected=true#L0 http://code.openhub.net/file?fid=pDrrTI8lyh0LO_6rTCk9npC96SE&cid=Y8jg8v1AyqU&s=raise%20StopIteration&fp=41191&mp&projSelected=true#L0 http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0 http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0 http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in... http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-th... That last example not only uses raise to exit the generator, but the author actually guesses that it is the more Pythonic way to do so. Here is a description of the generator protocol which could easily lead the reader to conclude that raising StopIteration is the correct way to exit a generator: To support this protocol, functions with yield statement are compiled specially as generators. They return a generator object when they are called. The returned object supports the iteration interface with an automatically created __next__() method to resume execution. Generator functions may have a return simply terminates the generation of values by raising a StopIteration exceptions after any normal function exit. http://www.bogotobogo.com/python/python_generators.php At this point, I'm convinced that there is a good argument for a __future__ import changing this behaviour. But I suspect that making this the default behaviour in the future will break a lot of code. -- Steven

On Thu, Nov 20, 2014 at 12:34:08PM +1100, Chris Angelico wrote:

On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

Examples of code in the wild using StopIteration to exit:

http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0

Trivially unnecessary, and as soon as there's a bug report, the "What's New In 3.7" page will explain that it needs to be removed.

The point isn't that it is easy to fix. I'm sure that there will be cases of code that are not easy to fix. The point is that we're breaking working code and causing code churn. We're not fixing a bug. We're changing behaviour people rely on. That ought to make us more conservative about breaking their code.

This may result in the odd question here or there, but it's not a major problem.

And neither is the existing behaviour. We're weighing up whether the small benefit in fixing this wart is worth the pain. The PEP isn't approved yet, and right from the beginning Guido said that he feared that fixing this might be too disruptive. I'm trying to get a feel for how disruptive it will be. I did a quick and informal survey of the developers I work with. The dreaded lurgy has gone through our office, so most of them are away ill, but of those still here (all two of them) one of them couldn't remember whether you exit a generator with "yield nil" or "return nil" (he's a Ruby and Objective-C guy when we're not paying him to write Python) and the other one said that the whole problem is that generators exist in the first place, Python should get rid of them and allow people to define their own using macros (he likes to think of himself as a Lisp and Scheme guru :-) Make of that what you will. -- Steven

On 11/19/2014 04:25 PM, Steven D'Aprano wrote:

Here is a description of the generator protocol which could easily lead the reader to conclude that raising StopIteration is the correct way to exit a generator:

To support this protocol, functions with yield statement are compiled specially as generators. They return a generator object when they are called. The returned object supports the iteration interface with an automatically created __next__() method to resume execution. Generator functions may have a return simply terminates the generation of values by raising a StopIteration exceptions after any normal function exit.

http://www.bogotobogo.com/python/python_generators.php

We are not, however, responsible for third-party documentation.

At this point, I'm convinced that there is a good argument for a __future__ import changing this behaviour. But I suspect that making this the default behaviour in the future will break a lot of code.

Isn't that the case with every __future__ directive that becomes the standard? Folks have an entire minor release to make the adjustment. -- ~Ethan~

On 11/19/2014 11:24 AM, Chris Angelico wrote:

On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan <ncoghlan-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

...

OTOH, I'm also not sure the status quo is sufficiently problematic to be worth changing. Yes, it's a little weird, but is it *that* much weirder than the unavoidable issues with exceptions thrown in __next__, __getitem__, __getattr__ and other special methods where a particular type of exception is handled directly by the interpreter?

If you write __next__, you write in a "raise StopIteration" when it's done. If you write __getattr__, you write in "raise AttributeError" if the attribute shouldn't exist. Those are sufficiently explicit that it should be reasonably clear that the exception is the key. But when you write a generator, you don't explicitly raise:

def gen(): yield 1 yield 2 yield 3 return 4

The distinction in __next__ is between returning something and raising something. The distinction in a generator is between "yield" and "return".

Which, as I said a week ago, is why there is no need for "raise StopIteration" in a generator function. The doc clearly states the limited intended use of StopIteration. ''' exception StopIteration Raised by built-in function next() and an iterator‘s __next__() method to signal that there are no further items produced by the iterator. ''' StopIteration is exposed so it can be raised in user coded __next__() and caught when using explicit next(). If it was only used for builtins and for loops, it would not need to be visible.

Why should a generator author have to be concerned about one particular exception having magical meaning?

I am not sure of your intent with this rhetorical (?) question.

Imagine this scenario:

def producer(): """Return user input, or raise KeyboardInterrupt""" return input("Enter the next string: ")

The prompt should be "Enter the next string or hit ^C to quit: ".

def consumer(): """Process the user's input""" while True: try: command = producer() except KeyboardInterrupt: break dispatch(command)

Okay, now let's make a mock producer:

strings = ["do stuff","do more stuff","blah blah"] def mock_producer() if strings: return strings.pop(0) raise KeyboardInterrupt

That's how __next__ works, only with a different exception, and I think people would agree that this is NOT a good use of KeyboardInterrupt.

It is avoidable because the return type of producer is limited to strings. Therefore, producer could (and perhaps should) itself catch KeyboardInterrupt and return None, which is intended for such use. Consumer would then be simplified by replacing 3 lines with "if command is None: break".

If you put a few extra layers in between the producer and consumer, you'd be extremely surprised that an unexpected KeyboardInterrupt just quietly terminated a loop. Yet this is exactly what the generator-and-for-loop model creates: a situation in which StopIteration, despite not being seen at either end of the code, now has magical properties. Without the generator, *only* __next__ has this effect, and that's exactly where it's documented to be.

Does that make for more justification? Unexpected exceptions bubbling up is better than unexpected exceptions quietly terminating loops?

-- Terry Jan Reedy

On Thu, Nov 20, 2014 at 9:57 AM, Chris Angelico <rosuav@gmail.com> wrote:

Since generators can distinguish yield from return, they don't need to distinguish return from raise.

Bad grammar, should edit before posting. Since generators can distinguish value from no value by using yield and return, they don't need to use yield and raise. ChrisA

On Thu, Nov 20, 2014 at 1:06 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote:

...

If you write __next__, you write in a "raise StopIteration" when it's done. If you write __getattr__, you write in "raise AttributeError" if the attribute shouldn't exist. Those are sufficiently explicit that it should be reasonably clear that the exception is the key. But when you write a generator, you don't explicitly raise:

That's not true in practice. See my reply to Nick, there is lots of code out there which uses StopIteration to exit generators. Some of that code isn't very good code -- I've seen "raise StopIteration" immediately before falling out the bottom of the generator -- but until now it has worked and the impression some people have gotten is that it is actually preferred.

Yes, I thought it was rare. I stand corrected. Reword that to "you don't *need to* explicitly raise", since you can simply return, and it becomes true again, though.

...

def gen(): yield 1 yield 2 yield 3 return 4

Until 3.2, that was a syntax error. For the majority of people who are still using Python 2.7, it is *still* a syntax error. To write this in a backwards-compatible way, you have to exit the generator with:

raise StopIteration(2)

In most cases you won't need to put a value on it, so bare "return" will work just fine. I just put a return value onto it so it wouldn't look trivially useless.

...

The distinction in __next__ is between returning something and raising something. The distinction in a generator is between "yield" and "return". Why should a generator author have to be concerned about one particular exception having magical meaning?

I would put it another way: informally, the distinction between a generator and a function is that generators use yield where functions use return. Most people are happy with that informal definition, a full pedantic explanation of co-routines will just confuse them or bore them. The rule they will learn is:

* use return in functions * use yield in generators

That makes generators that use both surprising. Since most generators either run forever or fall out the bottom when they are done, I expect that seeing a generator with a return in it is likely to surprise a lot of people. I've known that return works for many years, and I still give a double-take whenever I see it in a generator.

But it's just as surprising to put "raise StopIteration" into it. It's normal to put that into __next__, it's not normal to need it in a generator. Either way, it's something unusual; so let's go with the unusual "return" rather than the unusual "raise".

...

That's how __next__ works, only with a different exception, and I think people would agree that this is NOT a good use of KeyboardInterrupt.

Why not? How else are you going to communicate something out of band to the consumer except via an exception?

We can argue about whether KeyboardInterrupt is the right exception to use or not, but if you insist that this is a bad protocol then you're implicitly saying that the iterator protocol is also a bad protocol.

Well, that's exactly what I do mean. KeyboardInterrupt is not a good way for two parts of a program to communicate with each other, largely because it can be raised unexpectedly. Which is the point of this PEP: raising StopIteration unexpectedly should also result in a noisy traceback.

I trust that we all expect to be able to factor out the raise into a helper function or method, yes? It truly would be surprising if this failed:

class MyIterator: def __iter__(self): return self def __next__(self): return something()

def something(): # Toy helper function. if random.random() < 0.5: return "Spam!" raise StopIteration

Now let's write this as a generator:

def gen(): while True: yield something()

which is much nicer than:

def gen(): while True: try: yield something() except StopIteration: return # converted by Python into raise StopIteration

Sure. There was a suggestion that "return yield from something()" would work, though, which - I can't confirm that this works, but assuming it does - would be a lot tidier. But there's still a difference. Your first helper function was specifically a __next__ helper. It was tied intrinsically to the iterator protocol. If you want to call a __next__ helper (or actually call next(iter) on something) inside a generator, you'll have to - if this change goes through - cope with the fact that generator protocol says "return" where __next__ protocol says "raise StopIteration". If you want a generator helper, it'd look like this: def something(): # Toy helper function. if random.random() < 0.5: yield "Spam!" def gen(): yield from something() Voila! Now it's a generator helper, following generator protocol. Every bit as tidy as the original. Let's write a __getitem__ helper: def something(x): # Toy helper function. if random.random() < 0.5: return "Spam!" raise KeyError(x) class X: def __getitem__(self, x): return something(x) Same thing. As soon as you get into raising these kinds of exceptions, you're tying your helper to a specific protocol. All that's happening with PEP 479 is that generator and iterator protocol are being distinguished slightly. ChrisA

On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:

[...] Hmm, I'm not convinced by these toy examples, but I did inspect some of my own code for incompatibility with the proposed change. I found that there really is only one recurring pattern I use that I'd have to change and that is how I've implemented several file parsers. I tend to write them like this:

def parser (file_object): while True: title_line = next(file_object) # will terminate after the last record

try: # read and process the rest of the record here except StopIteration: # this record is incomplete raise OSError('Invalid file format') yield processed_record

So I'm catching StopIteration raised by the underlying IOWrapper only if it occurs in illegal places (with regard to the file format the parser expects), but not when it indicates the end of a correct file. I always thought of letting the Error bubble up as a way to keep the parser transparent. Now in this case, I think, I would have to change this to:

def parser (io_object): while True: try: title_line = next(io_object) except StopIteration: return ...

which I could certainly do without too much effort, but could this be one of the more widespread sources of incompatibility that Steve imagines ?

There's probably something important missing from your examples. The above while-loop is equivalent to for title_line in io_object: ... If you're okay with getting RuntimeError instead of OSError for an undesirable StopIteration, you can just drop the except clause altogether. -- --Guido van Rossum (python.org/~guido)

On Fri, Nov 21, 2014 at 9:19 PM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:

My reason for not using a for loop here is that I'm trying to read from a file where several lines form a record, so I'm reading the title line of a record (and if there is no record in the file any more I want the parser generator to terminate/return. If a title line is read successfully then I'm reading the record's body lines inside a try/except, i.e. where it says "# read and process the rest of the record here" in my shortened code I am actually calling next several times again to retrieve the body lines (and while reading these lines an unexpected StopIteration in the IOWrapper is considered a file format error). I realize that I could also use a for loop and still call next(file_object) inside it, but I find this a potentially confusing pattern that I'm trying to avoid by using the while loop and all explicit next().

I agree. The last example in the PEP is a cut-down form of your parser, and I raise the exact same concern: https://www.python.org/dev/peps/pep-0479/#examples The use of the for loop strongly implies that the loop body will be executed once for each thing in the iterable, which isn't true if you next() it in the body. Legal? Sure. Confusing? Definitely. ChrisA

On Fri, Nov 21, 2014 at 2:19 AM, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:

On 21.11.2014 00:51, Guido van Rossum wrote:

...

On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de <mailto:wolfgang.maier@biologie.uni-freiburg.de>> wrote:

[...] Hmm, I'm not convinced by these toy examples, but I did inspect some of my own code for incompatibility with the proposed change. I found that there really is only one recurring pattern I use that I'd have to change and that is how I've implemented several file parsers. I tend to write them like this:

def parser (file_object): while True: title_line = next(file_object) # will terminate after the last record

try: # read and process the rest of the record here except StopIteration: # this record is incomplete raise OSError('Invalid file format') yield processed_record

There's probably something important missing from your examples. The above while-loop is equivalent to

for title_line in io_object: ...

My reason for not using a for loop here is that I'm trying to read from a file where several lines form a record, so I'm reading the title line of a record (and if there is no record in the file any more I want the parser generator to terminate/return. If a title line is read successfully then I'm reading the record's body lines inside a try/except, i.e. where it says "# read and process the rest of the record here" in my shortened code I am actually calling next several times again to retrieve the body lines (and while reading these lines an unexpected StopIteration in the IOWrapper is considered a file format error). I realize that I could also use a for loop and still call next(file_object) inside it, but I find this a potentially confusing pattern that I'm trying to avoid by using the while loop and all explicit next(). Compare:

for title_line in file_object: record_body = next(file_object) # in reality record_body is generated using several next calls # depending on the content found in the record body while it's read yield (title_line, record_body)

vs

while True: title_line = next(file_object) body = next(file_object) yield (title_line, body)

To me, the for loop version suggests to me that the content of file_object is read in line by line by the loop (even though the name title_line tries to hint at this being not true). Only when I inspect the loop body I see that further items are retrieved with next() and, thus, skipped in the for iteration. The while loop, on the other hand, makes the number of iterations very clear by showing all of them in the loop body.

Would you agree that this is justification enough for while instead of for or is it only me who thinks that a for loop makes the code read awkward ?

Now that you have explained it I see your point.

If you're okay with getting RuntimeError instead of OSError for an

...

undesirable StopIteration, you can just drop the except clause altogether.

Right, I could do this if the PEP-described behavior was in effect today.

So shouldn't you be voting *for* the PEP? -- --Guido van Rossum (python.org/~guido)

On Fri, Nov 21, 2014 at 9:58 AM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:

I find this example a compelling argument against the PEP. Personally, I'm dealing a lot more often with refactoring a generator function into a iterator class than I'm rewriting generator expressions into comprehensions (at least the exotic kinds that would reveal their inequality). So for me at least, the burden of having to remember that I can let (and should let) StopIteration bubble up inside __next__, but not in generator functions weighs in heavier than the equality argument and the protection against hard-to-diagnose (but rarely occurring) bugs in nested generator functions.

Compare my earlier response to Steven, though: it's not difficult to refactor a generator into a helper-generator, rather than refactor a generator into a helper-__next__. This proposal would force a decoupling of generator protocol from __next__ protocol. The ugliness in Steven's examples comes from trying to use a __next__ helper in a generator. It'd be just as ugly trying to refactor __getitem__ to make use of a __getattr__ helper - you'd have to catch AttributeError and turn it into KeyError at the boundary between the two protocols. ChrisA

On Fri, Nov 21, 2014 at 10:24 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

Please let me know if I'm reading the PEP correctly. Does the proposal break all existing code in generators that uses next() to raise StopIteration or that raises StopIteration explicitly?

For example, here is the pure python recipe for itertools.accumulate() show in the docs at https://docs.python.org/3/library/itertools.html#itertool-functions :

def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) total = next(it) yield total for element in it: total = func(total, element) yield total

The case where the iterable is empty would now raise, yes.

Or would it break the traditional examples of how to write something like izip() using a generator?

def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2

assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]

Yes, this would be affected. This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing.

Also, the PEP motivation seemed somewhat weak. Instead of listing known bugs or real-world development difficulties, it seems to hinge almost entirely some "being surprised" that list comprehensions and generator expressions aren't the same in every regard (they aren't).

The main point is one of exceptions being silently suppressed. Iterator protocol involves the StopIteration exception; generator protocol doesn't, yet currently a generator that raises StopIteration will quietly terminate. It's as if every generator is wrapped inside "try: ..... except StopIteration: pass". Would you accept any function being written with that kind of implicit suppression of any other exception?

P.S. On a more general note, I think that our biggest problem in the Python world is getting people to switch to Python 3. If we really want that to happen, we should develop a strong aversion to proposals that further increase the semantic difference between Python 2 and Python 3.

The recommended form of the code will work exactly the same way in both versions: explicitly catching StopIteration and using it as a signal that the function should terminate. The only difference is the behaviour of the non-recommended practice of allowing an exception to bubble part-way and then be implicitly caught. ChrisA

On Nov 21, 2014, at 10:55 PM, Nick Coghlan wrote:

I suspect enough evidence of breakage is accumulating to tip the balance back to "not worth the hassle", but it would also be possible to just *add* the "from __future__ import generator_stop" feature, and postpone a decision on making that the only available behaviour.

I have no opinion on the actual PEP, but I'm not sure the above is a good resolution. future imports should be for things that have a clear path to default behavior in some future release. I don't think we should incur technical debt to future-ize a feature that won't eventually get adopted. Such a thing will just be another wart that will be difficult to remove for backward compatibility. Cheers, -Barry

On Sat, Nov 22, 2014 at 3:30 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote:

...

Yes, this would be affected. This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing.

But generators and iterators *are the same thing*. (Generator functions are not iterators, but generators themselves are.) Iterators don't have a specific type, but they obey the iterator protocol:

I can write many other factory functions which return iterators. They are not, themselves, iterators, and therefore should not be expected to follow iterator protocol. def gen(): return iter([1,2])

py> it = gen() py> iter(it) is it True py> hasattr(it, '__next__') True

`it` is an iterator.

The above function works with those tests, too. Generator functions are functions that return iterators, and the __next__ method of the returned object is what follows iterator protocol.

...

The main point is one of exceptions being silently suppressed. Iterator protocol involves the StopIteration exception; generator protocol doesn't, yet currently a generator that raises StopIteration will quietly terminate. It's as if every generator is wrapped inside "try: ..... except StopIteration: pass". Would you accept any function being written with that kind of implicit suppression of any other exception?

Yes.

That's how the classic pre-iterator iteration protocol works:

py> class K: ... def __getitem__(self, i): ... if i == 5: raise IndexError ... return i ... py> x = K() py> list(x) [0, 1, 2, 3, 4]

That's following getitem protocol, and it's part of that protocol for the raising of IndexError to be the way of not returning any value. But what's more surprising is that raising StopIteration will also silently halt iteration, which I think is not good:

...

...

class K: def __getitem__(self, i): if i == 5: raise StopIteration return i

...

...

list(K()) [0, 1, 2, 3, 4]

Context managers support suppressing any exception which occurs:

If the suite was exited due to an exception, and the return value from the __exit__() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement.

https://docs.python.org/3/reference/compound_stmts.html#the-with-statement

Context managers get a chance to function like a try/except block. If one silently and unexpectedly suppresses an exception, it's going to be surprising; but more likely, it's as clear and explicit as an actual try/except block. This isn't "as soon as you use a 'with' block, any XyzError will jump to the end of the block and keep going". ChrisA

On Fri, Nov 21, 2014 at 08:52:59AM -0800, Ethan Furman wrote:

On 11/21/2014 08:30 AM, Steven D'Aprano wrote:

...

But generators and iterators *are the same thing*. (Generator functions are not iterators, but generators themselves are.) Iterators don't have a specific type, but they obey the iterator protocol:

Um, no, they aren't. One cannot 'send()' into any ol' iterator.

"Must not support send()" has never been part of the definition of iterators. The `Iterator` ABC also recognises generators as iterators: py> def gen(): ... yield 1 ... py> from collections import Iterator py> isinstance(gen(), Iterator) True and they are documented as iterators: Python’s generators provide a convenient way to implement the iterator protocol. If a container object’s __iter__() method is implemented as a generator, it will automatically return an iterator object (technically, a generator object) supplying the __iter__() and __next__() methods. https://docs.python.org/3/library/stdtypes.html#generator-types I don't understand where this idea that generators aren't iterators has come from, unless it is confusion between the generator *function* and the generator object itself. -- Steven

On Nov 21, 2014, at 8:52, Ethan Furman <ethan@stoneleaf.us> wrote:

On 11/21/2014 08:30 AM, Steven D'Aprano wrote:

...

But generators and iterators *are the same thing*. (Generator functions are not iterators, but generators themselves are.) Iterators don't have a specific type, but they obey the iterator protocol:

Um, no, they aren't. One cannot 'send()' into any ol' iterator.

Generators are a subtype of iterators. They support the iterator protocol completely, and in the same way as any other iterator. They also support extensions to that protocol--e.g., send(). But they also have a relationship to a generator function or generator expression, which you could call a "protocol" but if so it's not one expressible at the level of the language. I think that leads to a bit of confusion when speaking loosely. When someone says "the generator protocol vs. the iterator protocol" the "obviously correct" meaning is send and throw, but it's not what people always mean. Then again, the word "generator" itself leads to confusion when speaking loosely. Maybe it would be clearer if "generator" had no meaning; generator functions return generator iterators. But I don't think this confusion has caused serious problems over the decades, so I doubt the more minor confusion at issue here is likely to be serious.

On Sat, Nov 22, 2014 at 4:51 AM, Chris Barker <chris.barker@noaa.gov> wrote:

Anyway, I just went back and read the PEP, and I'm still confused -- would the PEP make generators behave more like iterator classes, or less like them?

Neutral. A generator function, an iterator class, etc, etc, etc, exists solely to construct an iterator. That iterator has a __next__ method, which either returns a value or raises StopIteration, or raises some other exception (which bubbles up). There are two easy ways to write iterators. One is to construct a class: class Iter: def __init__(self): self.x = 0 def __iter__(self): return self def __next__(self): if self.x == 3: raise StopIteration self.x += 1 return self.x Another is to write a generator function: def gen(): yield 1 yield 2 yield 3 Both Iter and gen are callables which return iterators. Both of them will produce three integers and then raise StopIteration. Both will, as is good form for iterators, continue to raise StopIteration thereafter. And neither Iter nor gen is, itself, an iterator. One is a class which constructs iterators. The other is a generator function, which also constructs iterators. That's all. In Iter.__next__, I wrote code which chose between "return" and "raise StopIteration" to define its result; in gen(), I wrote code which chose between "yield" and "return" (in this case, the implicit return at the end of the function) to define its result. The only change made by this proposal is that StopIteration becomes, in a generator, like any other unexpected exception. It creates a separation between "iterator protocol" (which is implemented by __next__) and "generator protocol" (which is written in the body of a function with 'yield' in it). ChrisA

On Sat, Nov 22, 2014 at 3:53 AM, Chris Barker <chris.barker@noaa.gov> wrote:

As you can write a "generator" in the sense above in a class that supports the iterator protocol (and, can, in fact, write an "iterator" with a generator function), then I say that generator functions really are only syntactic sugar -- they are short and sweet and do much of the book keeping for you.

If you want to consider them that way, then sure - but part of the bookkeeping they do for you is the management of the StopIteration exception. That becomes purely an implementation detail. You don't actually use it when you write a generator function. ChrisA

On Fri, Nov 21, 2014 at 2:29 PM, Guido van Rossum <guido@python.org> wrote:

Honestly, I'm a bit lost -- but my point is this -- generators and

...

iterators should behave as much the same as possible.

I'm sorry you see it that way; we must have done a terrible job explaining this in the past. :-(

well, others have found examples in old docs that mingle StopIteration and generators...so I guess so, but I'm not sure I'm that misinformed. It still seems to me that there are two ways to write the same thing. The behavior for the *consumer* of the iteration is unchanged

got it -- the issue at hand is what happens to a StopIteration that is raised by something the generator calls. I think the point of this PEP is that the author og a generator function is thinking about using "yield" to provide the next value, and return (explicit or implicit) to stop the generation of objects. That return is raise a StopIteration, but the author isn't thinking about it. So why would they want to think about having to trap StopIteration when calling other functions. While the author of a iterator class is thinking about the __next__ method and raising a StopIteration to terminate. So s/he would naturally think about trapping StopIteration when calling functions? I suppose that makes some sense, but to me it seems like a generator function is a different syntax for creating what is essentially the same thing -- why shouldn't it have the same behavior? and of you are writing a generator, presumably you know how it's going to get use -- i.e. by somethign that expects a StopIteration -- it's not like you're ignorant of the whole idea. Consider this far fetched situation: Either a iterator class or a generator function could take a function object to call to do part of its work. If that function happened to raise a StopIteration -- now the user would have to know which type of object they were workign with, so they would know how to handle the termination of the iter/gener-artion OK -- far more far fetched than the proceeding example of confusion, but the point is this: AFAIU, the current distinction between generators and iterators is how they are written -- i.e. syntax, essentially. But this PEP would change the behavior of generators in some small way, creating a distinction that doesn't currently exist. So, again, the PEP does not change anything about iterators, and generators

will continue to follow the iterator protocol. The change is only for generator authors

I guess this is where I'm not sure -- it seems to me that the behavior of generators is being change, not the syntax -- so while mostly of concern to generator authors, it is, in fact, a chance in behavior that can be seen by the consumer of (maybe only an oddly designed) generator. In practice, that difference may only matter to folks using that particular hack in generator expression, but it is indeed a change. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Sat, Nov 22, 2014 at 10:40:37AM +1100, Chris Angelico wrote:

Let's suppose you use a Python-like macro language to generate Python code. In the macro language, you can write stuff like this:

Is there really any point in hypothesing imaginary macro languages when we have a concrete and existing language (Python itself) to look at? [snip made-up example]

This is a reasonably plausible macro language, right? It's basically still a class definition, but it lets you leave out a whole bunch of boilerplate. Now, the question is: As you write the simplified version, should you ever need to concern yourself with StopIteration?

Sure, why not? It is part of the concrete protocol: iterators raise StopIteration to halt. That's not a secret, and it is not an implementation detail, it is a concrete, public part of the API.

I posit no, you should not; it's not a part of the macro language at all.

This is why talking about imaginary macro languages is pointless. You say it is not part of the macro language. I say it is. Since the language doesn't actually exist, who is to say which is right? In real Python code, "raise StopIteration" does exist, and does work in generators. Sometimes the fact that it works is a nuisance, when you have an unexpected StopIteration. Sometimes the fact that it works is exactly what you want, when you have an expected StopIteration. You seem to think that allowing a generator function to delegate the decision to halt to a helper function is a Bad Thing. I say it is a Good Thing, even if it occasionally makes buggy code a bit harder to debug. [...]

...

and of you are writing a generator, presumably you know how it's going to get use -- i.e. by somethign that expects a StopIteration -- it's not like you're ignorant of the whole idea.

Not necessarily. Can we get someone here who knows asyncio and coroutines, and can comment on the use of such generators?

What about them? I don't understand your question. -- Steven

On Sat, Nov 22, 2014 at 2:57 AM, Chris Angelico <rosuav@gmail.com> wrote:

If you have a lengthy nested chain of coroutines, and one of them unexpectedly raises StopIteration, is it right for something to quietly terminate, or should the exception bubble up and be printed to console?

Couldn't you have a nested pile of iterator classes as well that would exhibit the exact same behavior? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

got it -- the issue at hand is what happens to a StopIteration that is raised by something the generator calls.

I think the point of this PEP is that the author og a generator function is thinking about using "yield" to provide the next value, and return (explicit or implicit) to stop the generation of objects. That return is raise a StopIteration, but the author isn't thinking about it.

So why would they want to think about having to trap StopIteration when calling other functions.

While the author of a iterator class is thinking about the __next__ method and raising a StopIteration to terminate. So s/he would naturally think about trapping StopIteration when calling functions?

I suppose that makes some sense, but to me it seems like a generator function is a different syntax for creating what is essentially the same thing -- why shouldn't it have the same behavior?

and of you are writing a generator, presumably you know how it's going to get use -- i.e. by somethign that expects a StopIteration -- it's not like you're ignorant of the whole idea.

Consider this far fetched situation:

Either a iterator class or a generator function could take a function object to call to do part of its work. If that function happened to raise a StopIteration -- now the user would have to know which type of object they were workign with, so they would know how to handle the termination of the iter/gener-artion

I think this is a good point. Maybe a way to obtain equivalency to the generator functions in this case is to "break" this example for the iterator object as well, in that StopIteration has to be raised in the frame of the generator object; if it raised in a different context, e.g., a function called by __next__, that StopIteration should also be converted to a RuntimeError similar to what is proposed in the PEP for the generator functions. Maybe this is not what Chris intends to happen, but it would make things consistent. -Alexander

On Tue, Nov 25, 2014 at 3:38 AM, Chris Barker <chris.barker@noaa.gov> wrote:

On Fri, Nov 21, 2014 at 4:56 PM, Alexander Heger <python@2sn.net> wrote:

...

Maybe a way to obtain equivalency to the generator functions in this case is to "break" this example for the iterator object as well, in that StopIteration has to be raised in the frame of the generator object; if it raised in a different context, e.g., a function called by __next__, that StopIteration should also be converted to a RuntimeError similar to what is proposed in the PEP for the generator functions. Maybe this is not what Chris intends to happen, but it would make things consistent.

I"mn not sure which Chris you are refering to, but for my part, yes and no:

That's one of the perils of geeky communities - there'll often be multiple people named Chris. I have a brother named Michael who's a rail enthusiast, and that world has the same issue with his name - he was once in a car with three other people named Michael.

Yes, that would keep iterator classes and generator functions consistent, which would be a good thing.

No: I don't think we should do that -- StopIteration is part of the iterator protocol -- generators are another way to write something that complies with the iterator protocol -- generators should handle StopIteration the same way that iterator classes do.

I've done my "explain it twice, then shut up" on this subject, so I'll just point you to the list archive, where it's been stated clearly that generators are like __iter__, not like __next__. Please, could you respond to previously-given explanations, rather than simply restating that generators should be like __next__? I'd like to move forward with that discussion, rather than reiterating the same points. ChrisA

On Tue, Nov 25, 2014 at 4:18 AM, Chris Barker <chris.barker@noaa.gov> wrote:

On Mon, Nov 24, 2014 at 9:06 AM, Chris Angelico <rosuav@gmail.com> wrote:

...

I've done my "explain it twice, then shut up" on this subject, so I'll just point you to the list archive, where it's been stated clearly that generators are like __iter__, not like __next__. Please, could you respond to previously-given explanations, rather than simply restating that generators should be like __next__?

I'm not sure if I've responded or not to previously given explanations -- but you're right, it's time for me to shut up having made my point, too.

Well, there is probably more to be said about this - along the lines of *why* generators ought to be more like iterators. (They're iterables, not iterators.) It's just that we seem to be rehashing the same arguments - or maybe that's just my impression, as there's been discussion on three different channels (-ideas, -dev, and the issue tracker - mercifully very little on the latter). ChrisA

On Mon, Nov 24, 2014 at 10:02 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico <rosuav@gmail.com> wrote:

...

Well, there is probably more to be said about this - along the lines of *why* generators ought to be more like iterators. (They're iterables, not iterators.)

If generators are not iterators, how do you explain this?

...

...

...

import collections def g(): ... yield 42 ... isinstance(g(), collections.Iterator) True

I think Chris A was overzealous here. The word "generator" is ambiguous; it can refer to either a generator function (a function definition containing at least one "yield") or to the object you obtain by calling a generator function. The latter is definitely an iterator (it has a __next__ method). You can't really call a generator function an iterable (since calling iter() on it raises TypeError) but it's not an iterator either. For the rest see my explanation in response to Mark Shannon in python-dev: http://code.activestate.com/lists/python-dev/133428/ -- --Guido van Rossum (python.org/~guido)

On Mon, Nov 24, 2014 at 5:48 PM, Chris Angelico <rosuav@gmail.com> wrote:

As Guido said, "generator" is ambiguous; though I was inaccurate as well. A generator *object* is, as you show above, an iterator; a generator *function* is not actually iterable, but it is an iterator factory. An iterator class is also an iterator factory.

This is correct, and I don't think there is any ambiguity:

...

...

def g(): yield 42 ... type(g) <class 'function'> type(g()) <class 'generator'>

As explained in PEP 255, "a Python generator is a kind of Python iterator[1], but of an especially powerful kind." The other term introduced by PEP 255 is "generator function": "A function that contains a yield statement is called a generator function." In my view, PEP 479 naturally follows from careful reading of PEP 225. All one needs to understand is the difference between a function that returns an iterator and its value.

On 21.11.2014 12:24, Raymond Hettinger wrote:

...

On Nov 15, 2014, at 1:29 AM, Chris Angelico <rosuav@gmail.com <mailto:rosuav@gmail.com>> wrote:

Abstract ========

This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat.

Please let me know if I'm reading the PEP correctly. Does the proposal break all existing code in generators that uses next() to raise StopIteration or that raises StopIteration explicitly?

For example, here is the pure python recipe for itertools.accumulate() show in the docs at https://docs.python.org/3/library/itertools.html#itertool-functions :

def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) total = next(it) yield total for element in it: total = func(total, element) yield total

Or would it break the traditional examples of how to write something like izip() using a generator?

def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2

assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')]

Since I already learnt quite a lot from following this thread: I checked yesterday what the docs have to say about the pure-python equivalent of python3's zip() because I expected it to look like the above izip recipe (making it incompatible with the PEP behavior). However, I found that the given equivalent code is: def zip(*iterables): # zip('ABCD', 'xy') --> Ax By sentinel = object() iterators = [iter(it) for it in iterables] while iterators: result = [] for it in iterators: elem = next(it, sentinel) if elem is sentinel: return result.append(elem) yield tuple(result) i.e., there is no unprotected next call in this example. What surprised me though is that the protection here is done via the default argument of next() while more typically you'd use a try/except clause. So what's the difference between the two ? Specifically, with a default value given will next just catch StopIteration, which you could do less verbosely yourself and/or is there some speed gain from the fact that the Error has to be propagated up one level less ? Is there a guideline when to use try/except vs. next with a default value ? Thanks, Wolfgang

On 11/25/2014 12:03 AM, Chris Angelico wrote:

On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:

...

In addition, the PEP leaves an iterator's __next__() method as the only reasonable place where user-code should raise StopIteration. So I would like to argue that instead of just turning StopIteration into some other error when it's about to bubble out of a generator frame, it should be converted whenever it bubbles out of *anything except an iterator's __next__()*. This would include comprehensions, but also any other code.

There'd have to be a special case for next(), where StopIteration is part of the definition of the function. The question then becomes, what's the boundary where StopIteration is converted?

The current proposal is quite simple. All the conversion happens in the one function that (re)starts a generator frame, gen_send_ex() in Objects/genobject.c. To do this for other functions, there'd need to be some way of saying which ones are allowed to raise StopIteration and which aren't.

Well, I'm not familiar with every implementation detail of the interpreter so I can't judge how difficult to implement certain things would be, but one solution that I could think of is: allow StopIteration to be raised anywhere, but let it bubble up only *one* frame. So if the next outer frame does not deal with it, the exception would be converted to UnhandledStopIteration (or something else) when it's about to bubble out of that outer frame. The builtin next() would simply reset the frame count by catching and reraising StopIteration raised inside its argument (whether that's an iterator's __next__ or a generator; note that in this scenario using raise StopIteration instead of return inside a generator would remain possible). Examples of what would happen: using next on a generator that raises StopIteration explicitly: => next catches the error and reraises StopIteration using next on a generator that returns: => next behaves like currently, raising StopIteration using next on the __next__ method of an iterator: => next catches the error and reraises StopIteration every direct call of an iterator's __next__ method: => has to be guarded by a try/except StopIteration Likewise in the first three cases, the calling frame, which resumes when next returns, (and only this frame) is given a chance to handle the error. If that doesn't happen (i.e. the error would bubble out) it gets converted. So different from the current PEP where a StopIteration must be dealt with explicitly using try/except only inside generators, but bubbles up everywhere else, here StopIteration will be special everywhere, i.e., it must be passed upwards explicitly through all frames or will get converted. Back to Steven's generator expression vs comprehension example: iterable = [iter([])] list(next(x) for x in iterable) would raise UnhandledStopIteration since there is no way, inside the generator expression to catch the StopIteration raised by next(x). ... and if that's all complete nonsense because of some technical detail I'm not aware of, then please excuse my ignorance. Wolfgang

On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker@noaa.gov> wrote:

[...] I like how it support my goal of keeping the behavior of iterator classes and generators consistent.

From the POV of *implementor*, iterator classes and generators have never been at all alike -- one uses *return* to produce a new value, the other uses *yield*. Returning without a value from a __next__() method is seen as

This is a great summary of the general confusion I am trying to clear up. The behavior of all types of iterators (including generators) from the *caller's* perspective is not in question and is not changing. It is very simple: you call next(it) (or it.__next__(), and it returns either the next value or raises StopIteration (and any other exception is, indeed, an exception). producing a None value by the caller; returning from a generator is translated into a StopIteration which will be interpreted by the caller as the end of series. -- --Guido van Rossum (python.org/~guido)

On Tue, Nov 25, 2014 at 12:31 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum <guido@python.org> wrote:

...

On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker <chris.barker@noaa.gov> wrote:

...

[...] I like how it support my goal of keeping the behavior of iterator classes and generators consistent.

This is a great summary of the general confusion I am trying to clear up. The behavior of all types of iterators (including generators) from the *caller's* perspective is not in question and is not changing. It is very simple: you call next(it) (or it.__next__(), and it returns either the next value or raises StopIteration (and any other exception is, indeed, an exception).

From the POV of *implementor*, iterator classes and generators have never been at all alike -- one uses *return* to produce a new value, the other uses *yield*. Returning without a value from a __next__() method is seen as producing a None value by the caller; returning from a generator is translated into a StopIteration which will be interpreted by the caller as the end of series.

Once you start nesting these things, the distinction between "implementor" and "caller" gets mingled.

Hm. An implementer of one protocol is likely the caller of many other protocols. It's not clear that calling something that implements the *same* protocol should deserve special status. For example I could be implementing an iterator processing the lines of a CSV file. Inside this iterator I may be using another iterator that loops over the fields of the current line. (This doesn't sound too far-fetched.) But if I run out of fields in a line, why should that translate into terminating the outer iterator? And the outer iterator may itself be called by another, more outer iterator that iterators over a list of files.

And I think this is all about how nested generators behave, yes?

The inner iterator doesn't have to be a generator (apart from send() and throw(), they have the same interface). And the point of the PEP is that an exhausted inner iterator shouldn't be taken to automatically terminate the outer one. (You could point out that I don't do anything about the similar problem when the outer iterator is implemented as a class with a __next__() method. If I could, I would -- but that case is different because there you *must* raise StopIteration to terminate the iteration, so it becomes more similar to an accidental KeyError being masked when it occurs inside a __getitem__() method.)

If I am implementing a iterator of some sort (generator function or iterator class), and I call next() inside my code, then I am both a implementor and caller. And if I'm also writing helper functions, then I need to know about how StopIteration will be handled, and it will be handled a bit differently by generators and iterator classes.

A helper function also defines an interface. If you are writing a helper function for a generator (and the helper function is participating in the same iteration as the outer generator, i.e. not in the CSV fies / lines / fields example), the best way to do it is probably to write it as a helper generator, and use "yield from" in the outer generator.

But not a big deal, agreed, probably a much smaller deal that all the other stuff you'd better understand to write this kind of code anyway.

Which I'm sorry to see is much less widely understood than I had assumed. -- --Guido van Rossum (python.org/~guido)

On Tue, Nov 25, 2014, at 15:31, Chris Barker wrote:

Once you start nesting these things, the distinction between "implementor" and "caller" gets mingled. And I think this is all about how nested generators behave, yes?

For something more concrete, we can consider a naive implementation of iteration over adjacent pairs: def pairs(x): i = iter(x) while True: yield next(i), next(i)

...

...

list(pairs(range(1))) [] list(pairs(range(2))) [(0, 1)] list(pairs(range(3))) [(0, 1)] list(pairs(range(4))) [(0, 1), (2, 3)] list(pairs(range(5))) [(0, 1), (2, 3)] list(pairs(range(6))) [(0, 1), (2, 3), (4, 5)]

To work under the new paradigm, you need to catch StopIteration explicitly: def pairs(x): i = iter(x) while True: try: a = next(i) b = next(i) except StopIteration: return yield a, b

On Wed, Nov 26, 2014 at 8:56 AM, <random832@fastmail.us> wrote:

For something more concrete, we can consider a naive implementation of iteration over adjacent pairs:

def pairs(x): i = iter(x) while True: yield next(i), next(i)

Okay, it's simple and naive. How about this version: def pairs(x): i = iter(x) for val in i: yield val, next(i) Also simple, but subtly different from your version. What's the difference? Will it be obvious to everyone who reads it? ChrisA

On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going to raise a StopIteration, which under PEP 479 will become a RunTimeError.

Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :)

Yes, I meant the silent termination of an iterable with an odd number of items. That's pretty subtle. ChrisA

On Wed, Nov 26, 2014 at 11:46 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

On 11/25/2014 04:37 PM, Chris Angelico wrote:

...

On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

...

I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going to raise a StopIteration, which under PEP 479 will become a RunTimeError.

Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :)

Yes, I meant the silent termination of an iterable with an odd number of items. That's pretty subtle.

You mean an even number of items, yes?

Presumably the even case is the correct one. It's intended to work that way. If you give it an odd number of items, pre-479 they'll both silently terminate. (Post-479, the next(),next() one will always raise RuntimeError, which indicates clearly that it's not written appropriately, but that's not subtle.) ChrisA

On Tue, Nov 25, 2014, at 22:51, Juancarlo Añez wrote:

On Tue, Nov 25, 2014 at 5:26 PM, <random832@fastmail.us> wrote:

...

def pairs(x): i = iter(x) while True: yield next(i), next(i)

That's not too pythonic, and trying to support non-pythonic code while evolving the language is a dead-end street.

Out of curiosity, what explicit uses of next are pythonic?

The web documents pythonic ways to obtain pairs from an iterable:

def pairs(x): i = iter(x) return zip(i, i)

Or even:

def pairs(x): return zip(*[iter(x)]*2)

See, both of those seem way too implicit to me.

On Wed, Nov 26, 2014 at 4:30 AM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:

Well, I'm not familiar with every implementation detail of the interpreter so I can't judge how difficult to implement certain things would be, but one solution that I could think of is:

I don't know much about the internal details of CPython either, but let's just ignore that for the moment and consider specs for the Python language. AFAIK, not one of the concerns raised (by PEP 479 or your proposal here) is CPython-specific.

allow StopIteration to be raised anywhere, but let it bubble up only *one* frame. So if the next outer frame does not deal with it, the exception would be converted to UnhandledStopIteration (or something else) when it's about to bubble out of that outer frame. The builtin next() would simply reset the frame count by catching and reraising StopIteration raised inside its argument (whether that's an iterator's __next__ or a generator; note that in this scenario using raise StopIteration instead of return inside a generator would remain possible).

Interesting. Makes a measure of sense, and doesn't have much magic to it.

So different from the current PEP where a StopIteration must be dealt with explicitly using try/except only inside generators, but bubbles up everywhere else, here StopIteration will be special everywhere, i.e., it must be passed upwards explicitly through all frames or will get converted.

Back to Steven's generator expression vs comprehension example:

iterable = [iter([])] list(next(x) for x in iterable)

would raise UnhandledStopIteration since there is no way, inside the generator expression to catch the StopIteration raised by next(x).

Downside of this is that it's harder to consciously chain iterators, but maybe that's a cost that has to be paid. Suggestion for this: Have a new way of "raise-and-return". It's mostly like raise, except that (a) it can't be caught by a try/except block in the current function (because there's no point), and (b) it bypasses the "this exception must not pass unnoticed". It could then also be used for anything else that needs the "return any object, or signal lack of return value" option, covering AttributeError and so on. So it'd be something like this: class X: def __iter__(self): return self def __next__(self): if condition: return value signal StopIteration The 'signal' statement would promptly terminate the function (not sure exactly how it'd interact with context managers and try/finally, but something would be worked out), and then raise StopIteration in the calling function. Any other StopIteration which passes out of a function would become a RuntimeError. Magic required: Some way of knowing which exceptions should be covered by this ban on bubbling; also, preferably, some way to raise StopIteration in the calling function, without losing the end of the backtrace. This could be a viable proposal. It'd be rather more complicated than PEP 479, though, and would require a minimum of five hundred bikeshedding posts before it comes to any kind of conclusion, but if you feel this issue is worth it, I'd certainly be an interested participant in the discussion. ChrisA

On Wed, Nov 26, 2014 at 4:53 AM, Guido van Rossum <guido@python.org> wrote:

It's not viable. It will break more code than PEP 479, and it will incur a larger interpreter overhead (every time any exception bubbles out of any frame we'd have to check whether it is (derived from) StopIteration and replace it, rather than only when exiting a generator frame. (The check for StopIteration is relatively expensive -- it's easy to determine that an exception *is* StopIteration, but in order that it doesn't derive from StopIteration you have to walk the inheritance tree.)

Please stop panicking.

Heh. If that last comment was aimed at me, I wasn't panicking... I just thought it an intellectually curious concept, having a measure of interpreter support for the "not-a-return-value" concept. ChrisA

On 16 November 2014 00:37, Chris Angelico <rosuav@gmail.com> wrote:

...

...

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

There's an additional subtlety with this idea: if we add a new GeneratorReturn exception as a subclass of StopIteration, then generator iterators would likely also have to change to replace GeneratorReturn with a regular StopIteration (chaining appropriately via __cause__, and copying

On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: the

...

return value across).

Would have to do so automatically, meaning this is no simpler than the current proposal? Or would have to be always explicitly written to handle it?

When GeneratorReturn escaped a generator frame, the interpreter would automatically convert it into an ordinary StopIteration instance. It's still simpler because it won't need the __future__ dance (as it doesn't involve any backwards incompatible changes).

...

I definitely see value in adding a GeneratorReturn subclass to be able to tell the "returned" vs "raised StopIteration" cases apart from outside the generator (the current dance in contextlib only works because we have existing knowledge of the exact exception that was thrown in). I'm substantially less convinced of the benefit of changing generators to no longer suppress StopIteration. Yes, it's currently a rather odd corner case, but changing it *will* break code (at the very least, anyone using an old version of contextlib2, or who are otherwise relying on their own copy of contextlib rather than standard library one).

This is why it's proposed to use __future__ to protect it.

Using __future__ still imposes a large cost on the community - docs need updating, code that relies on the existing behaviour has to be changed, developers need to adjust their mental models of how the language works. There needs to be a practical payoff for those costs - and at the moment, it's looking like we can actually get a reasonably large fraction of the gain without most of the pain by instead pursuing Guido's idea of a separate StopIteration subclass to distinguish returning from the outermost generator frame from raising StopIteration elsewhere in the generator.

If anyone's still using an old version of contextlib2 once 3.7 comes along, it'll break; but is there any reason to use Python 3.7 with a contextlib from elsewhere than its standard library?

Same reason folks use it now: consistent behaviour and features across a range of Python versions. However, that's not the key point - the key point is that working through the exact changes that would need to be made in contextlib persuaded me that I was wrong when I concluded that contextlib wouldn't be negatively affected. It's not much more complicated, but if we can find a fully supported example like that in the standard library, what other things might folks be doing with generators that *don't* fall into the category of "overly clever code that we don't mind breaking"?

(I'm not familiar with contextlib2 or what it offers.)

contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially created it as a proving ground for the idea that eventually become contextlib.ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16 November 2014 01:56, Chris Angelico <rosuav@gmail.com> wrote:

On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

On 16 November 2014 00:37, Chris Angelico <rosuav@gmail.com> wrote:

...

On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan <ncoghlan@gmail.com>

wrote:

...

...

...

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

There's an additional subtlety with this idea: if we add a new GeneratorReturn exception as a subclass of StopIteration, then generator iterators would likely also have to change to replace GeneratorReturn with a regular StopIteration (chaining appropriately via __cause__, and copying the return value across).

Would have to do so automatically, meaning this is no simpler than the current proposal? Or would have to be always explicitly written to handle it?

When GeneratorReturn escaped a generator frame, the interpreter would automatically convert it into an ordinary StopIteration instance.

Okay, let me see if I have this straight. When a 'return' statement (including an implicit one at end-of-function) is encountered in any function which contains a 'yield' statement, it is implemented as "raise GeneratorReturn(value)" rather than as "raise StopIteration(value)" which is the current behaviour. However, if any GeneratorReturn would be raised in any way other than the 'return' statement, it would magically become a StopIteration instead. Is that correct?

That's not quite how generators work. While the "returning from a generator is equivalent to raise StopIteration" model is close enough that it's functionally equivalent to the actual behaviour in most cases (with the main difference being in how try/except blocks and context managers inside the generator react), this particular PEP is a situation where it's important to have a clear picture of the underlying details. When you have a generator iterator (the thing you get back when calling a generator function), there are two key components: * the generator iterator object itself * the generator frame where the code is running When you call next(gi), you're invoking the __next__ method on the *generator iterator*. It's that method which restarts evaluation of the generator frame at the point where it last left off, and interprets any results. Now, there are three things that can happen as a result of that frame evaluation: 1. It hits a yield point. In that case, gi.__next__ returns the yielded value. 2. It can return from the frame. In that case. gi.__next__ creates a *new* StopIteration instance (with an appropriate return value set) and raises it 3. It can throw an exception. In that case, gi.__next__ just allows it to propagate out (including if it's StopIteration) The following example illustrates the difference between cases 2 and 3 (in both cases, there's a StopIteration that terminates the hidden loop inside the list() call, the difference is in where that StopIteration is raised):

...

...

def genreturn(): ... yield ... try: ... return ... except: ... print("No exception") ... raise ... list(genreturn()) [None]

...

...

def genraise(): ... yield ... try: ... raise StopIteration ... except: ... print("Exception!") ... raise ... list(genraise()) Exception! [None]

(The possible outcomes of gi.send() and gi.throw() are the same as those of next(gi). gi.throw() has the novel variant where the exception thrown in may propagate back out) The two change proposals being discussed are as follows: Current PEP (backwards incompatible): Change outcome 3 to convert StopIteration to RuntimeError (or a new exception type). Nothing else changes. Alternative (backwards compatible): Change outcome 2 to raise GeneratorReturn instead of StopIteration and outcome 3 to convert GeneratorReturn to StopIteration. The alternative *doesn't* do anything about the odd discrepancy between comprehensions and generator expressions that started the previous thread. It just adds a new capability where code that knows it's specifically dealing with a generator (like contextlib or asyncio) can more easily tell the difference between outcomes 2 and 3.

This does sound simpler. All the magic is in the boundary of the generator itself, nothing more. If a __next__ method raises either StopIteration or GeneratorReturn, or if any other function raises them, there's no special handling.

All the magic is actually at the generator boundary regardless. The key differences between the two proposals are the decision to keep StopIteration as a common parent exception, and allow it to continue propagating out of generator frames unmodified.

Question: How does it "become" StopIteration? Is a new instance of StopIteration formed which copies in the other's ``value``? Is the type of this exception magically altered? Or is it a brand new exception with the __cause__ or __context__ set to carry the original?

I'd suggest used the exception chaining machinery and creating a new exception with __cause__ and the generator return value set appropriately.

...

(I'm not familiar with

...

contextlib2 or what it offers.)

contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially created it as a proving ground for the idea that eventually become contextlib.ExitStack).

Thanks, I figured it'd be like that. Since contextlib exists in 2.7, is contextlib2 meant to be legacy support only?

contextlib has actually been around since 2.5, but some features (most notably ExitStack) weren't added until much later. Like unittest2, contextlib2 allows access to newer stdlib features on older versions (I haven't used it as a testing ground for new ideas since ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 16 November 2014 01:56, Chris Angelico <rosuav@gmail.com> wrote:

...

Okay, let me see if I have this straight. When a 'return' statement (including an implicit one at end-of-function) is encountered in any function which contains a 'yield' statement, it is implemented as "raise GeneratorReturn(value)" rather than as "raise StopIteration(value)" which is the current behaviour. However, if any GeneratorReturn would be raised in any way other than the 'return' statement, it would magically become a StopIteration instead. Is that correct?

When you call next(gi), you're invoking the __next__ method on the *generator iterator*. It's that method which restarts evaluation of the generator frame at the point where it last left off, and interprets any results.

Now, there are three things that can happen as a result of that frame evaluation:

1. It hits a yield point. In that case, gi.__next__ returns the yielded value.

2. It can return from the frame. In that case. gi.__next__ creates a *new* StopIteration instance (with an appropriate return value set) and raises it

3. It can throw an exception. In that case, gi.__next__ just allows it to propagate out (including if it's StopIteration)

Thank you for explaining. -- Cameron In case others were also oversimplifying in their heads, I've summarized the above into the PEP.

(The possible outcomes of gi.send() and gi.throw() are the same as those of next(gi). gi.throw() has the novel variant where the exception thrown in may propagate back out)

Should that variant affect this proposal? What should happen if you throw StopIteration or GeneratorReturn into a generator?

The two change proposals being discussed are as follows:

Current PEP (backwards incompatible): Change outcome 3 to convert StopIteration to RuntimeError (or a new exception type). Nothing else changes.

Alternative (backwards compatible): Change outcome 2 to raise GeneratorReturn instead of StopIteration and outcome 3 to convert GeneratorReturn to StopIteration.

The alternative *doesn't* do anything about the odd discrepancy between comprehensions and generator expressions that started the previous thread. It just adds a new capability where code that knows it's specifically dealing with a generator (like contextlib or asyncio) can more easily tell the difference between outcomes 2 and 3.

Text along these lines added to PEP, thanks!

...

Question: How does it "become" StopIteration? Is a new instance of StopIteration formed which copies in the other's ``value``? Is the type of this exception magically altered? Or is it a brand new exception with the __cause__ or __context__ set to carry the original?

I'd suggest used the exception chaining machinery and creating a new exception with __cause__ and the generator return value set appropriately.

Makes sense. If the __cause__ is noticed at all (ie this doesn't just quietly stop a loop), it wants to be very noisy.

...

Thanks, I figured it'd be like that. Since contextlib exists in 2.7, is contextlib2 meant to be legacy support only?

contextlib has actually been around since 2.5, but some features (most notably ExitStack) weren't added until much later. Like unittest2, contextlib2 allows access to newer stdlib features on older versions (I haven't used it as a testing ground for new ideas since ExitStack).

If there is breakage from this, it would simply mean "older versions of contextlib2 are not compatible with Python 3.7, please upgrade your contextlib2" - several of the variants make it perfectly possible to write cross-version-compatible code. I would hope that this remains the case. Latest version of PEP text incorporating the above changes: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt (My apologies if this email has gone through more than once. I'm having major issues with my internet connection at the moment, and delivery is failing and being retried. Hopefully it really *is* failing, and not just saying so.) ChrisA

Exactly! The current behavior is not only likely undesirable, but it is also undocumented. Even if parts of stdlib rely on the current behavior, there's no need for a deprecation (read __future__) period. Undocumented features may change any time, because are mostly about implementation quirks (Isn't that rule documented somewhere in the Python docs?). In short: -1 deprecation (__future__); no need, because nothing documented gets broken +1 fix it now (3.5); the fix may be a change in the docs to validate the current behavior, and deprecate it (Yuk!) +1 Nick's design, which kind of leaves it the same and kind of fixes it p.s. What about 2.7? This fix is *not* a new feature. Cheers, -- Juanca On Sat Nov 15 2014 at 1:50:06 PM Steve Dower <Steve.Dower@microsoft.com> wrote:

Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.)

Cheers, Steve

Top-posted from my Windows Phone ------------------------------ From: Chris Angelico <rosuav@gmail.com> Sent: ‎11/‎15/‎2014 1:30 To: python-ideas <python-ideas@python.org> Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators

PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014

Abstract ========

This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat.

Rationale =========

The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct.

Proposal ========

If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_

Consequences to existing code =============================

This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail.

(Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""")

As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7.

Alternate proposals ===================

Supplying a specific exception to raise on return -------------------------------------------------

Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed.

Making return-triggered StopIterations obvious ----------------------------------------------

For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement.

Criticism =========

Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain.

References ==========

.. [1] Initial mailing list comment ( https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

.. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby)

.. [3] Proposal by GvR ( https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

.. [4] Response by Steven D'Aprano ( https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)

Copyright =========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Nov 16, 2014 at 5:20 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:

Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.)

The behaviour selection would have to be based on the generator's definition. This proposal, in all its variants, is about what happens as the generator terminates; if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct. ChrisA

On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:

Chris Angelico wrote:

...

if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct.

This suggests that the use of a __future__ directive is not really appropriate, since it can affect code outside of the module with the __future__ directive in it.

I don't see how that is different from any other __future__ directive. They are all per-module, and if you gain access to an object from another module, it will behave as specified in the module that created it, not the module that imported it. How is this different? -- Steven

(I'm catching up on this thread from the end.) On Sun, Nov 16, 2014 at 5:29 PM, Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:

...

Chris Angelico wrote:

...

if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct.

This suggests that the use of a __future__ directive is not really appropriate, since it can affect code outside of the module with the __future__ directive in it.

I don't see how that is different from any other __future__ directive. They are all per-module, and if you gain access to an object from another module, it will behave as specified in the module that created it, not the module that imported it. How is this different?

Well, let's see. For feature in sorted(__future__.all_feature_names):

absolute_import: Affects implementation of a keyword barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) division: Changes the meaning of one operator. generators: Introduces a keyword nested_scopes: Alters the compilation of source to byte-code(?) print_function: Removes a keyword unicode_literals: Alters the type used for literals with_statement: Introduces a keyword

Apart from the joke, it seems that every __future__ directive is there to affect the compilation, not execution, of its module: that is, once a module has been compiled to .pyc, it shouldn't matter whether it used __future__ or not. Regardless of unicode_literals, you can create bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not entirely sure about division (can you call on true-division without the future directive?), but in any case, it's all done at compilation time, as can be seen interactively:

Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

def division1(x,y): ... return x/y, x//y, x%y ... from __future__ import division def division2(x,y): ... return x/y, x//y, x%y ... import dis dis.dis(division1) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE dis.dis(division2) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_TRUE_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE

So to make this consistent with all other __future__ directives, there would need to be some kind of safe way to define this: perhaps an attribute on the generator object. Something like this:

...

...

...

def gen(x): ... yield x ... raise StopIteration(42) ... g=gen(123) list(g) [123] gen.__distinguish_returns__ = True g=gen(123) list(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in gen StopIteration: 42

The attribute on the function would be what affects behaviour; the __future__ directive applies that attribute to all generator functions in its module (including genexprs). Once the __future__ directive becomes automatic, the attribute can and will be dropped - any code which interrogates it MUST be prepared to stop interrogating it once the feature applies to all modules.

Does that sound reasonable? Should it be added to the PEP?

I agree with you and Steven that this is a fine use of __future__. What a generator does with a StopIteration that is about to bubble out of its frame is up to that generator. I don't think it needs to be a flag on the *function* though -- IMO it should be a flag on the code object. (And the flag should somehow be transferred to the stack frame when the function is executed, so the right action can be taken when an exception is about to bubble out of that frame.) One other small point: let's change the PEP to just propose RuntimeError, and move the "some other exception" to the "rejected ideas" section. -- --Guido van Rossum (python.org/~guido)

On 11/17/2014 03:11 AM, Chris Angelico wrote:

On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum <guido@python.org> wrote:

...

I agree with you and Steven that this is a fine use of __future__. What a generator does with a StopIteration that is about to bubble out of its frame is up to that generator. I don't think it needs to be a flag on the *function* though -- IMO it should be a flag on the code object. (And the flag should somehow be transferred to the stack frame when the function is executed, so the right action can be taken when an exception is about to bubble out of that frame.)

One other small point: let's change the PEP to just propose RuntimeError, and move the "some other exception" to the "rejected ideas" section.

Changes incorporated, thanks! I'm not familiar with the details of stack frame handling, so I've taken the cop-out approach and just quoted you directly into the PEP.

PEP draft: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt

I fixed a typo and syntax and made a content clarification ("generator constructed" -> "generator function constructed"; the generator object itself is constructed when the function is called) and committed the changes. Georg

On Mon, Nov 17, 2014 at 2:27 AM, Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl <g.brandl@gmx.net> wrote:

...

I fixed a typo and syntax and made a content clarification ("generator constructed" -> "generator function constructed"; the generator object itself is constructed when the function is called) and committed the changes.

Thanks Georg! This means today's version is now visible here:

http://legacy.python.org/dev/peps/pep-0479/

Off-topic: the new python.org site now supports PEPs, so please switch to URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't like the formatting send a pull request to https://github.com/python/pythondotorg). -- --Guido van Rossum (python.org/~guido)

On 17/11/2014 2:29 p.m., Chris Angelico wrote:

barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)

It determines whether "not equal" is spelled "!=" or "<>", so it fits the pattern of being compile-time-only. -- Greg