On 2 November 2014 02:50, Guido van Rossum wrote:

I think you're on to something. But I think both your examples have a problem, even though your second one "works".

If we weren't forced by backward compatibility I would have made it much harder for StopIteration to "leak out". Currently a generator can either return or raise StopIteration to signal it is done, but I think it would have been better if StopIteration was treated as some kind of error in this case. Basically I think any time a StopIteration isn't caught by a for-loop or an explicit try/except StopIteraton, I feel there is a bug in the program, or at least it is hard to debug.

I'm afraid that ship has sailed, though...

The closest existing example of this kind of generator instance specific StopIteration handling that I can think of is the special case handling of StopIteration in contexlib._GeneratorContextManager.__exit__() (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There, the exception handling differentiates between "a specific StopIteration instance that we just threw into the subgenerator" (which it will allow to propagate) and "any other StopIteration instance, which indicates that the wrapped generator iterator terminated as expected" (which it will suppress). We had that wrong initially - if I recall correctly, it was PJE that noticed the problem before 2.5 was released. However, the only reason we were able to make it work is that we knew the exact identity of the exception we were throwing in, rather than just its type - we don't have that luxury in the general case. Getting back to the behaviour that prompted the thread, like a lot of exception handling quirks, it gets back to being very careful about the scope of exception handlers. In this case, the "next(it)" call is inside a generator expression, and hence inside the scope of the expression's StopIteration handling. By contrast, the comprehension version doesn't *have* any implicit exception handling, so the StopIteration escapes to terminate the containing generator. In terms of changing the behaviour of generator expressions to allow other StopIteration instances to propagate, I believe I do see one possible way to do it that limits the degree of backwards incompatibility. Firstly, you'd need to add a general purpose capability to generator iterators: def _set_default_exception(exc): """Supply a specific StopIteration instance to raise when the generator frame returns None""" ... Normally, when the generator execution falls off the end of the frame by returning, the interpreter raises StopIteration if the result is None, or StopIteration(result) if the result is not None. With the new method, you could set a specific instance to be raised when the underlying result of the frame is None. (Side note: "return" and "raise StopIteration" in a generator function aren't *exactly* the same, as only the former relies on the return->raise conversion supplied by the surrounding generator iterator object) That part would be entirely backwards compatible, and would allow you to distinguish whether calling "next", "send" or "throw" on any generator threw StopIteration because the underlying frame returned None (by checking if the StopIteration instance was the one you configured to be raised on a None result), or because it either returned a non-None value or else something running inside that frame threw StopIteration. The backwards incompatible part would be to then also change generator expressions to set a specific StopIteration instance to be raised when the underlying frame returned, and allow all other StopIteration instances to escape, just as contextlib._GeneratorContextManager.__exit__ allows StopIteration instances thrown from the body of the with statement to escape. I think the end result of such a change would definitely be less surprising, as it would make generator expressions behave more like the corresponding comprehensions, and eliminate a hidden infinite loop bug. However, I'm not sure if it's *sufficiently* less surprising to be worth changing - especially since it would mean incurring a small amount of additional runtime overhead for each generator expression. Regards, Nick. P.S. As additional background on the current difference in behaviour between list comprehensions and generator expressions, that has its roots in the same idiosyncrasy where putting a yield expression inside a comprehension actually *turns it into a generator expression*. Comprehensions are full closures, but they don't contain a yield expression, so you get a normal function, which the interpreter then calls. The interpreter doesn't actually do anything particularly special to make a generator expression instead - it just implicitly inserts a yield expression into the closure, which then automatically makes it a generator function instead of a normal one. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP), because of cases like: def ensure_positive(x): if x<=0: raise StopIteration return True eggs = list(x for x in spam if raise_on_negative(x)) def ensure_positive(x); if x<=0: raise StopIteration return x eggs = list(ensure_positive(x) for x in spam) In both cases, this acts like a "takewhile": eggs ends up as a list of the initial positive values, and the first non-positive value is consumed and discarded. But if you do the same thing with a list comprehension, the comprehension is aborted by the StopIteration, and eggs never gets set (although the same values are consumed from spam, of course). IIRC, you asked me what the performance costs would be of changing listcomps to match, and for a trivial comp it worked out to be about 40% for the naive solution (build a genexpr, call it, call list) and about 20% with specially-optimized bytecode. So everyone agreed that even if this is a bug, that would be too much of a cost for too small of a fix. Of course here the issue is almost the opposite. But they're clearly related; how different comprehensions "leak" exceptions differs. Sent from a random iPhone On Nov 2, 2014, at 2:15, Nick Coghlan wrote:

On 2 November 2014 19:53, Nick Coghlan wrote:

...

On 2 November 2014 02:50, Guido van Rossum wrote:

...

I think you're on to something. But I think both your examples have a problem, even though your second one "works".

If we weren't forced by backward compatibility I would have made it much harder for StopIteration to "leak out". Currently a generator can either return or raise StopIteration to signal it is done, but I think it would have been better if StopIteration was treated as some kind of error in this case. Basically I think any time a StopIteration isn't caught by a for-loop or an explicit try/except StopIteraton, I feel there is a bug in the program, or at least it is hard to debug.

I'm afraid that ship has sailed, though...

The closest existing example of this kind of generator instance specific StopIteration handling that I can think of is the special case handling of StopIteration in contexlib._GeneratorContextManager.__exit__() (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There, the exception handling differentiates between "a specific StopIteration instance that we just threw into the subgenerator" (which it will allow to propagate) and "any other StopIteration instance, which indicates that the wrapped generator iterator terminated as expected" (which it will suppress). We had that wrong initially - if I recall correctly, it was PJE that noticed the problem before 2.5 was released. However, the only reason we were able to make it work is that we knew the exact identity of the exception we were throwing in, rather than just its type - we don't have that luxury in the general case.

Getting back to the behaviour that prompted the thread, like a lot of exception handling quirks, it gets back to being very careful about the scope of exception handlers. In this case, the "next(it)" call is inside a generator expression, and hence inside the scope of the expression's StopIteration handling. By contrast, the comprehension version doesn't *have* any implicit exception handling, so the StopIteration escapes to terminate the containing generator.

Bah, I should have fully read Terry's reply before responding. He's right, it's the tuple call that's suppressing the exception, not the generator expression itself.

That changes the possible solution, by tweaking it to be an optional extension to the iterator protocol, allowing iterators to make the terminating exception configurable.

def __setiterexc__(exc): """Specify the exception instance to raise when the iterator is exhausted""" ...

Iterator consumers (like tuple) could then check for that method and use it to set a specific StopIteration instance, allowing all others to escape.

I believe actually doing this would be adding too much complexity for too little gain, but it *is* possible.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Nov 2, 2014, at 13:00, Terry Reedy wrote:

On 11/2/2014 2:50 PM, Andrew Barnert wrote:

...

This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP),

That purported equivalence is a common meme, which I may have helped spread. If it is implied in the doc, it should be changed. Before I answered on this thread yesterday, I looked for such an implication in the Language Reference (though not the Tutorial), and only found this carefully written description in the expressions chapter.

We looked through this last year and decided nothing needed to be changed in the docs, so I doubt it's worth repeating that effort. IIRC, the tutorial may have been confusing in the past but wasn't as of 3.3, and the only place that might confuse anyone was the what's new in 3.0, which is generally only changed to add notes about things which were un-/re-changed (like u"" strings). But again, even if I'm remembering wrong, I don't think it matters. All that being said, I don't really love the reference docs here. Neither 6.2.8 not anything else explicitly says what the semantics are. It's a pretty obvious guess that the syntax is interpreted the same as for comprehensions (as in 6.2.4), and that the values yielded are those that would be used to produce elements. But the docs don't actually say that.

"In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. Note that the comprehension is executed in a separate scope,"

IE, a comprehension is equivalent to the result of for and if statements in a separate Python function that first initializes a collection object (list, set, or dict) and augments the collection in the innermost scope with the element produced (which is a key,value pair for dicts).

Such an equivalent function would not catch any exception raised by the innermost element expression. On the other hand, collection initializers, such as list or tuple, specifically catch StopIteration when fed an iterable. Hence the two cannot be equivalent when the element expression raises StopIteration.

-- Terry Jan Reedy

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 3 November 2014 13:01, Guido van Rossum wrote:

On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote:

...

On 11/2/2014 2:50 PM, Andrew Barnert wrote:

...

This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP),

That purported equivalence is a common meme, which I may have helped spread.

I may have started it. I was aware of the non-equivalence (only mostly-equivalence) in Python 2 and I wanted to make then identical in Python 3 -- having one construct being exactly equivalent to another reduce the amount of explaining needed. Unfortunately, people had started to depend on the (in my *current* opinion deplorable) behavior of generator expressions in the face of StopIteration thrown by arbitrary parts of the expression or condition, and the equivalence is still imperfect. At least the variable leakage has been fixed.

I think I'm guilty as well - when I was working on the Python 3 changes, getting the *scoping* behaviour to be identical between comprehensions and generator expressions was one of the key objectives, so I regularly described it as making "[x for x in seq]" equivalent to "list(x for x in seq)". I unfortunately didn't notice the remaining exception handling differences at the time, or we might have been able to do something about it for 3.0 :(

However, I don't think this idea has panned out. I haven't done a survey, but I have a feeling that in most cases where an explicit next() call is used (as opposed to a for-loop) there's a try/except Stopiteration around it, and a fair amount if time is wasted debugging situations where a StopIteration unexpectedly escapes and silently interrupts some loop over an unrelated generator (instead of loudly bubbling up to the top and causing a traceback, which would be more debuggable). And the use case of raising StopIteration from a condition used in a generator expression is iffy at best (it makes the condition function hard to use in other contexts, and it calls to attention the difference between generators and comprehensions).

So I will go out on a limb here and say that this was a mistake and if we can think of easing the transitional pain it would be a good thing to fix this eventually.

Having had to do the dance to work around the current behaviour in contextlib, I'm inclined to agree - there's a significant semantic difference between the "this iterable just terminated" StopIteration, and the "something threw StopIteration and nobody caught it", and the current model hides it. However, I also see potentially significant backwards compatibility problems when it comes to helper functions that throw StopIteration to terminate the calling generator - there would likely need to be some kind of thread local state and a helper "iterexit()" builtin and "PyIter_Exit()" C API to call instead of raising StopIteration directly. Making such a change would involve a lot of code churn just to phase out a relatively obscure issue that mostly just makes certain bugs harder to diagnose (as was the case with the generator expression based izip implementation in this thread), rather than directly causing bugs in its own right. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 3 November 2014 15:59, Guido van Rossum wrote:

On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote:

...

However, I also see potentially significant backwards compatibility problems when it comes to helper functions that throw StopIteration to terminate the calling generator - there would likely need to be some kind of thread local state and a helper "iterexit()" builtin and "PyIter_Exit()" C API to call instead of raising StopIteration directly.

That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion -- but you sound like you have helpers for context managers that couldn't be easily dismissed like that.

Sorry, I didn't mean to give that impression. I'm in a similar situation to you in that regard - any specific examples I can think of trip my "that's too obscure to be maintainable" alarm (if there is a reasonable use case, I'd expect it to involve full generators, rather than generator expressions). The code in contextlib relies on the way *generators* handle StopIteration, and if understand your proposal correctly, that would remain unchanged - only ordinary function calls would convert StopIteration to a different exception type, preserving the functional equivalence of a generator returning, raising StopIteration, or having StopIteration thrown into a yield point (it's that last one that contextlib relies on).

...

Making such a change would involve a lot of code churn just to phase out a relatively obscure issue that mostly just makes certain bugs harder to diagnose (as was the case with the generator expression based izip implementation in this thread), rather than directly causing bugs in its own right.

Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix.

FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration.

That's far more elegant than the stateful possibilities I was considering. So generators would continue to leave StopIteration untouched (preserving the equivalence between returning from the frame and explicitly raising StopIteration from the generator body), and only ordinary function invocations would gain the StopIteration -> UnexpectedStopIteration translation (assuming we went with a new exception type)?

It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics).

Aye. I'm reasonably OK with the notion of breaking overly clever (and hence hard to follow) generator expressions, but I'm a little more nervous about any change that means that factoring out "raise StopIteration" in a full generator function would stop working. That said, such a change would bring generator functions fully into line with the "flow control should be locally visible" principle that guided both the with statement design and asyncio - only a local return or raise statement could gracefully terminate the generator, StopIteration from a nested function call would always escape as the new exception type. If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()". Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Nov 2, 2014, at 21:59, Guido van Rossum wrote:

On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote:

...

On 3 November 2014 13:01, Guido van Rossum wrote:

...

On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote:

...

On 11/2/2014 2:50 PM, Andrew Barnert wrote:

...

This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP),

That purported equivalence is a common meme, which I may have helped spread.

I may have started it. I was aware of the non-equivalence (only mostly-equivalence) in Python 2 and I wanted to make then identical in Python 3 -- having one construct being exactly equivalent to another reduce the amount of explaining needed. Unfortunately, people had started to depend on the (in my *current* opinion deplorable) behavior of generator expressions in the face of StopIteration thrown by arbitrary parts of the expression or condition, and the equivalence is still imperfect. At least the variable leakage has been fixed.

I think I'm guilty as well - when I was working on the Python 3 changes, getting the *scoping* behaviour to be identical between comprehensions and generator expressions was one of the key objectives, so I regularly described it as making "[x for x in seq]" equivalent to "list(x for x in seq)".

I unfortunately didn't notice the remaining exception handling differences at the time, or we might have been able to do something about it for 3.0 :(

...

However, I don't think this idea has panned out. I haven't done a survey, but I have a feeling that in most cases where an explicit next() call is used (as opposed to a for-loop) there's a try/except Stopiteration around it, and a fair amount if time is wasted debugging situations where a StopIteration unexpectedly escapes and silently interrupts some loop over an unrelated generator (instead of loudly bubbling up to the top and causing a traceback, which would be more debuggable). And the use case of raising StopIteration from a condition used in a generator expression is iffy at best (it makes the condition function hard to use in other contexts, and it calls to attention the difference between generators and comprehensions).

So I will go out on a limb here and say that this was a mistake and if we can think of easing the transitional pain it would be a good thing to fix this eventually.

Having had to do the dance to work around the current behaviour in contextlib, I'm inclined to agree - there's a significant semantic difference between the "this iterable just terminated" StopIteration, and the "something threw StopIteration and nobody caught it", and the current model hides it.

However, I also see potentially significant backwards compatibility problems when it comes to helper functions that throw StopIteration to terminate the calling generator - there would likely need to be some kind of thread local state and a helper "iterexit()" builtin and "PyIter_Exit()" C API to call instead of raising StopIteration directly.

That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion

The category is usually represented by the even more trivial and even more abusive example: (prime if prime<100 else throw(StopIteration) for prime in primes) I do see these, mostly on places like StackOverflow, where someone was shown this "cool trick" by someone else, used it without understanding it, and now has no idea how to debug his code. (However, a few people on this list suggested it as an alternative to adding some kind of "syntactic takewhile" to the language, so it's possible not everyone sees it as abuse, even though I think you and others called it abuse back then as well.) Anyway, I agree that explicitly disallowing it would make the language simpler, eliminate more bugs than useful idioms, and possibly open the door to other improvements. But if you can't justify this abuse as being actually illegal by a reading of the docs in 3.0-3.4, and people are potentially using it in real code, wouldn't that require a period of deprecation before it can be broken?

-- but you sound like you have helpers for context managers that couldn't be easily dismissed like that.

...

Making such a change would involve a lot of code churn just to phase out a relatively obscure issue that mostly just makes certain bugs harder to diagnose (as was the case with the generator expression based izip implementation in this thread), rather than directly causing bugs in its own right.

Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix.

FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration.

It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics).

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Nov 3, 2014 at 1:38 PM, Greg Ewing wrote:

Guido van Rossum wrote:

FWIW the implementation of my proposal is easy to describe (which the Zen

...

approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration.

Wouldn't that break all existing iterators whose next() method is implemented in Python? Since the code that catches the StopIteration is always in a different frame then.

Or are you only talking about generator frames here, not frames in general?

I have to apologize, I pretty much had this backwards. What I should have said is that a generator should always be terminated by a return or falling off the end, and if StopIteration is raised in the generator (either by an explicit raise or raised by something it calls) and not caught by an except clause in the same generator, it should be turned into something else, so it bubbles out as something that keeps getting bubbled up rather than silently ending a for-loop. OTOH StopIteration raised by a non-generator function should not be mutated. I'm sorry if this invalidates Nick's endorsement of the proposal. I definitely see this as a serious backward incompatibility: no matter how often it leads to buggy or obfuscated code, it's how things have always worked. Regarding Akira Li's examples of groupby(), unfortunately I find both versions inscrutable -- on a casual inspection I have no idea what goes on. I would have to think about how I would write groupby() myself (but I'm pretty sure I wouldn't us functools.partial(). :-) -- --Guido van Rossum (python.org/~guido)

Just a very minor correction, list(group(itr) instead of list comp. ... On 11/03/2014 09:17 PM, Ron Adam wrote:

def group_by(iterable): """ Yield (key, (group))'s of like values from an iterator. """ itr = PeekIter(iterable)

def group(it): # Yield only like values. k = it.peek() while it.peek() == k: yield next(it)

while itr.ok(): yield itr.peek(), [x for x in group(itr)]

def group_by(iterable): """ Yield (key, (group))'s of like values from an iterator. """ def group(it): # Yield only like values. k = it.peek() while it.peek() == k: yield next(it) itr = PeekIter(iterable) while itr.ok(): yield itr.peek(), list(group(itr)) Cheers, Ron

On 4 November 2014 00:29, Guido van Rossum wrote:

Regarding Akira Li's examples of groupby(), unfortunately I find both versions inscrutable -- on a casual inspection I have no idea what goes on. I would have to think about how I would write groupby() myself (but I'm pretty sure I wouldn't us functools.partial(). :-)

It's worth noting that the functools.partial doesn't really do anything. Just changing next_value = partial(next, iter(iterable)) to iterator = iter(iterable) and "next_value()" to "next(iterator)" gets rid of it. I would have tackled it by letting the inner iterator set a "finished" flag and I would have exhausted the iterable by iterating over it: def groupby(iterable): # Make sure this is one-pass iterator = iter(iterable) finished = False # Yields a group def yield_group(): nonlocal finished, group_key # This was taken off the iterator # by the previous group yield group_key for item in iterator: if item != group_key: # Set up next group group_key = item return yield item # No more items in iterator finished = True # This is always the head of the next # or current group group_key = next(iterator) while not finished: group = yield_group() yield group_key, group # Make sure the iterator is exhausted for _ in group: pass # group_key will now be the head of the next group This does have a key difference. Whereas with groupby you have the confusing property that from itertools import groupby grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> ['|', '|', '|'] with mine this does not happen: grps = groupby("|||---|||") grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> [] Was this an oversight in the original design or is this actually desired? I would guess it's an oversight.

Firstly, sorry for my buggy code that began all this mess. But what about adding a named parameter to the next function, specifying an exception to be raised on StopIteration, allowing it to propagate? ב-4 בנוב 2014, בשעה 06:49, Guido van Rossum כתב/ה:

I guess implementing groupby() would be a good interview question. :-)

Can we get back to the issue at hand, which is whether and how we can change the behavior of StopIteraton to be less error-prone?

On Mon, Nov 3, 2014 at 8:10 PM, Joshua Landau wrote:

...

On 4 November 2014 00:29, Guido van Rossum wrote:

...

Regarding Akira Li's examples of groupby(), unfortunately I find both versions inscrutable -- on a casual inspection I have no idea what goes on. I would have to think about how I would write groupby() myself (but I'm pretty sure I wouldn't us functools.partial(). :-)

It's worth noting that the functools.partial doesn't really do anything. Just changing

next_value = partial(next, iter(iterable))

to

iterator = iter(iterable)

and "next_value()" to "next(iterator)" gets rid of it.

I would have tackled it by letting the inner iterator set a "finished" flag and I would have exhausted the iterable by iterating over it:

def groupby(iterable): # Make sure this is one-pass iterator = iter(iterable) finished = False

# Yields a group def yield_group(): nonlocal finished, group_key

# This was taken off the iterator # by the previous group yield group_key

for item in iterator: if item != group_key: # Set up next group group_key = item return

yield item

# No more items in iterator finished = True

# This is always the head of the next # or current group group_key = next(iterator)

while not finished: group = yield_group() yield group_key, group

# Make sure the iterator is exhausted for _ in group: pass

# group_key will now be the head of the next group

This does have a key difference. Whereas with groupby you have the confusing property that

from itertools import groupby

grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> ['|', '|', '|']

with mine this does not happen:

grps = groupby("|||---|||")

grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> []

Was this an oversight in the original design or is this actually desired? I would guess it's an oversight. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 5/11/2014 1:55 p.m., Terry Reedy wrote:

There is already a 'default' option, so the two would have to be mutually exclusive.

"next(iterator[, default])

If it were a keyword-only argument, there wouldn't be any conflict. But I'm not sure I like the idea of adding more complexities to the iterator protocol. Anything that changes the signature of __next__ or __iter__ would be a bad idea, since all existing iterables would need to be updated to take it into account. If this is is to be done, it might be better to add a new optional dunder method, so that existing iterables would continue to work. -- Greg

On 4 Nov 2014 10:31, "Guido van Rossum" wrote:

...

I have to apologize, I pretty much had this backwards.

What I should have said is that a generator should always be terminated

by a return or falling off the end, and if StopIteration is raised in the generator (either by an explicit raise or raised by something it calls) and not caught by an except clause in the same generator, it should be turned into something else, so it bubbles out as something that keeps getting bubbled up rather than silently ending a for-loop. OTOH StopIteration raised by a non-generator function should not be mutated.

...

I'm sorry if this invalidates Nick's endorsement of the proposal.

I actually thought this was what you meant originally, and while it *would* require changes to contextlib, they're fairly minor: the current check for StopIteration would change to catch "UnexpectedStopIteration" instead, and the exception identity check would look at __cause__ rather

On 6 Nov 2014 00:09, "Nick Coghlan" wrote: than directly at the caught exception.

...

I definitely see this as a serious backward incompatibility: no matter

how often it leads to buggy or obfuscated code, it's how things have always worked.

Aye, breaking the equivalence between "return" and "raise StopIteration"

is pretty major.

I'm not even sure a plausible transition plan is possible, as at least

contextlib would trigger any warning we might issue. And having said that... what if we introduced UnexpectedStopIteration but initially made it a subclass of StopIteration? We could issue a deprecation warning whenever we triggered the StopIteration -> UnexpectedStopIteration conversion, pointing out that at some point in the future (3.6? 3.7?), UnexpectedStopIteration will no longer be a subclass of StopIteration (perhaps becoming a subclass of RuntimeError instead?). contextlib could avoid the warning by preconstructing a suitable UnexpectedStopIteration instance and throwing *that* into the generator, rather than throwing in a StopIteration raised from the with statement body. Regards, Nick.

Regards, Nick.

On 11/05/2014 03:47 PM, Nick Coghlan wrote:

...

...

I'm sorry if this invalidates Nick's endorsement of the proposal.

I actually thought this was what you meant originally, and while it *would* require changes to contextlib, they're fairly minor: the current check for StopIteration would change to catch "UnexpectedStopIteration" instead, and the exception identity check would look at __cause__ rather than directly at the caught exception.

...

I definitely see this as a serious backward incompatibility: no matter how often it leads to buggy or obfuscated code, it's how things have always worked.

Aye, breaking the equivalence between "return" and "raise StopIteration" is pretty major.

I'm not even sure a plausible transition plan is possible, as at least contextlib would trigger any warning we might issue.

And having said that... what if we introduced UnexpectedStopIteration but initially made it a subclass of StopIteration?

We could issue a deprecation warning whenever we triggered the StopIteration -> UnexpectedStopIteration conversion, pointing out that at some point in the future (3.6? 3.7?), UnexpectedStopIteration will no longer be a subclass of StopIteration (perhaps becoming a subclass of RuntimeError instead?).

contextlib could avoid the warning by preconstructing a suitable UnexpectedStopIteration instance and throwing *that* into the generator, rather than throwing in a StopIteration raised from the with statement body.

I really don't like the name Unexpected anything. It makes me think of blue screens and page faults. :-/ I'm also not sure how it's supposed to work or where it should come from. As near as I can tell these two examples below are equivalent. I think the thing that needs to be avoided is the case of the endless loop. It would be better to let the Exception be noisy. How would the second example be changed in order to do that? Or is there some other thing that needs to be fixed? Cheers, Ron def izip(*args): iters = [iter(obj) for obj in args] while True: yield list(next(it) for it in iters) #StopIteration suppressed #by list comprehension. #resulting in empty lists. #While Loop never exits. print("never printed") a = izip([1,2],[3,4]) print(next(a),next(a),next(a)) # (1, 3) (2, 4) () #list(izip([1,2],[3,4])) #Currently never returns def izip2(*args): iters = [iter(obj) for obj in args] while True: L = [] for it in iters: try: obj = next(it) # StopIteration suppressed here. except StopIteration: break # exit for-loop # "return" instead of "break"? L.append(obj) yield L # While Loop never exits. print("never printed") a = izip2([5,6],[7,8]) print(next(a),next(a),next(a)) # (5, 7) (6, 8) () list(izip2([5,6],[7,8])) #Currently never returns # Not only doesn't it exit, but it's building # an endless list of empty lists!

On 11/06/2014 04:15 AM, Steven D'Aprano wrote:

The current situation is simple to learn and understand:

(1) Generator expressions*emit* StopIteration when they are done:

py> next(iter([])) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

It's the "when they are done" part that's having the issue. In some cases, they are never completely done because the StopIteration Error is handled incorrectly. The reason this doesn't show up more is that most iterators only have one loop, which is exited and the generator ends causing a different StopIteration to be emitted from the one that ends the loop.

(2) Functions such as tuple, list, set, dict*absorb* StopIteration:

py> list(iter([])) [] py> it = iter([]) py> list(next(it) for y in range(1000)) []

Right, sometimes this doesn't happen.

For-loops do the same, if StopIteration is raised in the "for x in iterable" header. That's how it knows the loop is done. The "for" part of a comprehension is the same.

I think this part is working as it should.

(3) But raising StopIteration in the expression part (or if part) of a comprehension does not absord the exception, it is treated like any other exception:

py> [next(iter([])) for y in range(1000)] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <listcomp> StopIteration

This is not always working. When a StopIteration is raised from the next call in the example that started the thread, it's getting replaced with the equivalent of a break, so it's never exiting the generator completely. I think it needs to be replaced with the equivalent of return, which will end the generator, and cause a StopIteration to be emitted. (As you describe here.) It looks like it's a bug in the C code for generator expressions. Since it only effects generator expressions that have iterators with nested loops, it may be fixible. Cheers, Ron

I believe the issue is that, in some situations, the generator is absorbing StopIteration when it shouldn't. -- ~Ethan~

On Thu, 6 Nov 2014 10:54:51 -0800 Guido van Rossum wrote:

If I had had the right foresight, I would have made it an error to terminate a generator with a StopIteration, probably by raising another exception chained to the StopIteration (so the traceback shows the place where the StopIteration escaped).

The question at hand is if we can fix this post-hoc, using clever tricks and (of course) a deprecation period.

Is there any point in fixing it? Who relies on such borderline cases? Regards Antoine.

On Thu, Nov 06, 2014 at 08:01:32PM -0430, Juancarlo Añez wrote:

On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou wrote:

...

Is there any point in fixing it? Who relies on such borderline cases?

import this?

What about it? There are 19 lines in the Zen, and I don't think any of them are particularly relevent here. Which line, or lines, were you thinking of? -- Steven

On Fri, Nov 7, 2014 at 11:57 PM, Juancarlo Añez wrote:

It is not in the Zen of Python to leave it as is just because some existing code may be relying on the odd behavior.

Actually, it is. Practicality beats purity. :) ChrisA

On 7 November 2014 07:45, Antoine Pitrou wrote:

On Thu, 6 Nov 2014 10:54:51 -0800 Guido van Rossum wrote:

...

If I had had the right foresight, I would have made it an error to terminate a generator with a StopIteration, probably by raising another exception chained to the StopIteration (so the traceback shows the place where the StopIteration escaped).

The question at hand is if we can fix this post-hoc, using clever tricks and (of course) a deprecation period.

Is there any point in fixing it? Who relies on such borderline cases?

It's not about people relying on the current behaviour (it can't be, since we're talking about *changing* that behaviour), it's about "Errors should never pass silently". That is, the problematic cases that (at least arguably) may be worth fixing are those where: 1. StopIteration escapes from an expression (Error!) 2. Instead of causing a traceback, it terminates a containing generator (Passing silently!) As asyncio coroutines become more popular, I predict some serious head scratching from StopIteration escaping an asynchronous operation and getting thrown into a coroutine, which then terminates with a "return None" rather than propagating the exception as you might otherwise expect. The problem with this particular style of bug is that the only trace it leaves is a generator iterator that terminates earlier than expected - there's no traceback, log message, or any other indication of where something strange may be happening. Consider the following, from the original post in the thread: def izip(*args): iters = [iter(obj) for obj in args] while True: yield tuple([next(it) for it in iters]) The current behaviour of that construct is that, as soon as one of the iterators is empty: 1. next(it) throws StopIteration 2. the list comprehension unwinds the frame, and allows the exception to propagate 3. the generator iterator unwinds the frame, and allows the exception to propagate 4. the code invoking the iterator sees StopIteration and assumes iteration is complete If you switch to the generator expression version instead, the flow control becomes: 1. next(it) throws StopIteration 2. the generator expression unwinds the frame, and allows the exception to propagate 3. the iteration inside the tuple constructor sees StopIteration and halts 4. the generator iterator never terminates In that code, "next(it)" is a flow control operation akin to break (it terminates the nearest enclosing generator iterator, just as break terminates the nearest enclosing loop), but it's incredibly unclear that this is the case - there's no local indication that it may raise StopIteration, you need to "just know" that raising StopIteration is a possibility. Guido's suggestion is to consider looking for a viable way to break the equivalence between "return" and "raise StopIteration" in generator iterators - that way, the only way for the above code to work would be through a more explicit version that clearly tracks the flow control. Option 1 would be to assume we use a new exception, and are OK with folks catching it explicitly from __future__ import explicit_generator_return def izip(*args): iters = [iter(obj) for obj in args] while True: try: t = tuple(next(it) for it in iters) except UncaughtStopIteration: return # One of the iterators has been exhausted yield t Option 2 would be to assume the new exception is something generic like RuntimeError, requiring the inner loop to be converted to statement form: def izip(*args): iters = [iter(obj) for obj in args] while True: entry = [] for it in iters: try: item = next(it) except StopIteration: return # One of the iterators has been exhausted entry.append(item) yield tuple(entry) With option 2, you can also still rely on the fact that list comprehensions don't create a generator frame: def izip(*args): iters = [iter(obj) for obj in args] while True: try: entry = [next(it) for it in iters] except StopIteration: return # One of the iterators has been exhausted yield tuple(entry) The upside of the option 2 spellings is they'll work on all currently supported versions of Python, while the downside is the extra object construction they have to do if you want to yield something other than a list. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Ron Adam wrote:

Can you point me to the source file that implements generator expression byte code or C code? I wanted to look at that to see what was actually going on, but it looks like it may be a combination of a regular generator with a condition at some point to handle it slightly different.

There's not much difference between a generator expression and an ordinary generator. A generator expression gets compiled into a call of an anonymous generator function.

...

...

from dis import dis def f(): ... return list(x for x in (1, 2, 3)) ... 2 0 LOAD_GLOBAL 0 (list) 3 LOAD_CONST 1 () 6 MAKE_FUNCTION 0 9 LOAD_CONST 5 ((1, 2, 3)) 12 GET_ITER 13 CALL_FUNCTION 1 16 CALL_FUNCTION 1 19 RETURN_VALUE dis(f.__code__.co_consts[1]) 2 0 LOAD_FAST 0 (.0) >> 3 FOR_ITER 11 (to 17) 6 STORE_FAST 1 (x) 9 LOAD_FAST 1 (x) 12 YIELD_VALUE 13 POP_TOP 14 JUMP_ABSOLUTE 3 >> 17 LOAD_CONST 0 (None) 20 RETURN_VALUE

-- Greg