<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 7 November 2014 07:45, Antoine Pitrou <span dir="ltr"><<a href="mailto:solipsis@pitrou.net" target="_blank">solipsis@pitrou.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, 6 Nov 2014 10:54:51 -0800<br>
<span class="">Guido van Rossum <<a href="mailto:guido@python.org">guido@python.org</a>> wrote:<br>
><br>
</span><span class="">> If I had had the right foresight, I would have made it an error to<br>
> terminate a generator with a StopIteration, probably by raising another<br>
> exception chained to the StopIteration (so the traceback shows the place<br>
> where the StopIteration escaped).<br>
><br>
> The question at hand is if we can fix this post-hoc, using clever tricks<br>
> and (of course) a deprecation period.<br>
<br>
</span>Is there any point in fixing it? Who relies on such borderline cases?<br></blockquote><div><br></div><div>It's not about people relying on the current behaviour (it can't be, since we're talking about *changing* that behaviour), it's about "Errors should never pass silently". That is, the problematic cases that (at least arguably) may be worth fixing are those where:<br><br></div><div>1. StopIteration escapes from an expression (Error!)<br></div><div>2. Instead of causing a traceback, it terminates a containing generator (Passing silently!)<br></div><div><br></div><div>As asyncio coroutines become more popular, I predict some serious head scratching from StopIteration escaping an asynchronous operation and getting thrown into a coroutine, which then terminates with a "return None" rather than propagating the exception as you might otherwise expect.<br></div><div><br></div><div>The problem with this particular style of bug is that the only trace it leaves is a generator iterator that terminates earlier than expected - there's no traceback, log message, or any other indication of where something strange may be happening.<br><br></div><div>Consider the following, from the original post in the thread:<br></div><div><br><div dir="ltr"> def izip(*args):</div><div dir="ltr"> iters = [iter(obj) for obj in args]</div><div dir="ltr"> while True:</div><div dir="ltr"> yield tuple([next(it) for it in iters])</div><div dir="ltr"><br></div><div>The current behaviour of that construct is that, as soon as one of the iterators is empty:<br><br></div><div>1. next(it) throws StopIteration<br></div><div>2. the list comprehension unwinds the frame, and allows the exception to propagate</div><div>3. the generator iterator unwinds the frame, and allows the exception to propagate<br></div><div>4. the code invoking the iterator sees StopIteration and assumes iteration is complete<br></div><div><br></div><div>If you switch to the generator expression version instead, the flow control becomes:<br><br><div>1. next(it) throws StopIteration<br></div>2. the generator expression unwinds the frame, and allows the exception to propagate<br></div><div>3. the iteration inside the tuple constructor sees StopIteration and halts<br></div><div>4. the generator iterator never terminates<br></div><div><br></div><div>In that code, "next(it)" is a flow control operation akin to break (it terminates the nearest enclosing generator iterator, just as break terminates the nearest enclosing loop), but it's incredibly unclear that this is the case - there's no local indication that it may raise StopIteration, you need to "just know" that raising StopIteration is a possibility.<br></div><br></div><div>Guido's suggestion is to consider looking for a viable way to break the equivalence between "return" and "raise StopIteration" in generator iterators - that way, the only way for the above code to work would be through a more explicit version that clearly tracks the flow control.<br><br></div><div>Option 1 would be to assume we use a new exception, and are OK with folks catching it explicitly<br></div><div><br></div><div> from __future__ import explicit_generator_return<br></div><div><div dir="ltr"> def izip(*args):</div><div dir="ltr"> iters = [iter(obj) for obj in args]</div><div dir="ltr"> while True:<br></div><div> try:<br></div><div dir="ltr"> t = tuple(next(it) for it in iters)<br></div><div> except UncaughtStopIteration:<br></div><div> return # One of the iterators has been exhausted<br></div><div> yield t<br></div><div dir="ltr"><br></div><div>Option 2 would be to assume the new exception is something generic like RuntimeError, requiring the inner loop to be converted to statement form:<br><br><div dir="ltr"><br> def izip(*args):</div><div dir="ltr"> iters = [iter(obj) for obj in args]</div><div dir="ltr"> while True:<br></div><div> entry = []<br></div><div> for it in iters: <br> try:<br></div><div dir="ltr"> item = next(it)<br></div><div> except StopIteration:<br></div><div> return # One of the iterators has been exhausted<br></div><div> entry.append(item)<br></div><div> yield tuple(entry)<br></div><div dir="ltr"><br></div><div>With option 2, you can also still rely on the fact that list comprehensions don't create a generator frame:<br><br> def izip(*args):<div dir="ltr"> iters = [iter(obj) for obj in args]</div><div dir="ltr"> while True:<br></div><div> try:<br></div><div dir="ltr"> entry = [next(it) for it in iters]<br></div><div> except StopIteration:<br></div><div> return # One of the iterators has been exhausted<br></div><div> yield tuple(entry)<br></div><div dir="ltr"><br></div>The upside of the option 2 spellings is they'll work on all currently supported versions of Python, while the downside is the extra object construction they have to do if you want to yield something other than a list.<br></div><br></div><div>Cheers,<br>Nick.<br></div><br></div></div>-- <br><div class="gmail_signature">Nick Coghlan | <a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a> | Brisbane, Australia</div>
</div></div>