On Fri, Dec 12, 2014 at 10:10 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

> On Dec 12, 2014, at 4:36 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
>
> Terry Reedy wrote:
>> PEP 479 reverses this acceptance by having generator.__next turn StopIteration raised in a user-written generator function body into a RuntimeError.  I propose that other builtin iterator.__next__ methods that execute a passed in function do the same.
>
> Maybe also any user-written function whose
> name isn't "__next__"?

That does seem to be where this is headed :-(

I'm -1 on this proposal.

Me too. PEP 479 was a clear win. __next__ methods are a much murkier area and we should not mess with them.
 
Generators are part of the language internals and language spec, so Guido can reasonably decide to take this in whatever direction he wants.   In contrast, now you're moving on to parts of the language library that merely call functions and return results.

It is not the responsibility of accumulate(), filter(), map(), or any other higher-order functions to impose rules about what those functions are allowed to do  -- they can take as much time as they want, they can hold the GIL, they can manipulate signals, they can use tons of memory, they can return scalar values or complex objects, and they can raise any exception, but now StopIteration would become an event that gets special treatment.

Further, this new made-up rule (for which there is zero demonstrated need in any language I know) would have to be applied throughout the standard library and possibly be extended to third-party code. It would be yet another language idiosyncrasy that would have to be learned, remembered, and StackOverflowed about for an eternity.

In the case of generators, the PEP 479 rule is easily applied (not hard to search for or to mitigate).  In contrast, the "fix" in this case would need to be applied to the *called* functions or their callees, possibly far removed from the map/filter/accumulate call.  If that function is in third-party code or in a C-library, then the mitigation would require redesigning the call logic completely (not fun) or to wrap the function in something transforms a StopIteration into a custom exception and then re-catches the custom exception upstream from the higher-order function (also not fun).

For anyone who has tested code that is currently working correctly but becomes broken by this proposal, the simplest mitigation will be for them to write their own variants of map/filter/accumulate/dropwhile/takewhile/groupby/starmap that just ignore this proposal and restore the behavior that has been happily in place for ages.

When it comes to map() and filter() and friends the situation is murkier yet. In Python 2, raising StopIteration in the function passed to map/filter was just bubbled out. In Python 3, where these have become lazy, a StopIteration raised by the function terminates the iteration. Was that intentional? I doubt it. Is it now a feature? Maybe (though it would make more sense to use something like takewhile). Could it mask a subtle bug? Probably (but it doesn't sound like a common situation -- map/filter functions are usually small and simple). Should we "fix" it? I don't think so. It's not sufficiently broken, the fix would lead us onto a slippery slope. Enough is enough.
 
Raymond
 
--
--Guido van Rossum (python.org/~guido)