[Python-ideas] Map and filter should also convert StopIteration to RuntimeError

[Steven D'Aprano]

On Fri, Dec 12, 2014 at 04:14:07PM -0500, Terry Reedy wrote:

> This proposal comes from Oscar Benjamin's comments on the PEP in the 
> 'Generator are iterators' thread.  In one post, he gave this example.
> 
> >>> def func(x):
> ...     if x < 0:
> ...         raise StopIteration
> ...     return x ** 2
> ...
> >>> it = map(func, [1, 2, 3, -1, 2])
> >>> list(it)
> [1, 4, 9]
> >>> list(it)  # map continues to yield values...
> [4]
> 
> Function func violates what I think should be the guideline.

The term used by the docs is that it is "broken", which in this context 
has a technical meaning. An iterator is broken if it fails the rule that 
once it raises StopIteration, it will continue to always raise 
StopIteration.

Whether or not map() and filter() return a "broken iterator" is 
completely at the mercy of the user's function. func() can silently 
break the iterator in many different ways:

- it may call os._exit() or os.abort()
- it may call threaded code which deadlocks
- it may call time.sleep(2**10000)
- it may enter an infinite loop

to mention just a few. The documentation's promise that map() will 
return an iterator that "applies function to every item of iterable" is 
not an unconditional promise. It cannot possibly be, and neither map() 
nor Python can detect every possible failure condition in advance.

It is not Python's responsibility to police that all iterators are 
non-broken. That is the responsibility of the coder. If you write func() 
such that it "breaks" map(), then you either have a good reason for 
doing so, or you are responsible for your own actions. We are all 
consenting adults here. Let's not complicate things in a futile attempt 
to prevent people from shooting themselves in the foot.

There's a thread on python-dev at the moment decrying the difficulty of 
writing polylingual Python 2 + 3 code and how some people find it sucks 
all the joy out of writing Python code. Every backwards incompatible 
change we add just makes it more difficult to deal with the 2/3 
transition. New features are an incentive to upgrade to 3.x. This is not 
an incentive to upgrade, but it will make it harder to write and 
understand polylingual iterator code. Gratuitously fixing perceived 
weaknesses in the iterator protocol which have been there since it was 
first introduced will, in my opinion, cause more pain than benefit.

In another thread, Nick wrote to Oscar:

    The problem you think PEP 479 is trying to solve *is* the
    one where the discussion started, but it is *not* the one 
    where it ended when Guido accepted PEP 479. The problems 
    PEP 479 solves are generator specific - they have nothing 
    to do with the iterator protocol in general.

Let's leave the iterator protocol alone.



-- 
Steven