On Thu, Oct 14, 2021 at 8:04 AM Oscar Benjamin oscar.j.benjamin@gmail.com wrote:
On Wed, 13 Oct 2021 at 18:30, Chris Angelico rosuav@gmail.com wrote:
On Thu, Oct 14, 2021 at 1:36 AM Oscar Benjamin oscar.j.benjamin@gmail.com wrote:
Your suggestion is that this is a bug in map() which is a fair alternative view. Following through to its conclusion your suggestion is that every possible function like map, filter, and all the iterator implementations in itertools and in the wild should carefully wrap any internal non-next function call in try/except to change any potential StopIteration into a different exception type.
Yes, because it is the map function that is leaking StopIteration.
But it is not the map function that *raises* StopIteration. The exception "leaks" through map just like *all* exceptions "leak" through *all* Python functions in the absence of try/except. This is not normally referred to as "leaking" but rather as "propagating" and it is precisely the design of exceptions that they should propagate to the calling frame. The difference in the case of StopIteration is that it can be caught even if there is no try/except.
Wrong. You still won't catch StopIteration unless it is in one very specific place: a __next__ function. Exactly the same as AttributeError can be silently caught inside a __getattr__ function. See my earlier post for full context, but the problem is right here:
def __next__(self): return self.func(next(self.it))
The problem isn't the transformation function; the problem is that __next__ is putting two completely different concepts (pumping the iterator, and calling the transformation function) inside, effectively, the same exception handling context.
My view is that it would be better to have a basic primitive for getting an element from an iterable or for advancing an iterator that does not raise StopIteration in the first place. I would probably call that function something like "take" rather than "first" though. The reason I prefer introducing an alternative to next() is because I think that if both primitives were available then in the majority of situations next() would not be the preferred option.
How will that solve anything though? You still need a way to advance an iterator and get a value from it, or get told that there is no such value. No matter what exception you choose, it will ALWAYS be possible for the same problem to occur. Exceptions like ValueError will, instead of early-aborting a map(), cause something to mistakenly think that it couldn't parse a number, or something like that.
I find it surreal that I am arguing that StopIteration is a uniquely problematic exception and that you seem to be arguing that it is not. Yet at the same time you are an author of a (successful!) PEP that was *entirely* about this very subject: https://www.python.org/dev/peps/pep-0479/
I find it surreal that people keep holding up PEP 479, disagreeing with the document's wording, and assuming that I believe the altered wording. I don't.
The first two paragraphs of the rationale from the PEP: """ The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.
The main goal of the proposal is to ease debugging in the situation where an unguarded next() call (perhaps several stack frames deep) raises StopIteration and causes the iteration controlled by the generator to terminate silently. (Whereas, when some other exception is raised, a traceback is printed pinpointing the cause of the problem.) """ I agree entirely with the above but every occurence of "generators" should have been generalised to "iterators" in order to address the problem fully.
No! Generators *ARE* special, because they don't have that same concept. You can write map safely like this:
def map(func, iterable): for value in iterable: yield func(value)
Since there's no call to next(), there's no expectation of StopIteration. Since there's no __next__ function being defined, there's no expectation that StopIteration has meaning. That's why generators are different.
You think this should be fixed in map. I think that the root of the problem is next. The PEP discusses changing next: https://www.python.org/dev/peps/pep-0479/#converting-the-exception-inside-ne... The idea was rejected on backward compatibility grounds: I am proposing that an alternative function could be added which unlike changing next would not cause compatibility problems.
Although we may disagree about what is the best way to fix this I don't see how we can disagree that StopIteration is a uniquely problematic exception to raise (as you seem to argue above).
Where do I argue that StopIteration is unique? It was unique pre-479 only in the odd interaction with generators. It is now exactly like every other exception that is used in a protocol: a signal that there is no value to be returned.
In fact, NotImplemented is more special - it would be more consistent to have __add__ raise a special exception than to return a magical value. StopIteration, AttributeError, LookupError, etc, are all used the same way.
ChrisA