What is the "classic" use case for next() raising StopIteration, to be silently caught ? We need __next__ to do so in for loops, but when do we need it in the functional form? Elazar 2014-02-21 11:25 GMT+02:00 Peter Otten <__peter__@web.de>:
Oscar Benjamin wrote:
On 20 February 2014 22:38, אלעזר <elazarg@gmail.com> wrote:
2014-02-21 0:00 GMT+02:00 Steven D'Aprano <steve@pearwood.info>:
On Thu, Feb 20, 2014 at 04:14:17PM +0000, Oscar Benjamin wrote:
The thing is just that bare next is not something that's widely recognised as being dangerous. I've seen examples of this kind of bug in samples from many Python aficionados (including at least one from you Terry).
Why is it dangerous, and what kind of bug?
If you're talking about the fact that next(it) can raise StopIteration, I think you are exaggerating the danger. Firstly, quite often you don't mind if it raises StopIteration, since that's what you would have done anyway. Secondly, I don't see why raising StopIteration is so much more dangerous than (say) IndexError or KeyError.
I had this bug just the other day. I did not plan for the empty case, since it was obvious that the empty case is a bug, so I relied on the exception being raised in this case. But I did not get the exception since it was caught in a completely unrelated for loop. It took me a while to figure out what's going on, and it would've taken even more for someone else, not familiar with my assumption or with the whole StopIteration thing (which I believe is the common case). An IndexError or a KeyError would have been great in such a case.
Exactly. The bug I had manifested in a StopIteration that was raised in a semi-deterministic (dependent on slowly varying data) fashion after ~1 hour of processing. Had it resulted in an IndexError I would have seen a traceback and could have fixed the bug within about 5 minutes.
But StopIteration doesn't necessarily bubble up in the same way as other exceptions because it can be caught and silently supressed by a for loop (or any consumer of iterators). So instead of a traceback I had truncated output data. It took some time to discover and verify that the output data was truncated. Then it took some time rerunning the script under pdb which was no help since it couldn't latch into the suppressed exception. I assumed that it must be an exception but there were no try/except clauses anywhere in the code. Eventually I found it by peppering the code with:
try: ... except Exception as e: import pdb; pdb.set_trace()
It took most of a day for me to track that down instead of 5 minutes precisely because StopIteration is not like other exceptions. Admittedly I'd spot a similar bug much quicker now simply because I'm aware of the possibility.
A simplified version of the bug is shown below:
def itermerge(datasources): for source in datasources: iterator = iter(source) first = next(iterator) for item in iterator: yield first * item
data = [ [1, 1, 2, 3], [1, 4, 5, 6], [], # Who put that there? [1, 7, 8, 9], ]
for item in itermerge(data): print(item)
If you run the above then you get:
$ python tmp.py 1 2 3 4 5 6
So the data is silently truncated at the empty iterable.
It is *very* similar to the "and or" story.
I think the difference is that once you've learned the lesson you stop using `and...or` while you change your usage pattern for next() to minimal scopes
def process_source(source): it = iter(source) first = next(it) for item in it: yield first * item
def itermerge(sources): for source in sources: yield from process_source(source)
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/