[Python-ideas] Add lookahead iterator (peeker) to itertools
Masklinn
masklinn at masklinn.net
Tue Feb 26 11:57:45 CET 2013
On 2013-02-26, at 11:37 , Serhiy Storchaka wrote:
> On 26.02.13 12:06, Masklinn wrote:
>> Although I can trivially get a segfault with islice:
>>
>> from itertools import islice, count
>>
>> it = count()
>> while True:
>> it = islice(it, 0)
>>
>> Run this, C-c at some point, every CPython version on my machine
>> segfaults (pypy does not)
>
> Thank you. This looks as pretty same issue as with tee(). I knew that there must be more such catches, but could not find them.
>
> http://bugs.python.org/issue17300
>
Cool.
And I looked back in the VCS, turns out the code hasn't been lost but
the issue was not in the stdlib, it was a custom iterator (used as a
wrapper for a bunch of operations) which needed to be reapplied very
often (code basically went Iterator -> mix of tee, chain and
dropwhile -> Iterator -> same mix), essentially doing the following:
import itertools
_placeholder = object()
class It(object):
def __init__(self, stream):
self.stream = iter(stream)
self.stopped = False
def __iter__(self):
return self
def __next__(self):
if self.stopped: raise StopIteration()
val = next(self.stream, _placeholder)
if val is _placeholder:
self.stopped = True
raise StopIteration()
return val
it = itertools.count()
while True:
it = It(it)
next(it)
I'm not sure if there's any way to implement such a wrapping iterator in
a way which does not ultimately blow the stack (save in C taking after
itertools implementations I guess as they don't seem to have the issue)
More information about the Python-ideas
mailing list