On Sat, Dec 19, 2015 at 7:02 AM Michael Mitchell <epsilonmichael@gmail.com> wrote:

Have you considered doing this at the plain Python level? Something such as the following would have the desired semantics from my understanding.

def buffered_iterator(it, size):
while True:
buffer = [next(it) for _ in range(size)]
for element in buffer:
yield element

There's a recipe in the itertools module for something like this (https://docs.python.org/3.6/library/itertools.html#itertools-recipes). Check out ``def grouper``.

A combination of starmap, repeat, and islice might work fine as well.

args = (iterable, buffersize)

chunks = starmap(islice, repeat(args))

Either way, you could then yield from the chunks to make it appear like a regular iterator.

Not being a PyPy or Pyston expert, I have no clue if this scenario exists -- that a JIT compiling interpreter would not be able to prefetch chunks of the iterator without the extra buffering layer, but would be able to prefetch after the chunking step is added.