On Sat, Dec 19, 2015 at 7:02 AM Michael Mitchell <epsilonmichael@gmail.com> wrote:
Have you considered doing this at the plain Python level? Something such as the following would have the desired semantics from my understanding.
def buffered_iterator(it, size): while True: buffer = [next(it) for _ in range(size)] for element in buffer: yield element
There's a recipe in the itertools module for something like this ( https://docs.python.org/3.6/library/itertools.html#itertools-recipes). Check out ``def grouper``. A combination of starmap, repeat, and islice might work fine as well. args = (iterable, buffersize) chunks = starmap(islice, repeat(args)) Either way, you could then yield from the chunks to make it appear like a regular iterator. Not being a PyPy or Pyston expert, I have no clue if this scenario exists -- that a JIT compiling interpreter would not be able to prefetch chunks of the iterator without the extra buffering layer, but would be able to prefetch after the chunking step is added.