[Python-ideas] Buffering iterators?

[Michael Selik]

On Sat, Dec 19, 2015 at 7:02 AM Michael Mitchell <epsilonmichael at gmail.com>
wrote:

> Have you considered doing this at the plain Python level? Something such
> as the following would have the desired semantics from my understanding.
>
> def buffered_iterator(it, size):
>   while True:
>     buffer = [next(it) for _ in range(size)]
>     for element in buffer:
>       yield element
>

There's a recipe in the itertools module for something like this (
https://docs.python.org/3.6/library/itertools.html#itertools-recipes).
Check out ``def grouper``.

A combination of starmap, repeat, and islice might work fine as well.
    args = (iterable, buffersize)
    chunks = starmap(islice, repeat(args))

Either way, you could then yield from the chunks to make it appear like a
regular iterator.

Not being a PyPy or Pyston expert, I have no clue if this scenario exists
-- that a JIT compiling interpreter would not be able to prefetch chunks of
the iterator without the extra buffering layer, but would be able to
prefetch after the chunking step is added.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151222/31f822bb/attachment-0001.html>