[Python-ideas] itertools.chunks()

[Wolfgang Maier]

On Tue, 9 Apr 2013 20:19:50 +0100
 Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>
> Thanks. I also ran your code using different conditions
> like 100000
> elements in chunks of 1024 because that's the sort of
> situation I was
> interested in. strict_grouper is faster for the bulk of
> the iteration
> but is not very fast at the end when the chunk size is
> large and the
> last chunk has fill values.
>
> 
> This code is the cause of the slow end performance:
> 
> >             while prev[-1] is fillvalue:
> >                 prev = prev[:-1]
> 
> I think where I was assuming large chunk sizes, Peter was
> assuming
> small chunk sizes as this is quadratic in the chunk size.

Yes, and this is also my major use case. It's funny how
apparently different problems lead people to similar
questions. My most important use of strict_grouper now is
for GB-size files with logical units composed of only a
handful of lines, and it's doing a great job there.
 
> If you
> change these lines to
> 
>     n = len(prev)-1
>     while prev[n] is fillvalue:
>         n -= 1
>     del prev[n+1:]

almost, but strict_grouper returns tuples, so you can't use
del here, instead:

      prev=prev[:n+1]

>     yield prev
> 
> then it will probably be as fast or faster than the one I
> posted in
> pretty much all cases.
>
Right, I just spotted this only weakness in Peter's code
when I ran the comparisons to your code, and figured out
the almost identical solution above, which works for
tuples.

Wolfgang