Consume an iterable

[Peter Otten]

Muhammad Alkarouri wrote:

> Thanks everyone, but not on my machine (Python 2.6.1, OS X 10.6) it's
> not:
> 
> 
> In [1]: from itertools import count, islice
> 
> In [2]: from collections import deque
> 
> In [3]: i1=count()
> 
> In [4]: def consume1(iterator, n):
>    ...:     deque(islice(iterator, n), maxlen=0)
>    ...:
>    ...:
> 
> In [5]: i2=count()
> 
> In [6]: def consume2(iterator, n):
>    ...:     for _ in islice(iterator, n): pass
>    ...:
>    ...:
> 
> In [7]: timeit consume1(i1, 10)
> 1000000 loops, best of 3: 1.63 us per loop
> 
> In [8]: timeit consume2(i2, 10)
> 1000000 loops, best of 3: 846 ns per loop
> 
> 
> Can somebody please test whether it is only my machine or is this
> reproducible?

I can reproduce it. The deque-based approach has a bigger constant overhead 
but better per-item performance. Its asymptotical behaviour is therefore 
better.

$ python consume_timeit.py
consume_deque
    10: 1.77500414848
   100: 3.73333001137
  1000: 24.7235469818

consume_forloop
    10: 1.22008490562
   100: 5.86271500587
  1000: 52.2449371815

consume_islice
    10: 0.897439956665
   100: 1.51542806625
  1000: 7.70061397552

$ cat consume_timeit.py
from collections import deque
from itertools import islice, repeat

def consume_deque(n, items):
    deque(islice(items, n), maxlen=0)

def consume_forloop(n, items):
    for _ in islice(items, n):
        pass

def consume_islice(n, items):
    next(islice(items, n-1, None), None)

def check(fs):
    for consume in fs:
        items = iter(range(10))
        consume(3, items)
        rest = list(items)
        assert rest == range(3, 10), consume.__name__

if __name__ == "__main__":
    fs = consume_deque, consume_forloop, consume_islice
    check(fs)


    items = repeat(None)

    from timeit import Timer
    for consume in fs:
        print consume.__name__
        for n in (10, 100, 1000):
            print "%6d:" % n,
            print Timer("consume(%s, items)" % n,
                        "from __main__ import consume, items").timeit()
        print
$

With next(islice(...), None) I seem to have found a variant that beats both  
competitors.

Peter