On Wed, Dec 23, 2015 at 4:01 AM, Nicolas P. Rougier <Nicolas.Rougier@inria.fr> wrote:

But my implementation is quite slow, especially when you add one item at a time:

>>> python benchmark.py
Python list, append 100000 items: 0.01161
Array list, append 100000 items: 0.46854

DEFAULT_BUFFER_SIZE = 128 # original buffer created.
BUFFER_EXTEND_SIZE = 1.25 # array.array uses 1+1/16 -- that seems small to me.In [11]: run profile_accumulator.py


In [12]: timeit accum1(10000)

100 loops, best of 3: 3.91 ms per loop

In [13]: timeit list1(10000)

1000 loops, best of 3: 1.15 ms per loop

These are simply appending 10,000 integers in a loop -- with teh list, the list is turned into a numpy array at the end. So it's still faster to accumulate in a list, then make an array, but only a about a factor of 3 -- I think this is because you are staring with a python integer -- with the accumulator function, you need to be checking type and pulling a native integer out with each append. but a list can append a python object with no type checking or anything.

Then the conversion from list to array is all in C.

Note that the accumulator version is still more memory efficient...

In [14]: timeit accum2(10000)

100 loops, best of 3: 3.84 ms per loop

this version pre-allocated the whole internal buffer -- not much faster the buffer re-allocation isn't a big deal (thanks to ndarray.resize using realloc(), and not creating a new numpy array)

In [24]: timeit list_extend1(100000)

100 loops, best of 3: 4.15 ms per loop

In [25]: timeit accum_extend1(100000)

1000 loops, best of 3: 1.37 ms per loop