<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Dec 30, 2015 at 6:34 AM, Nicolas P. Rougier <span dir="ltr"><<a href="mailto:Nicolas.Rougier@inria.fr" target="_blank">Nicolas.Rougier@inria.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On 28 Dec 2015, at 19:58, Chris Barker <<a href="mailto:chris.barker@noaa.gov">chris.barker@noaa.gov</a>> wrote:<br>
><br>> >>> python benchmark.py<br>
> Python list, append 100000 items: 0.01161<br>
> Array list, append 100000 items: 0.46854<br>
><br>
> are you pre-allocating any extra space? if not -- it's going to be really, really pokey when adding a little bit at a time.<br>
<br>
<br>
</span>Yes, I’m preallocating but it might not be optimal at all given your implementation is much faster.<br>
I’ll try to adapt your code. Thanks.</blockquote><div><br></div><div>sounds good -- I'll try to take a look at yours soon - maybe we can merge the projects. MIne is only operational in one small place, I think.</div><div><br></div><div>-CHB</div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
<br>
><br>
> With my Accumulator class:<br>
><br>
> <a href="https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py" rel="noreferrer" target="_blank">https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py</a><br>
><br>
> I pre-allocate a larger numpy array to start, and it gets re-allocated, with some extra, when filled, using ndarray.resize()<br>
><br>
> this is quite fast.<br>
><br>
> These are settable parameters in the class:<br>
><br>
> DEFAULT_BUFFER_SIZE = 128 # original buffer created.<br>
> BUFFER_EXTEND_SIZE = 1.25 # array.array uses 1+1/16 -- that seems small to me.<br>
><br>
><br>
> I looked at the code in array.array (and list, I think), and it does stuff to optimize very small arrays, which I figured wasn't the use-case here :-)<br>
><br>
> But I did a bunch of experimentation, and as long as you pre-allocate _some_ it doesn't make much difference how much :-)<br>
><br>
> BTW,<br>
><br>
> I just went in an updated and tested the Accumulator class code -- it needed some tweaks, but it's working now.<br>
><br>
> The cython version is in an unknown state...<br>
><br>
> some profiling:<br>
><br>
> In [11]: run profile_accumulator.py<br>
><br>
><br>
> In [12]: timeit accum1(10000)<br>
><br>
> 100 loops, best of 3: 3.91 ms per loop<br>
><br>
> In [13]: timeit list1(10000)<br>
><br>
> 1000 loops, best of 3: 1.15 ms per loop<br>
><br>
> These are simply appending 10,000 integers in a loop -- with teh list, the list is turned into a numpy array at the end. So it's still faster to accumulate in a list, then make an array, but only a about a factor of 3 -- I think this is because you are staring with a python integer -- with the accumulator function, you need to be checking type and pulling a native integer out with each append. but a list can append a python object with no type checking or anything.<br>
><br>
> Then the conversion from list to array is all in C.<br>
><br>
> Note that the accumulator version is still more memory efficient...<br>
><br>
> In [14]: timeit accum2(10000)<br>
><br>
> 100 loops, best of 3: 3.84 ms per loop<br>
><br>
> this version pre-allocated the whole internal buffer -- not much faster the buffer re-allocation isn't a big deal (thanks to ndarray.resize using realloc(), and not creating a new numpy array)<br>
><br>
> In [24]: timeit list_extend1(100000)<br>
><br>
> 100 loops, best of 3: 4.15 ms per loop<br>
><br>
> In [25]: timeit accum_extend1(100000)<br>
><br>
> 1000 loops, best of 3: 1.37 ms per loop<br>
><br>
> This time, the stuff is added in chunks 100 elements at a time -- the chunks being created ahead of time -- a list with range() the first time, and an array with arange() the second. much faster to extend with arrays...<br>
><br>
> -CHB<br>
><br>
><br>
><br>
> --<br>
><br>
> Christopher Barker, Ph.D.<br>
> Oceanographer<br>
><br>
> Emergency Response Division<br>
> NOAA/NOS/OR&R (206) 526-6959 voice<br>
> 7600 Sand Point Way NE (206) 526-6329 fax<br>
> Seattle, WA 98115 (206) 526-6317 main reception<br>
><br>
> <a href="mailto:Chris.Barker@noaa.gov">Chris.Barker@noaa.gov</a><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
> <a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>