On Sun, May 16, 2010 at 1:18 PM, Eric Firing wrote:

On 05/16/2010 09:24 AM, Keith Goodman wrote:

...

On Sun, May 16, 2010 at 12:14 PM, Davide Lasagna  wrote:

...

Hi all, What is the fastest and lowest memory consumption way to compute this? y = np.arange(2**24) bases = y[1:] + y[:-1] Actually it is already quite fast, but i'm not sure whether it is occupying some temporary memory is the summation. Any help is appreciated.

Is it OK to modify y? If so:

...

...

y = np.arange(2**24) z = y[1:] + y[:-1]  #<--- Slow way y[:-1] += y[1:]  #<--- Fast way (y[:-1] == z).all()     True

It's not faster on my machine, as timed with ipython:

In [8]:y = np.arange(2**24)

In [9]:b = np.array([1,1], dtype=int)

In [10]:timeit np.convolve(y, b, 'valid') 1 loops, best of 3: 484 ms per loop

In [11]:timeit y[1:] + y[:-1] 10 loops, best of 3: 181 ms per loop

In [12]:timeit y[:-1] += y[1:] 10 loops, best of 3: 183 ms per loop

If we include the fake data generation in the timing, to reduce cache bias in the repeated runs, the += method is noticeably slower.

In [13]:timeit y = np.arange(2**24); z = y[1:] + y[:-1] 1 loops, best of 3: 297 ms per loop

In [14]:timeit y = np.arange(2**24); y[:-1] += y[1:]; z = y[:-1] 1 loops, best of 3: 322 ms per loop

That's interesting. On my computer it is faster:

...

timeit y = np.arange(2**24); z = y[1:] + y[:-1] 10 loops, best of 3: 144 ms per loop timeit y = np.arange(2**24); y[:-1] += y[1:]; z = y[:-1] 10 loops, best of 3: 114 ms per loop

What accounts for the performance difference? Cache size? I assume the in-place version uses less memory. Neat if timeit reported memory usage. I haven't tried numexp, that might be something to try too.

Well, actually np.arange(2**24) was just to test the following line ;). I'm particularly concerned about memory consumption rather than speed. On 16 May 2010 22:53, Brent Pedersen wrote:

On Sun, May 16, 2010 at 12:14 PM, Davide Lasagna wrote:

...

Hi all, What is the fastest and lowest memory consumption way to compute this? y = np.arange(2**24) bases = y[1:] + y[:-1] Actually it is already quite fast, but i'm not sure whether it is occupying some temporary memory is the summation. Any help is appreciated. Cheers Davide _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

how about something like this? may have off-by-1 somewhere.

...

...

...

bases = np.arange(1, 2*2**24-1, 2)

---

NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Mon, May 17, 2010 at 11:06 AM, Francesc Alted wrote:

A Sunday 16 May 2010 21:14:34 Davide Lasagna escrigué:

...

Hi all,

What is the fastest and lowest memory consumption way to compute this?

y = np.arange(2**24) bases = y[1:] + y[:-1]

Actually it is already quite fast, but i'm not sure whether it is occupying some temporary memory is the summation. Any help is appreciated.

Both y[1:] and y[:-1] are views of the original y array, so you are not wasting temporary space here.  So, as I see this, the above idiom is as efficient as it can get in terms of memory usage.

I thought that this y[:-1] += y[1:] uses half the memory of this bases = y[1:] + y[:-1]