[Numpy-discussion] Rolling window (moving average, moving std, and more)

[Keith Goodman]

On Mon, Jan 3, 2011 at 7:41 AM, Erik Rigtorp <erik at rigtorp.com> wrote:
> On Mon, Jan 3, 2011 at 10:36, Keith Goodman <kwgoodman at gmail.com> wrote:
>> On Mon, Jan 3, 2011 at 5:37 AM, Erik Rigtorp <erik at rigtorp.com> wrote:
>>
>>> It's only a view of the array, no copying is done. Though some
>>> operations like np.std()  will copy the array, but that's more of a
>>> bug. In general It's hard to imagine any speedup gains by copying a
>>> 10GB array.
>>
>> I don't think that np.std makes a copy of the input data if the input
>> is an array. If the input is, for example, a list, then an array is
>> created.
>
> When I tried it on a big array, it tried to allocate a huge amount of
> memory. As I said it's probably a bug.

Yes, that would be a big bug.

np.std does have to initialize the output array. If the window size is
small compared to arr.shape[axis] then the memory taken by the output
array is of the same order as that of the input array. Could that be
what you are seeing?

>> a = np.arange(10)

Small window, output array shape (8,):

>> rolling_window(a, 2)
array([[0, 1],
       [1, 2],
       [2, 3],
       [3, 4],
       [4, 5],
       [5, 6],
       [6, 7],
       [7, 8],
       [8, 9]])

Big window, output array shape (2,):

>> rolling_window(a, 9)
array([[0, 1, 2, 3, 4, 5, 6, 7, 8],
       [1, 2, 3, 4, 5, 6, 7, 8, 9]])