[Numpy-discussion] Rolling window (moving average, moving std, and more)

[Erik Rigtorp]

On Mon, Jan 3, 2011 at 10:52, Keith Goodman <kwgoodman at gmail.com> wrote:
> On Mon, Jan 3, 2011 at 7:41 AM, Erik Rigtorp <erik at rigtorp.com> wrote:
>> On Mon, Jan 3, 2011 at 10:36, Keith Goodman <kwgoodman at gmail.com> wrote:
>>> On Mon, Jan 3, 2011 at 5:37 AM, Erik Rigtorp <erik at rigtorp.com> wrote:
>>>
>>>> It's only a view of the array, no copying is done. Though some
>>>> operations like np.std()  will copy the array, but that's more of a
>>>> bug. In general It's hard to imagine any speedup gains by copying a
>>>> 10GB array.
>>>
>>> I don't think that np.std makes a copy of the input data if the input
>>> is an array. If the input is, for example, a list, then an array is
>>> created.
>>
>> When I tried it on a big array, it tried to allocate a huge amount of
>> memory. As I said it's probably a bug.
>
> Yes, that would be a big bug.
>
> np.std does have to initialize the output array. If the window size is
> small compared to arr.shape[axis] then the memory taken by the output
> array is of the same order as that of the input array. Could that be
> what you are seeing?
>
>>> a = np.arange(10)
>
> Small window, output array shape (8,):
>
>>> rolling_window(a, 2)
> array([[0, 1],
>       [1, 2],
>       [2, 3],
>       [3, 4],
>       [4, 5],
>       [5, 6],
>       [6, 7],
>       [7, 8],
>       [8, 9]])
>
> Big window, output array shape (2,):
>
>>> rolling_window(a, 9)
> array([[0, 1, 2, 3, 4, 5, 6, 7, 8],
>       [1, 2, 3, 4, 5, 6, 7, 8, 9]])

No the array was (500,2000) and i did np.std(rolling_window(a,252),-1)
and it started to allocate > 2GB.