[Numpy-discussion] Managing Rolling Data

[Robert Cimrman]

Alexander Michael wrote:
> I'm new to numpy and looking for advice on setting up and managing
> array data for my particular problem. I'm collecting observations of P
> properties for N objects over a rolling horizon of H sample times. I
> could conceptually store the data in three-dimensional array with
> shape (N,P,H) that would allow me to easily (and efficiently with
> strided slices) compute the statistics over both N and H that I am
> interested in. This is great, but the rub is that H, an interval of T,
>  is a rolling horizon. T is to large to fit in memory, so I need to
> load up H, perform my calculations, pop the oldest N x P slice and
> push the newest N x P slice into the data cube. What's the best way to
> do this that will maintain fast computations along the one-dimensional
> slices over N and H? Is there a commonly accepted idiom?
> 
> Fundamentally, I see two solutions. The first would be to essentially
> perform a memcpy to propagate the data. The second would be to manage
> the N x P slices as H discontiguous memory blocks and merely reorder
> the pointers with each new sample. Can I do either of these with
> numpy?

The 'pointers reordering' can be very nicely done via
deque from collections module - it behaves like a list, but has some
additional methods like rotate(). I have used it successfully as a
circular buffer for numpy arrays, but I can see that you need an
efficient slicing over H, so a contiguous memory would be better for you
IMHO.

r.