Alexander Michael wrote:
I'm new to numpy and looking for advice on setting up and managing array data for my particular problem. I'm collecting observations of P properties for N objects over a rolling horizon of H sample times. I could conceptually store the data in three-dimensional array with shape (N,P,H) that would allow me to easily (and efficiently with strided slices) compute the statistics over both N and H that I am interested in. This is great, but the rub is that H, an interval of T, is a rolling horizon. T is to large to fit in memory, so I need to load up H, perform my calculations, pop the oldest N x P slice and push the newest N x P slice into the data cube. What's the best way to do this that will maintain fast computations along the one-dimensional slices over N and H? Is there a commonly accepted idiom?
Fundamentally, I see two solutions. The first would be to essentially perform a memcpy to propagate the data. The second would be to manage the N x P slices as H discontiguous memory blocks and merely reorder the pointers with each new sample. Can I do either of these with numpy?
The 'pointers reordering' can be very nicely done via deque from collections module - it behaves like a list, but has some additional methods like rotate(). I have used it successfully as a circular buffer for numpy arrays, but I can see that you need an efficient slicing over H, so a contiguous memory would be better for you IMHO. r.