On 21/02/07, Alexander Michael <lxander.m@gmail.com> wrote:
On 2/21/07, Mike Ressler <mike.ressler@alum.mit.edu> wrote:
Would loading your data via memmap, then slicing it, do your job (using numpy.memmap)? ...
Interesting idea. I think Anne's suggestion that sliced assignment will reduce to an efficient memcpy fits my needs a bit better than memmap because I'll be pushing new N x P samples into the array that will be arriving while the monitor is running.
If you want a record of all of them on disk anyway, with careful management you can read from a file as it's being written, though you'll need some rather exotic numpy hackery to have an ever-growing array. It might actually be nice to have a little wrapper object for this sort of thing, as you can't be the only one who needs it.
Actually, I'm hoping sliced self-assignment is as efficient as memmove (i.e. without creating temporaries), since the dst and src are overlapping, but I haven't tried it yet to confirm if this is relatively efficient.
I think it is almost as efficient as memmove; in particular, it doesn't create any temporaries (be careful which way you do the assignment, in fact, or you'll have an array of all the same thing) but it does use a for loop in C (maybe several) instead of a highly-optimized C library bulk copy. (I'm not sure about this - it could be special-cased in the assignment code, but I don't think it is.) Anne