[Numpy-discussion] striding through arbitrarily large files

[Daniele Nicolodi]

On 04/02/2014 16:01, RayS wrote:
> I was struggling with  methods of reading large disk files into numpy
> efficiently (not FITS or .npy, just raw files of IEEE floats from
> numpy.tostring()). When loading arbitrarily large files it would be nice
> to not bother reading more than the plot can display before zooming in.
> There apparently are no built in methods that allow skipping/striding...

If you mmap the data file with np.memmap() you can access the data in a
strided way through the numpy array interface and the OS will handle the
scheduling of the reads from the disc.

Note however if that the data samples you need are quite dense, there is
no real advantage in doing this because the OS will have to read a whole
page anyway for each read.

Cheers,
Daniele