[Numpy-discussion] striding through arbitrarily large files

[Julian Taylor]

On Tue, Feb 4, 2014 at 4:27 PM, RayS <rays at blue-cove.com> wrote:

> At 07:09 AM 2/4/2014, you wrote:
> >On 04/02/2014 16:01, RayS wrote:
> > > I was struggling with  methods of reading large disk files into numpy
> > > efficiently (not FITS or .npy, just raw files of IEEE floats from
> > > numpy.tostring()). When loading arbitrarily large files it would be
> nice
> > > to not bother reading more than the plot can display before zooming in.
> > > There apparently are no built in methods that allow
> skipping/striding...
> >
> >If you mmap the data file with np.memmap() you can access the data in a
> >strided way through the numpy array interface and the OS will handle the
> >scheduling of the reads from the disc.
> >
> >Note however if that the data samples you need are quite dense, there is
> >no real advantage in doing this because the OS will have to read a whole
> >page anyway for each read.
>
> Thanks Daniele, I'll be trying mmap with Python64. With 32 bit the
> mmap method throws MemoryError with 2.5GB files...
> The idea is that we allow the users to inspect the huge files
> graphically, then they can "zoom" into regions of interest and then
> load a ~100 MB en block for the usual spectral analysis.
>
>
memory maps are limited to the size of the available address space (31 bits
with sign), so you would have to slide them, see e.g. the smmap module.
But its not likely this is going to be much faster than a loop with
explicit seeks depending on the sparseness of the data. memory maps have
relatively high overheads at the operating system level.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140204/a4335d6b/attachment.html>