[Numpy-discussion] Loading a > GB file into array

[Martin Spacek]

Martin Spacek wrote:
 > Would it be better to load the file one
 > frame at a time, generating nframes arrays of shape (height, width),
 > and sticking them consecutively in a python list?

I just tried this, and it works. Looks like it's all in physical RAM (no 
disk thrashing on the 2GB machine), *and* it's easy to index into. I 
guess I should of thought of this a while ago, since each entry in a 
python list can point to anywhere in memory. Here's roughly what the 
code looks like:

import numpy as np

f = file(fname, 'rb') # 1.3GB file
frames = [None] * nframes # init a list to hold all frames
for framei in xrange(nframes): # one frame at a time...
     frame = np.fromfile(f, np.uint8, count=framesize) # load next frame
     frame.shape = (height, width)
     frames[framei] = frame # save it in the list


--
Martin