[Tutor] Problems creating very large arrays (over 10 million indices)

[Reggie Dugard]

Edmund,

Assuming that fp is a file object opened for reading your file and that
you've already read past the header, maybe one of these will work for
you:

a = Numeric.fromstring(fp.read(), Numeric.Int16)
Numeric.reshape(a, (xsize, ysize, zsize))

OR

a = numarray.fromfile(fp, numarray.Int16, (xsize, ysize, zsize))

I don't have much experience reading in data from outside sources, so I
can't say that I've tested these, but I hope they steer you in the right
direction.


On Tue, 2003-07-01 at 11:20, DORSEY_EDMUND_K@LILLY.COM wrote:
> I tried creating a Numeric array but I couldn't get the binary data into 
> it. The array module was so great cause I could just use readfile to read 
> in the entire file at once.
> 
> I tried this ...
> 
> a = Numeric.ones((xsize, ysize, zsize), Numeric.Int16) #create 3d numeric 
> array of type short filled with ones (placeholders)
> a[0][0][0] = file.read(2) #read in two bytes and stick it in the array
> 
> Of course it says the that the array 'a' needs an int as input.  Plus if I 
> do this method I need to use 3 nested for loops and it is significantly 
> slower than using a normal 1d array.  Is this how I would have to do it 
> with the Numeric module?  Thanks for all the input you've given. 
> 
> ~Edmund
> 
> 
> 
> 
> 
> Edmund,
> 
> Given the size of your arrays and that they are 3 dimensional, you may 
> want
> to take a look at the Numeric package or the relatively new numarray
> package.  They can handle multi-dimensional data in a very efficient 
> manner.
> They both can be found on sourceforge at 
> 
> http://sourceforge.net/projects/numpy/
> 

-- 
Reggie