Here's another possible I/O approach which uses numarray.strings to create and evaluate a CharArray using the new for 1.1 method, fasteval(). This method is dependent on fixed length fields of data and doesn't currently handle 64-bit integer types or complex numbers the way I'd like, but it may be useful for your case. I created my test file like this:
for i in range(10**5): ... f.write("%10d %10d %10d\n" % (i, i, i)) ...
Then I created a CharArray from the file like this:
import numarray.strings as str f = open("test.dat", "r") c = str.fromfile(f, shape=(10**5, 3), itemsize = 11) c CharArray([[' 0', ' 0', ' 0'], [' 1', ' 1', ' 1'], [' 2', ' 2', ' 2'], ..., [' 99997', ' 99997', ' 99997'], [' 99998', ' 99998', ' 99998'], [' 99999', ' 99999', ' 99999']])
Finally, I converted it into a NumArray, with reasonable performance, like this:
n = c.fasteval(type=Int32) n array([[ 0, 0, 0], [ 1, 1, 1], [ 2, 2, 2], ..., [99997, 99997, 99997], [99998, 99998, 99998], [99999, 99999, 99999]])
Hope this helps, Todd