[Numpy-discussion] Assignment from a list is slow in Numarray

[Todd Miller]

Here's another possible I/O approach which uses numarray.strings to
create and evaluate a CharArray using the new for 1.1 method,
fasteval().  This method is dependent on fixed length fields of data and
doesn't currently handle 64-bit integer types or complex numbers the way
I'd like,  but it may be useful for your case.

I created my test file like this:

>>> for i in range(10**5):
...     f.write("%10d %10d %10d\n" % (i, i, i))
...


Then I created a CharArray from the file like this:

>>> import numarray.strings as str
>>> f = open("test.dat", "r")
>>> c = str.fromfile(f, shape=(10**5, 3), itemsize = 11)
>>> c
CharArray([['         0', '         0', '         0'],
           ['         1', '         1', '         1'],
           ['         2', '         2', '         2'],
           ...,
           ['     99997', '     99997', '     99997'],
           ['     99998', '     99998', '     99998'],
           ['     99999', '     99999', '     99999']])

Finally, I converted it into a NumArray, with reasonable performance,
like this:

>>> n = c.fasteval(type=Int32)
>>> n
array([[    0,     0,     0],
       [    1,     1,     1],
       [    2,     2,     2],
       ...,
       [99997, 99997, 99997],
       [99998, 99998, 99998],
       [99999, 99999, 99999]])

Hope this helps,
Todd