large arrays in python (scientific)
denro at NSPAMearthlink.net
Wed Jan 9 12:54:48 EST 2002
>> > I have a data set that contains values for positions in 3D space.
>> > There are about 2 million data points
>> > (128x128x128).
>> > I'm trying to fit a function to the data. I want to use the
>> > LeastSquaresFit procedure in ScientificPython,
>> I know nothing about Python, but you sure it's the best instrument
>> for this sort of stuff ? Sounds more like something for R to me.
>> ( http://www.cran.org ).
>> Anyway you have a huge data set .... going to need a lot of memory.
>> Tae Kyon
>> Non c'è nulla di più animale
>> della coscienza pulita,
>> sul terzo pianeta del sistema solare
For very large matrices, where most of the elements are 0, a different
data structure than a multi-dimensional array can be much more efficient:
look into Sparse Matrices. They will store only those elemenents that are
non-zero. If you need to multiply two sparse matrices, you can take the
transpose of one and then perform the multiplication scanning down the
rows of each matrix. (In a sparse matrix, it is difficult to travel down
a particular column, but the transpose turns a column traversal into an
easier row traversal.) The sparse matrix saves the huge amount of memory
that a regular matrix would use, and speeds matrix manipulation because of
denro at earthlink.net
More information about the Python-list