large arrays in python (scientific)

Dennis Roark denro at
Wed Jan 9 12:54:48 EST 2002

>> > I have a data set that contains values for positions in 3D space.
>> > There are about 2 million data points
>> > (128x128x128).
>> >
>> > I'm trying to fit a function to the data. I want to use the
>> > LeastSquaresFit procedure in ScientificPython,
>> I know nothing about Python, but you sure it's the best instrument
>> for this sort of stuff ? Sounds more like something for R to me.
>> ( ).
>> Anyway you have a huge data set .... going to need a lot of memory.
>> --
>> Tae Kyon
>> Non c'è nulla di più animale
>> della coscienza pulita,
>> sul terzo pianeta del sistema solare

For very large matrices, where most of the elements are 0, a different 
data structure than a multi-dimensional array can be much more efficient: 
look into Sparse Matrices.  They will store only those elemenents that are 
non-zero.  If you need to multiply two sparse matrices, you can take the 
transpose of one and then perform the multiplication scanning down the 
rows of each matrix.  (In a sparse matrix, it is difficult to travel down 
a particular column, but the transpose turns a column traversal into an 
easier row traversal.)  The sparse matrix saves the huge amount of memory 
that a regular matrix would use, and speeds matrix manipulation because of 

Dennis Roark

denro at
Starting Points:

More information about the Python-list mailing list