large arrays in python (scientific)

[ameoba]

"Jason Orendorff" <jason at jorendorff.com> wrote in 
news:mailman.1010453290.26070.python-list at python.org:

> If you're using Python 2.2, and you simply *must* have all the
> data points, but you just don't have near enough RAM, and you
> have plenty of time to sit around and wait, then try this:
> 
> from __future__ import generators
> from Scientific.Functions.LeastSquares import leastSquaresFit
> This reads the data from the file once per iteration
> (leastSquaresFit is an iterative algorithm), which is slow;
> but it does not store the whole data set in memory, which
> might be helpful.

Assuming that your data points are evenly distributed in a grid, you could 
probably trim that down so that you didn't have to explicitly store the 
(x,y,z).  If this is the case, even working with a list of lists of lists 
(defining matrixes was one of the first struggles I had with python, before 
I found the module that did it for me) of data would probably be more 
efficient than actually storing each point.  

Of course, calculating offsets into a single array (err... list.. this is 
python) would be quite efficient and any reasonable computro should have no 
problem with 2M element list, but I think calculating multi-dimensional 
array offsets into a 1D array kinda breaks the paradigm of python 
programming.



BTW- I keep reading interesting examples of generators... they seem 
interesting, and occassionally even useful.  Where can I find some good 
info on 'em?