[Matrix-SIG] Re: Reading and handling matrices.

[Herbert L. Roitblat]

Thanks very much for your really helpful response.  Let me digest that
before I bother anyone further.  Between you and Konrad I feel that I am on
my way.

The reason I used matlab at all was because it had the sparse matrix
functions I was looking for.  I have a sparse matrix consisting of integers
spread over a large matrix.  First I am practicing with a fairly "small"
matrix of only 2700 x 2700 element (about 350,000 of which are nonzero), but
I expect that the program will be used with matrices up to 50k x 50k, hence
the need for sparsity.  I need to get the eigen vectors for that matrix.

I am on the edge of my knowledge when dealing with many of these problems
(both in terms of the linerar algebra and in terms of python), so I really
appreciate your help.

Thanks a lot.
Herb

-----Original Message-----
From: Travis Oliphant <Oliphant.Travis@mayo.edu>
To: roitblat@hawaii.edu <roitblat@hawaii.edu>
Cc: matrix-sig@python.org <matrix-sig@python.org>
Date: Friday, April 23, 1999 8:25 AM
Subject: Reading and handling matrices.


>
>> I'm just starting with python and I'm having a few problems, which I =
>> suspect are rather simple-minded.
>
>Welcome aboard, starting with a new system can be intimidating.
>
>> 1. I was looking for some sparse matrix functions that would let me =
>> compute the eigen vectors of a rather large matrix in a reasonable =
>> amount of time.  I have found the eigen value functions in the numeric =
>> library and I have found some linear algebra routines that allow sparse
>> matrices, but I do not see a way to combine these.  I ended up saving =
>> the matrix as ascii, editing it using word, reading it using excel, and
>> then importing it to matlab.  There has to be a better way.
>
>Sparse matrices:
>
>As Konrad said there are several ways to handle sparse matrices, but
>no "comprehensive" package. There is a sparse matrix module that I can't
>say anything about except that it exists.  It defines a sparse matrix
>object and some methods for linear equation solving.  Look on
>http://www.python.org/topics/scicomp/numbercrunching.html
>for the Sparse Linear Equation Solver module by Neil Schemenauer
>
>Adding good sparse matrix support is something I'll be trying to add to my
>collection of useful packages for NumPy over the next year or two.  When I
>get to that point I'll likely start with the module on that page.
>
>It sounds like you went through a lot of work getting data in and out of
>NumPy arrays.  This was a source of confusion for me at first, too, so I
>understand a bit of the frustration.  The frustration does disappear
>as you get a feel for the structure of Python and the Numeric extensions.
>
>I wanted to read arbitrary binary files and so wrote a C-extension module
>that reads arbitrary binary data into Python NumPy arrays.  It turns out,
>I could have accomplished what I wanted to do with the fromstring function
>from the Numeric module and the read method of any file object.
>
>One method to get your data into MATLAB is to open a file object in
>Python and write the data in your NumPy array as a "string" (i.e. a
>sequence of raw bytes).
>
>>>> import Numeric
>>>> a = Numeric.ones((512,512))  # an uninteresting array of integers
>>>> fid = open("myfile","w")
>>>> fid.write(a.tostring())
>>>> fid.close()
>
>If you are on a 32-bit machine, you will now have a file called "myfile"
>in the current directory that should contain (512*512*4) bytes of 4-byte
>integer ones.
>
>You can read this data into MATLAB using matlab's fread function:
>
>>> fid = fopen('myfile');
>>> a = fread(fid,512*512,'int');
>>> fclose(fid);
>
>The same sequence works in reverse use MATLAB's fwrite function and
>Numeric Python's fromstring function and the read method of an open file
>object.
>
>You should NEVER have to edit numbers by hand in Word.  It makes me sad
>that you felt the need to do that.  This is a pretty low level way to read
>and write data, but is indispensable when interfacing with other code and
>outputs from other code.  A better solution is to use a higher level file
>format like NETCDF which you can read and write to from MATLAB and NumPy.
>
>Of course, I would wonder why you were doing much work in MATLAB anyway
>:-) (and if you are then what is NumPy missing and we'll add it.)
>
>If you want it I could send you a module I rarely use now that makes the
>NumPy way of reading and writing binary data more like MATLAB's which I
>was used, too when I started.
>
>Best of luck on your newfound tool.  Feel free to ask anymore questions as
>they arise.
>
>Sincerely,
>
>Travis Oliphant.
>
>(just a run-o'-the-mill NumPy fan)
>
>