get uniform binary data into array for further processing

[Paul Hughett]

chr_w at gmx.de wrote:

: I just started coding with python recently and ran into my first major
: problem. Ok, here's the story:

: I have to import pretty massive binary files containing double-precision
: real values (climate data) so I can do some operations to it (mainly
: simplyfying, getting the daily some etc.). The file can get really ugly in size... up to a gig eventually... 

: But my main concern:
: a) what's the best and fasted way to get this data into a 2-dimensional
: array (or list?)?
: b) which modul to use (Numeric, array, struct, pickle) ???
: c) I have some fortran code which handles this quite good - is there a
: similar way to do it in python?

If the data is in IEEE 754 format (common, but I don't recall
specifically if the SGI uses it), you can read the data using the
BBLimage/Pyvox package from

http://www.uphs.upenn.edu/bbl/publications_downloads/downloads/software.shtml

There is one catch: the data will be read into Pyvox arrays, which are
not yet compatible with Numeric arrays; but you can do many useful
data operations using Pyvox arrays themselves.  (Pyvox was designed
for volume image processing and happily handles data sets containg 10
million samples.)

You might want to wait a day or two; I've just released a new version
(0.65) but couldn't upload it to the website because the server was
down.


Paul Hughett