[MATRIX-SIG] Data import into Numeric arrays

[Paul F. Dubois]

The input function can be used for breaking up lines if you put commas in
the file.

Better is to write your own extension in C or C++ that knows how to read
your files. You can use the C API to create the Python numerical arrays and
then fill them up. Or, you can use the C API to Python "list" and then at
the end convert to numerical arrays. Doing the read in C avoids having the
loop in Python, for one thing.

I do have some help for making the file in the first place. Check out
PyHistory at http://xfiles.llnl.gov/pyhistory. Two different text file
formats are available as output choices, along with Pact/PDB (which is like
netCDF or HDF, only better).

-----Original Message-----
From: Kevin Rodgers <krodgers@tdyryan.com>
To: matrix-sig@python.org <matrix-sig@python.org>
Date: Monday, February 02, 1998 9:13 AM
Subject: [MATRIX-SIG] Data import into Numeric arrays


>I have the need to process large amounts of time history data with NumPy.
>The data are stored as ordinary ASCII files with each line being one
>timepoint and its associated data.  Currently, the only ways I know to get
>this data into NumPy  all involve reading the data into a Python list (say,
>via readline or readlines), splitting each line into fields, converting the
>string representations of numbers to actual numbers, then using the array()
>function to build an array.  This seems really inefficient.  What I'd like
>to do is just read the data directly into NumPy from disk as a Numeric
>array!  Similarly with writing arrays; it would be really nice to easily
>write this kind of data out to disk.  Any ideas on this?  Currently, about
>80% of my scripts processing time is in the reading and writing of the
>datafiles.  Thanks in advance . . .
>
>(BTW, if you're only suggestion is for me to store the data in a different
>format, thanks but no thanks; this data has to be used by lots of different
>people, and I am pretty much stuck with this format, as much as I'd like to
>move it to, say netCDF or HDF or even straight binary.)
>----------------------------------------------------------------------
>Kevin Rodgers     Teledyne Ryan Aeronautical  krodgers@tdyryan.com
>"This one goes up to eleven."  --  Nigel Tufnel
>----------------------------------------------------------------------
>
>_______________
>MATRIX-SIG  - SIG on Matrix Math for Python
>
>send messages to: matrix-sig@python.org
>administrivia to: matrix-sig-request@python.org
>_______________
>


_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________