[Numpy-discussion] fromfile and text files...

[Christopher Barker]

Hi all,

I'd like to use fromfile() to read text files that look like:

23.4,    123.43
456.321, 9568.00
32.0,    134.4

so they are comma-separated values, but separated by newlines. I tried 
this code:


import numpy as np

file = "fromfiletest.txt"
a = np.fromfile(file, dtype=np.float, sep=",", count=6)
print a

and got:
6 items requested but only 2 read
[  23.4   123.43]

So, fromfile is looking for commas, and when there is a newline, instead 
of a comma, it stops looking.

This kind of kills the point of fromfile for any text files other than 
whitespace - delimited ( i assume passing in " " for sep will get you an 
y whitespace -- it does seem to work for spaces and newlines, anyway).

Yes, I know there is loadtext() and any number of other solutions, but 
they are all processing line in python, and that is a lot slower when 
reading big files.

I thought fromtext() did this, and in any case, it could -- it would be 
a simple a blazingly fast way to red many common text formats.

I think it was inspired by some code I posted a couple years back that 
would work with virtually any separator. That code only did double 
arrays, but it was fast and easy to use for the easy cases. It used C 
fscanf to simple get "the next number" then is would skip along 'till it 
found another thing it could parse as a number. since C's fscanf already 
skips whitespace, couldn't sep be interpreted as "this character and 
white space"?

Or maybe be able to pass in a list of separators?

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov