[SciPy-User] Alternatives to genfromtxt and loadtxt?

[Yury V. Zaytsev]

Hi!

On Sat, 2011-05-14 at 12:25 +0000, Giorgos Tzampanakis wrote:

> Thanks for the suggestion! It wasn't quite as fast as Octave, in fact it
> was about 6 times slower, but I think it'll do for an initial load. Then I
> can save to numpy's native binary format.

That's also what I do for >1 Gb matrices: just save them in the native
NumPy format for later use and then the load times become negligible,
especially from /dev/shm mounts ;-)

> The question now is, why aren't genfromtxt and loadtxt using this approach
> if it is faster than what they're doing?

I think it all comes down to post-processing and heuristics. It seems
that these functions do quite a lot of extra work to make sure that the
data is loaded correctly, the precision isn't lost etc.

I even suppose that there is a way to speed them up by specifying
formats in function calls, but I've never really got time to figure it
out and went for my extra simple reader instead, since I had to perform
some weird pre-processing on the data row by row anyway.

-- 
Sincerely yours,
Yury V. Zaytsev