Reading a large csv file

Peter Otten __peter__ at
Tue Jun 23 02:47:43 EDT 2009

Mag Gam wrote:

> Yes, the system has 64Gig of physical memory.
> What I meant was, is it possible to load to a hdf5 dataformat
> (basically NumPy array) without reading the entire file at first? I
> would like to splay to disk beforehand so it would be a bit faster
> instead of having 2 copies in memory.

It is certainly possible to read a csv file one line at a time. What exactly 
are you doing to convert it into hdf5? 

Showing some of your code might help.


More information about the Python-list mailing list