Reading a large csv file

Terry Reedy tjreedy at udel.edu
Tue Jun 23 09:09:58 CEST 2009


Mag Gam wrote:
> Yes, the system has 64Gig of physical memory.

drool ;-).

> What I meant was, is it possible to load to a hdf5 dataformat
> (basically NumPy array) without reading the entire file at first? I
> would like to splay to disk beforehand so it would be a bit faster
> instead of having 2 copies in memory.

If you can write hdf5 a line at a time, you should be able to something like

<open cvs>
<open hdf5>
for line in cvs:
   process line
   write hdf5 line

this assumes 1-1 lines.




More information about the Python-list mailing list