Reading a large csv file
__peter__ at web.de
Tue Jun 23 02:47:43 EDT 2009
Mag Gam wrote:
> Yes, the system has 64Gig of physical memory.
> What I meant was, is it possible to load to a hdf5 dataformat
> (basically NumPy array) without reading the entire file at first? I
> would like to splay to disk beforehand so it would be a bit faster
> instead of having 2 copies in memory.
It is certainly possible to read a csv file one line at a time. What exactly
are you doing to convert it into hdf5?
Showing some of your code might help.
More information about the Python-list