Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison

Travis E. Oliphant oliphant.travis at ieee.org
Mon Jan 15 22:06:56 CET 2007


oyekomova wrote:
> Thanks to everyone for their excellent suggestions. I was able to
> acheive the following results with all your suggestions. However, I am
> unable to cross file size of 6 million rows. I would appreciate any
> helpful suggestions on avoiding memory errors. None of the solutions
> posted was able to cross this limit.

Did you try using numpy.fromfile ?

This will not require you to allocate more memory than needed.   If you 
specify a count, it will also not have to re-allocate memory in blocks 
as the array size grows.

It's limitation is that it is not a very sophisticated csv reader (it 
only understands a single separator (plus line-feeds are typically seen 
as a separator).

-Travis




More information about the Python-list mailing list