Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison
Travis E. Oliphant
oliphant.travis at ieee.org
Mon Jan 15 22:06:56 CET 2007
> Thanks to everyone for their excellent suggestions. I was able to
> acheive the following results with all your suggestions. However, I am
> unable to cross file size of 6 million rows. I would appreciate any
> helpful suggestions on avoiding memory errors. None of the solutions
> posted was able to cross this limit.
Did you try using numpy.fromfile ?
This will not require you to allocate more memory than needed. If you
specify a count, it will also not have to re-allocate memory in blocks
as the array size grows.
It's limitation is that it is not a very sophisticated csv reader (it
only understands a single separator (plus line-feeds are typically seen
as a separator).
More information about the Python-list