Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison

[oyekomova]

Travis-
Yes, I tried your suggestion, but found that it took longer to read a
large file. Thanks for your help.



Travis E. Oliphant wrote:
> oyekomova wrote:
> > Thanks to everyone for their excellent suggestions. I was able to
> > acheive the following results with all your suggestions. However, I am
> > unable to cross file size of 6 million rows. I would appreciate any
> > helpful suggestions on avoiding memory errors. None of the solutions
> > posted was able to cross this limit.
>
> Did you try using numpy.fromfile ?
>
> This will not require you to allocate more memory than needed.   If you
> specify a count, it will also not have to re-allocate memory in blocks
> as the array size grows.
>
> It's limitation is that it is not a very sophisticated csv reader (it
> only understands a single separator (plus line-feeds are typically seen
> as a separator).
> 
> -Travis