[Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt

Jeff Reback jeffreback at gmail.com
Sun Oct 26 07:54:14 EDT 2014


you should have a read here/
http://wesmckinney.com/blog/?p=543

going below the 2x memory usage on read in is non trivial and costly in terms of performance 

> On Oct 26, 2014, at 4:46 AM, Saullo Castro <saullogiovani at gmail.com> wrote:
> 
> I would like to start working on a memory efficient alternative for np.loadtxt and np.genfromtxt that uses arrays instead of lists to store the data while the file iterator is exhausted.
> 
> The motivation came from this SO question:
> 
> http://stackoverflow.com/q/26569852/832621
> 
> where for huge arrays the current NumPy ASCII readers are really slow and require ~6 times more memory. This case I tested with Pandas' read_csv() and it required 2 times more memory.
> 
> I would be glad if you could share your experience on this matter.
> 
> Greetings,
> Saullo
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20141026/b052cbc4/attachment.html>


More information about the NumPy-Discussion mailing list