[Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt

Daniele Nicolodi daniele at grinta.net
Sun Oct 26 12:42:32 EDT 2014


On 26/10/14 09:46, Saullo Castro wrote:
> I would like to start working on a memory efficient alternative for
> np.loadtxt and np.genfromtxt that uses arrays instead of lists to store
> the data while the file iterator is exhausted.

...

> I would be glad if you could share your experience on this matter.

I'm of the opinion that if your workflow requires you to regularly load
large arrays from text files, something else needs to be fixed rather
than the numpy speed and memory usage in reading data from text files.

There are a number of data formats that are interoperable and allow to
store data much more efficiently. hdf5 is one natural choice, maybe with
the blosc compressor.

Cheers,
Daniele




More information about the NumPy-Discussion mailing list