[Numpy-discussion] Efficient way to load a 1Gb file?

[Derek Homeier]

On 10 Aug 2011, at 22:03, Gael Varoquaux wrote:

> On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote:
>> A 1 Gb text file is a miserable object anyway, so it might be desirable
>> to convert to (say) HDF5 and then throw away the text file.
> 
> +1

There might be concerns about ensuring data accessibility agains throwing the text file away, but converting to HDF5 would be an elegant for reading in without the memory issues, too (I must confess though, I've regularly read ~ 1GB ASCII files into memory - with decent virtual memory management it did not turn out too bad...)

Cheers,
					Derek