[Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt
rays at blue-cove.com
Sun Oct 26 14:40:52 EDT 2014
At 06:32 AM 10/26/2014, you wrote:
>On Sun, Oct 26, 2014 at 1:21 PM, Eelco Hoogendoorn
><hoogendoorn.eelco at gmail.com> wrote:
> > Im not sure why the memory doubling is necessary. Isnt it possible to
> > preallocate the arrays and write to them?
>Not without reading the whole file first to know how many rows to preallocate
Seems to me that loadtxt()
should have an optional shape. I often know how many rows I have (#
of samples of data) from other meta data.
- if the file is smaller for some reason (you're not sure and pad
your estimate) it could do one of
- zero pad array
- return truncated view
- if larger
- return data read (this would act like fileObject.read( size ) )
- Ray S
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NumPy-Discussion