[Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt

Julian Taylor jtaylor.debian at googlemail.com
Tue Oct 28 16:30:40 EDT 2014


On 28.10.2014 21:24, Nathaniel Smith wrote:
> On 28 Oct 2014 20:10, "Chris Barker" <chris.barker at noaa.gov
> <mailto:chris.barker at noaa.gov>> wrote:
>>
>> Memory efficiency -- somethign like my growable array is not all that
> hard to implement and pretty darn quick -- you just do the usual trick_
> over allocate a bit of memory, and when it gets full re-allocate a
> larger chunk.
> 
> Can't you just do this with regular numpy using .resize()? What does
> your special class add? (Just curious.)
> 
>> From a quick loo, it seems that the Panda's code is pretty nice --
> maybe the 2X memory footprint should be ignored.
> 
> +1
> 
> It's fun to sit around and brainstorm clever implementation strategies,
> but Wes already went ahead and implemented all the tricky bits, and
> optimized them too. No point in reinventing the wheel.
> 

just to through it in there, astropy recently also added a faster ascii
file reader:
https://groups.google.com/forum/#!topic/astropy-dev/biCgb3cF0v0
not familiar with how it compares to pandas.

how is pandas support for unicode text files?
unicode is the big weak point of numpys current text readers and needs
to addressed.



More information about the NumPy-Discussion mailing list