[Numpy-discussion] Question about improving genfromtxt errors

[Skipper Seabold]

On Fri, Sep 25, 2009 at 3:51 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> One more thought:
>
> Pierre GM wrote:
>>>> That way, we don't slow
>>>> things down when everything works,
>
> how long can it really take to increment an integer as each line is
> parsed? I'd suspect no one would even notice!
>

A 1000 converters later...

FWIW, I have a script that creates and savez arrays from several text
files in total about 1.5 GB of text.

without the incrementing in genfromtxt

Run time: 122.043943 seconds

with the incrementing in genfromtxt

Run time: 131.698873 seconds

If we just want to always keep track of things, I would be willing to
take a poorly measured 8 % slowdown, because the info that I took from
the errors is the only thing that made what I was doing at all
feasible.

Skipper