[Numpy-discussion] Question about improving genfromtxt errors

[Skipper Seabold]

On Tue, Sep 29, 2009 at 4:36 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>
> Hi,
> The first case just has to handle a missing delimiter - actually I expect
> that most of my cases would relate this. So here is simple Python code to
> generate arbitrary large list with the occasional missing delimiter.
>
> I set it so it reads the desired number of rows and frequency of bad rows
> from the linux command line.
> $time python tbig.py 1000000 100000
>
> If I comment out the extra prints in io.py that I put in, it takes about 22
> seconds to finish if the delimiters are correct. If I have the missing
> delimiter it takes 20.5 seconds to crash.
>

One other point that perhaps goes without saying is that we want to
detect missing and extra delimiters (eg., commas for 1000s).

Skipper