[Numpy-discussion] Question about improving genfromtxt errors

[Skipper Seabold]

On Fri, Sep 25, 2009 at 2:03 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> On 09/25/2009 12:00 PM, Skipper Seabold wrote:
>> There have been some recent attempts to improve the error reporting in
>> genfromtxt<http://projects.scipy.org/numpy/ticket/1212>, which is
>> great, because hunting down the problems reading in big and messy
>> files is not fun.
>>
>> I am working on a patch that keeps up with the line number and column
>> number of where you are in parsing the file, so that this can be
>> reported in the error.  Is there a way to catch a raised error and add
>> to it?
>>
>> For instance, I have a problem in my file which leads to this error
>> being raised from np.lib._iotools.StringCoverter.upgrade
>>
>> ValueError: Converter is locked and cannot be upgraded
>>
>> I added this into np.lib.io.genfromtxt around line 995.
>>
>> linenum = 0
>> [...]
>> if dtype is None:
>>              try:
>>                  colnum = 0
>>                  for (converter, item) in zip(converters, values):
>>                      converter.upgrade(item)
>>                      colnum += 1
>>              except:
>>                  raise ValueError, "I don't report the error from
>> _iotools.StringConverter.upgrade, but I do know that there is a
>> problem trying to convert a value at line %s and column %s" %
>> (linenum,colnum)
>> [...]
>> linenum += 1
>>
>> I'd like to add line and column number information to original error
>> from _iotools.  Any suggestions?
>>
>> Cheers,
>> Skipper
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> Hi,
> I am guessing that the converter is most likely causing the error. So
> presumably the file is read correctly without using the converter. If
> not then you should address that first.
>
> If it is an input file, then what is it and the how is genfromtxt called?
>

The converter is certainly causing the error, I just need to know
where so then I can change it as appropriate.  I've had to define 100s
of converters for qualitative data/survey responses (loops doing most
of the work), so I'd rather just know where the converter is failing
and then I make the changes with the column number that is spit out in
the error.

Using your changes and my added exception has saved me so much time
that you have no idea compared with what I was doing.

>
> A question regarding the ticket than this, why do you want to raise an
> exception?
>
> The reason I did not do it was that it was helpful to identify all of
> the lines that have a specific problem. You can not assume that a user
> will fix all lines with this problem let alone fix all lines with
> similar problems.

Good point.  This would save me even more time.  Perhaps then a logger
is the way to go rather than print.  My understanding is that print
statements aren't allowed in numpy.  Is this correct?

Once I'm done, I will post my changes to the ticket and we can discuss
some more.

Skipper