[Numpy-discussion] np.loadtxt : yet a new implementation...

[John Hunter]

On Mon, Dec 1, 2008 at 12:21 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> Well, looks like the attachment is too big, so here's the implementation.
> The tests will come in another message.\


It looks like I am doing something wrong -- trying to parse a CSV file
with dates formatted like '2008-10-14', with::

    import datetime, sys
    import dateutil.parser
    StringConverter.upgrade_mapper(dateutil.parser.parse,
default=datetime.date(1900,1,1))
    r = loadtxt(sys.argv[1], delimiter=',', names=True)
    print r.dtype

I get the following::

Traceback (most recent call last):
  File "genload_proposal.py", line 734, in ?
    r = loadtxt(sys.argv[1], delimiter=',', names=True)
  File "genload_proposal.py", line 711, in loadtxt
    (output, _) = genloadtxt(fname, **kwargs)
  File "genload_proposal.py", line 646, in genloadtxt
    rows[i] = tuple([conv(val) for (conv, val) in zip(converters, vals)])
  File "genload_proposal.py", line 385, in __call__
    raise ValueError("Cannot convert string '%s'" % value)
ValueError: Cannot convert string '2008-10-14'

In debug mode, I see the following where the error occurs

ipdb> vals
('2008-10-14', '116.26', '116.40', '103.14', '104.08', '70749800', '104.08')
ipdb> converters
[<__main__.StringConverter instance at 0xa35fa6c>,
<__main__.StringConverter instance at 0xa35ff2c>,
<__main__.StringConverter instance at 0xa35ff8c>,
<__main__.StringConverter instance at 0xa35ffec>,
<__main__.StringConverter instance at 0xa15406c>,
<__main__.StringConverter instance at 0xa1540cc>,
<__main__.StringConverter instance at 0xa15412c>]

It looks like my registry of a custom converter isn't working.  Here
is what the _mapper looks like::

In [23]: StringConverter._mapper
Out[23]:
[(<type 'numpy.bool_'>, <function str2bool at 0xa2b8bc4>, None),
 (<type 'numpy.integer'>, <type 'int'>, -1),
 (<type 'numpy.floating'>, <type 'float'>, -NaN),
 (<type 'complex'>, <type 'complex'>, (-NaN+0j)),
 (<type 'numpy.object_'>,
  <function parse at 0x8cf1534>,
  datetime.date(1900, 1, 1)),
 (<type 'numpy.string_'>, <type 'str'>, '???')]