[Numpy-discussion] genfromtxt converter question

Fri Jun 17 11:39:39 EDT 2011

Thanks for the hints Olivier and Bruce. Based on them, the following
is a working solution, although I still have that itchy sense that genfromtxt
should be able to do it directly.

import numpy as np
from StringIO import StringIO

a = StringIO('''\
 (-3.9700,-5.0400) (-1.1318,-2.5693) (-4.6027,-0.1426) (-1.4249, 1.7330)
 (-5.4797, 0.0000) ( 1.8585,-1.5502) ( 4.4145,-0.7638) (-0.4805,-1.1976)
 ( 0.0000, 0.0000) ( 6.2673, 0.0000) (-0.4504,-0.0290) (-1.3467, 1.6579)
 ( 0.0000, 0.0000) ( 0.0000, 0.0000) (-3.5000, 0.0000) ( 2.5619,-3.3708)
''')

b = np.genfromtxt(a, dtype=str, delimiter=18)[:,:-1]
b = np.vectorize(lambda x: complex(*eval(x)))(b)

print b

On Sat, Jun 18, 2011 at 12:31 AM, Bruce Southey <bsouthey at gmail.com> wrote:
> On 06/17/2011 08:51 AM, Olivier Delalleau wrote:
>
> 2011/6/17 Bruce Southey <bsouthey at gmail.com>
>>
>> On 06/17/2011 08:22 AM, gary ruben wrote:
>> > Thanks Olivier,
>> > Your suggestion gets me a little closer to what I want, but doesn't
>> > quite work. Replacing the conversion with
>> >
>> > c = lambda x:np.cast[np.complex64](complex(*eval(x)))
>> > b = np.genfromtxt(a,converters={0:c, 1:c, 2:c,
>> > 3:c},dtype=None,delimiter=18,usecols=range(4))
>> >
>> > produces
>> >
>> > [[(-3.97000002861-5.03999996185j) (-1.1318000555-2.56929993629j)
>> >   (-4.60270023346-0.142599999905j) (-1.42490005493+1.73300004005j)]
>> >   [(-5.4797000885+0j) (1.85850000381-1.5501999855j)
>> >   (4.41450023651-0.763800024986j) (-0.480500012636-1.19760000706j)]
>> >   [0j (6.26730012894+0j) (-0.45039999485-0.0289999991655j)
>> >   (-1.34669995308+1.65789997578j)]
>> >   [0j 0j (-3.5+0j) (2.56189990044-3.37080001831j)]]
>> >
>> > which is not yet an array of complex numbers. It seems close to the
>> > solution though.
>> >
>> > Gary
>> >
>> > On Fri, Jun 17, 2011 at 8:40 PM, Olivier Delalleau<shish at keba.be>
>> >  wrote:
>> >> If I understand correctly, your error is that you convert only the
>> >> second
>> >> column, because your converters dictionary contains a single key (1).
>> >> If you have it contain keys from 0 to 3 associated to the same
>> >> function, it
>> >> should work.
>> >>
>> >> -=- Olivier
>> >>
>> >> 2011/6/17 gary ruben<gruben at bigpond.net.au>
>> >>> I'm trying to read a file containing data formatted as in the
>> >>> following example using genfromtxt and I'm doing something wrong. It
>> >>> almost works. Can someone point out my error, or suggest a simpler
>> >>> solution to the ugly converter function? I thought I'd leave in the
>> >>> commented-out line for future reference, which I thought was a neat
>> >>> way to get genfromtxt to show what it is trying to pass to the
>> >>> converter.
>> >>>
>> >>> import numpy as np
>> >>> from StringIO import StringIO
>> >>>
>> >>> a = StringIO('''\
>> >>>   (-3.9700,-5.0400) (-1.1318,-2.5693) (-4.6027,-0.1426) (-1.4249,
>> >>> 1.7330)
>> >>>   (-5.4797, 0.0000) ( 1.8585,-1.5502) ( 4.4145,-0.7638)
>> >>> (-0.4805,-1.1976)
>> >>>   ( 0.0000, 0.0000) ( 6.2673, 0.0000) (-0.4504,-0.0290) (-1.3467,
>> >>> 1.6579)
>> >>>   ( 0.0000, 0.0000) ( 0.0000, 0.0000) (-3.5000, 0.0000) (
>> >>> 2.5619,-3.3708)
>> >>> ''')
>> >>>
>> >>> #~ b = np.genfromtxt(a,converters={1:lambda
>> >>> x:str(x)},dtype=object,delimiter=18)
>> >>> b = np.genfromtxt(a,converters={1:lambda
>> >>> x:complex(*eval(x))},dtype=None,delimiter=18,usecols=range(4))
>> >>>
>> >>> print b
>> >>>
>> >>> --
>> >>>
>> >>> This produces
>> >>> [ (' (-3.9700,-5.0400)', (-1.1318-2.5693j), ' (-4.6027,-0.1426)', '
>> >>> (-1.4249, 1.7330)')
>> >>>   (' (-5.4797, 0.0000)', (1.8585-1.5502j), ' ( 4.4145,-0.7638)', '
>> >>> (-0.4805,-1.1976)')
>> >>>   (' ( 0.0000, 0.0000)', (6.2673+0j), ' (-0.4504,-0.0290)', '
>> >>> (-1.3467,
>> >>> 1.6579)')
>> >>>   (' ( 0.0000, 0.0000)', 0j, ' (-3.5000, 0.0000)', ' (
>> >>> 2.5619,-3.3708)')]
>> >>>
>> >>> which I just need to unpack into a 4x4 array, but I get an error if I
>> >>> try to apply a different view.
>> >>>
>> >>> thanks,
>> >>> Gary
>> >>> _______________________________________________
>> >>> NumPy-Discussion mailing list
>> >>> NumPy-Discussion at scipy.org
>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >>
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >>
>> >>
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> Just an observation for the StringIO object, you have multiple spaces
>> within the parentheses but, by default, you are using whitespace
>> delimiters in genfromtxt. So, yes, genfromtxt is going have issues.
>>
>> If you can rewrite the input, then you need a non-space and non-comma
>> delimiter then specify that delimiter to genfromtxt. Otherwise you are
>> probably going to have to write you own parser - for each line, split on
>> ' (' etc.
>>
>> Bruce
>
> It's funny though because that part (the parsing) actually seems to work.
>
> However I've been playing a bit with his example and indeed I can't get
> numpy to return a complex array. It keeps resulting in an "object" dtype.
> The only way I found was to convert it temporarily into a list to recast it
> in complex64, adding the line:
>   b = np.array(map(list, b), dtype=np.complex64)
>
> -=- Olivier
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> Yes, my mistake as I did not see that the delimiter was set to 18.
> The problem is about the conversion into complex which I do not think is the
> converter per se. So a brute force way is:
>
> b= np.genfromtxt(a, dtype=str,delimiter=18)
> nrow, ncol=b.shape
> print b
> d=np.empty((nrow, ncol), dtype=np.complex)
> for row in range(nrow):
>     for col in range(ncol):
>         d[row][col]=complex(*eval(b[row][col]))
> print d
>
>
> Bruce
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>