bug in genfromtxt with missing values?
Am I misreading the docs or missing something? Consider the following adapted from here: http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html from StringIO import StringIO import numpy as np data = "1, 2, 3\n4, ,5" np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values=" ", filling_values=0) array([(1.0, 2.0, 3.0), (4.0, nan, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')]) np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values={'b':" "}, filling_values={'b' : 0}) array([(1.0, 2.0, 3.0), (4.0, 0.0, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')]) Unless I use the dict for missing_values, it doesn't fill them in. Without named columns np.genfromtxt(StringIO(data), delimiter=",", missing_values=" ", filling_values=0) array([[ 1., 2., 3.], [ 4., nan, 5.]]) np.genfromtxt(StringIO(data), delimiter=",", missing_values={1 :" "}, filling_values={1 :0}) array([[ 1., 2., 3.], [ 4., 0., 5.]]) Skipper
On Jan 24, 2011, at 11:47 PM, Skipper Seabold wrote:
Am I misreading the docs or missing something? Consider the following adapted from here: http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html
from StringIO import StringIO import numpy as np
data = "1, 2, 3\n4, ,5"
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values=" ", filling_values=0) array([(1.0, 2.0, 3.0), (4.0, nan, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values={'b':" "}, filling_values={'b' : 0}) array([(1.0, 2.0, 3.0), (4.0, 0.0, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
Unless I use the dict for missing_values, it doesn't fill them in.
It's probably a bug . Mind opening a ticket ? I'll try to care of it when I can. Thx in advance P.
On Tue, Jan 25, 2011 at 11:17 AM, Pierre GM <pgmdevlist@gmail.com> wrote:
On Jan 24, 2011, at 11:47 PM, Skipper Seabold wrote:
Am I misreading the docs or missing something? Consider the following adapted from here: http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html
from StringIO import StringIO import numpy as np
data = "1, 2, 3\n4, ,5"
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values=" ", filling_values=0) array([(1.0, 2.0, 3.0), (4.0, nan, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values={'b':" "}, filling_values={'b' : 0}) array([(1.0, 2.0, 3.0), (4.0, 0.0, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
Unless I use the dict for missing_values, it doesn't fill them in.
It's probably a bug . Mind opening a ticket ? I'll try to care of it when I can. Thx in advance P.
http://projects.scipy.org/numpy/ticket/1722 Forgot to use the code formatting, and it doesn't look like I can edit. Thanks, Skipper
On 01/25/2011 10:56 AM, Skipper Seabold wrote:
On Tue, Jan 25, 2011 at 11:17 AM, Pierre GM<pgmdevlist@gmail.com> wrote:
On Jan 24, 2011, at 11:47 PM, Skipper Seabold wrote:
Am I misreading the docs or missing something? Consider the following adapted from here: http://docs.scipy.org/doc/numpy/user/basics.io.genfromtxt.html
from StringIO import StringIO import numpy as np
data = "1, 2, 3\n4, ,5"
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values=" ", filling_values=0) array([(1.0, 2.0, 3.0), (4.0, nan, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", missing_values={'b':" "}, filling_values={'b' : 0}) array([(1.0, 2.0, 3.0), (4.0, 0.0, 5.0)], dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])
Unless I use the dict for missing_values, it doesn't fill them in.
It's probably a bug . Mind opening a ticket ? I'll try to care of it when I can. Thx in advance P. http://projects.scipy.org/numpy/ticket/1722
Forgot to use the code formatting, and it doesn't look like I can edit.
Thanks,
Skipper Hi, Your filling_values is zero so there is this line (1295?) in the code: user_filling_values = filling_values or []
Which of cause presumes your filling_values is not something like 0 or [0]. Now it can be a code bug or just undocumented feature that filling_values can not a single zero. Thus something like these work: np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", filling_values=-90) np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", filling_values=[0,0]) np.genfromtxt(StringIO(data), delimiter=",", names="a,b,c", filling_values=[0,0,0]) Bruce
On Jan 25, 2011, at 9:06 PM, Bruce Southey wrote:
Your filling_values is zero so there is this line (1295?) in the code: user_filling_values = filling_values or []
Which of cause presumes your filling_values is not something like 0 or [0].
That's the bug. I forgot that filling_values could be 0. (I was more thinking of None) so it should be if filling_values is None: filling_values = [] user_filling_values = filling_values.
Now it can be a code bug or just undocumented feature that filling_values can not a single zero. Thus something like these work:
You're too kind. That's just sloppy coding... If you correct it before i do, don't forget to add a test case... Thx again P.
participants (3)
-
Bruce Southey -
Pierre GM -
Skipper Seabold