np.ma.argmax not respecting the mask?
Dear all, I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask? I expect for all data that are masked, it should also return a mask, but this is not the case. In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6) In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1]) In [98]: np.__version__ Out[98]: '1.7.1' Can I file a bug report on this? thanks, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
On Tue, Jul 9, 2013 at 2:55 PM, Chao YUE <chaoyuejoy@gmail.com> wrote:
I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask?
In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6)
In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1])
This is the result I would expect. If both values are masked, the fill value is used, so there is always an argmin value. The following workaround should have done the trick, but it exposes a different bug: x = np.ma.array([[0,1,2,3,4],[5,6,7,8, 9]], mask=[[1, 1, 1, 1, 0], [0, 1, 0, 0 ,0]], dtype=float) np.nanargmax(x.filled(np.nan), axis=0) This breaks with "ValueError: cannot convert float NaN to integer" Stéfan
On Tue, 2013-07-09 at 15:14 +0200, Stéfan van der Walt wrote:
On Tue, Jul 9, 2013 at 2:55 PM, Chao YUE <chaoyuejoy@gmail.com> wrote:
I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask?
In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6)
In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1])
This is the result I would expect. If both values are masked, the fill value is used, so there is always an argmin value.
To be honest, I would expect the exact opposite. If there is no value, there is no minimum argument -> either its an error, or it signals invalid in some other way. On masked arrays I would expect it to be masked to signal this. The error for nanargmax is annoying, but it is right to be an error IMO, due to lack of a better representation. (Ideally mabe the user would be given the option to pass an Identity element for those nanfuncs (basically this is always NaN now, which fails for argmax since the result is integer) for which the ufunc does not have an Identity, and for those that do, we should actually use it. - Sebastian
The following workaround should have done the trick, but it exposes a different bug:
x = np.ma.array([[0,1,2,3,4],[5,6,7,8, 9]], mask=[[1, 1, 1, 1, 0], [0, 1, 0, 0 ,0]], dtype=float) np.nanargmax(x.filled(np.nan), axis=0)
This breaks with "ValueError: cannot convert float NaN to integer"
Stéfan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Jul 9, 2013, at 16:08 , Sebastian Berg <sebastian@sipsolutions.net> wrote:
On Tue, 2013-07-09 at 15:14 +0200, Stéfan van der Walt wrote:
On Tue, Jul 9, 2013 at 2:55 PM, Chao YUE <chaoyuejoy@gmail.com> wrote:
I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask?
In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6)
In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1])
This is the result I would expect. If both values are masked, the fill value is used, so there is always an argmin value.
To be honest, I would expect the exact opposite. If there is no value, there is no minimum argument -> either its an error, or it signals invalid in some other way. On masked arrays I would expect it to be masked to signal this.
The doc is quite clear: masked values are replaced by `fill_value` when determining the argmax/argmin. Attaching a mask a posteriori is always doable, but making the output of np.ma.argstuff a MaskedArray may be a nuisance at this point (any input from heavy users?).
Sorry I didn't the docs very carefully. there is no doc for np.ma.argmax for indeed there is for np.ma.argmin so it's an expected behavior rather than a bug. Let some heavy users to say their ideas. Practicaly, the returned value of 0 will be always confused with the values which are not masked but do have the minimum or maximum values at the 0 position over the specified axis. One way to walk around is: data_mask = np.ma.mean(axis=0).mask np.ma.masked_array(np.ma.argmax(data,axis=0), mask=data_mask) Chao On Tue, Jul 9, 2013 at 4:26 PM, Pierre Gerard-Marchant <pgmdevlist@gmail.com
wrote:
On Jul 9, 2013, at 16:08 , Sebastian Berg <sebastian@sipsolutions.net> wrote:
On Tue, 2013-07-09 at 15:14 +0200, Stéfan van der Walt wrote:
On Tue, Jul 9, 2013 at 2:55 PM, Chao YUE <chaoyuejoy@gmail.com> wrote:
I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask?
In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6)
In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1])
This is the result I would expect. If both values are masked, the fill value is used, so there is always an argmin value.
To be honest, I would expect the exact opposite. If there is no value, there is no minimum argument -> either its an error, or it signals invalid in some other way. On masked arrays I would expect it to be masked to signal this.
The doc is quite clear: masked values are replaced by `fill_value` when determining the argmax/argmin. Attaching a mask a posteriori is always doable, but making the output of np.ma.argstuff a MaskedArray may be a nuisance at this point (any input from heavy users?).
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
On Jul 9, 2013, at 16:38 , Chao YUE <chaoyuejoy@gmail.com> wrote:
Sorry I didn't the docs very carefully. there is no doc for np.ma.argmax for indeed there is for np.ma.argmin
Yeah, the doc of the function asks you to go check the doc of the method… Not the best.
so it's an expected behavior rather than a bug. Let some heavy users to say their ideas.
Practicaly, the returned value of 0 will be always confused with the values which are not masked but do have the minimum or maximum values at the 0 position over the specified axis.
Well, it's just an index: if you take the corresponding value from the input array, it'll be masked...
One way to walk around is:
data_mask = np.ma.mean(axis=0).mask
np.ma.masked_array(np.ma.argmax(data,axis=0), mask=data_mask)
I find easier to use `mask=x.mask.prod(axis)` to get the combined mask along the desired axis (you could also use a `reduce(np.logical_and, x.mask)` for axis=0, but it's less convenient I think).
Thanks Pierre, good to know there are so many tricks available. Chao On Tue, Jul 9, 2013 at 4:55 PM, Pierre Gerard-Marchant <pgmdevlist@gmail.com
wrote:
On Jul 9, 2013, at 16:38 , Chao YUE <chaoyuejoy@gmail.com> wrote:
Sorry I didn't the docs very carefully. there is no doc for np.ma.argmax for indeed there is for np.ma.argmin
Yeah, the doc of the function asks you to go check the doc of the method… Not the best.
so it's an expected behavior rather than a bug. Let some heavy users to say their ideas.
Practicaly, the returned value of 0 will be always confused with the values which are not masked but do have the minimum or maximum values at the 0 position over the specified axis.
Well, it's just an index: if you take the corresponding value from the input array, it'll be masked...
One way to walk around is:
data_mask = np.ma.mean(axis=0).mask
np.ma.masked_array(np.ma.argmax(data,axis=0), mask=data_mask)
I find easier to use `mask=x.mask.prod(axis)` to get the combined mask along the desired axis (you could also use a `reduce(np.logical_and, x.mask)` for axis=0, but it's less convenient I think).
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
participants (4)
-
Chao YUE -
Pierre Gerard-Marchant -
Sebastian Berg -
Stéfan van der Walt