Numpy 'None' comparison FutureWarning
Hi,
I just encountered the following in my code:
FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
I'm very concerned about this. This is a very common programming pattern (lazy loading):
class A(object): def __init__(self): self._some_array = None
@property def some_array(self): if self._some_array == None: # perform some expensive setup of array return self._some_array
It seems to me that the new behavior will break this pattern. I think that redefining the "==" operator is a little too aggressive here. It strikes me as very nonstandard and not at all obvious to someone reading the code that the comparison is a very special case for numpy objects. Unless there's some aspect I'm missing here, I think an element-wise comparator should be more explicit.
Cheers, Demitri
_________________________________________ Demitri Muna
Department of Astronomy Le Ohio State University
On 2014/09/21, 11:10 AM, Demitri Muna wrote:
Hi,
I just encountered the following in my code:
FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
I'm very concerned about this. This is a very common programming pattern (lazy loading):
class A(object): def __init__(self): self._some_array = None
@property def some_array(self): if self._some_array == None: # perform some expensive setup of array return self._some_arrayIt seems to me that the new behavior will break this pattern. I think that redefining the "==" operator is a little too aggressive here. It strikes me as very nonstandard and not at all obvious to someone reading the code that the comparison is a very special case for numpy objects. Unless there's some aspect I'm missing here, I think an element-wise comparator should be more explicit.
I think what you are missing is that the standard Python idiom for this use case is "if self._some_array is None:". This will continue to work, regardless of whether the object being checked is an ndarray or any other Python object.
Eric
Cheers, Demitri
Demitri Muna
Department of Astronomy Le Ohio State University
http://trillianverse.org http://scicoder.org
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
That being said, I do wonder about related situations where the lhs of the equal sign might be an array, or it might be a None and you are comparing against another numpy array. In those situations, you aren't trying to compare against None, you are just checking if two objects are equivalent.
When is this change planned?
Ben Root
On Sun, Sep 21, 2014 at 5:19 PM, Eric Firing efiring@hawaii.edu wrote:
On 2014/09/21, 11:10 AM, Demitri Muna wrote:
Hi,
I just encountered the following in my code:
FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
I'm very concerned about this. This is a very common programming pattern (lazy loading):
class A(object): def __init__(self): self._some_array = None
@property def some_array(self): if self._some_array == None: # perform some expensive setup of array return self._some_arrayIt seems to me that the new behavior will break this pattern. I think that redefining the "==" operator is a little too aggressive here. It strikes me as very nonstandard and not at all obvious to someone reading the code that the comparison is a very special case for numpy objects. Unless there's some aspect I'm missing here, I think an element-wise comparator should be more explicit.
I think what you are missing is that the standard Python idiom for this use case is "if self._some_array is None:". This will continue to work, regardless of whether the object being checked is an ndarray or any other Python object.
Eric
Cheers, Demitri
Demitri Muna
Department of Astronomy Le Ohio State University
http://trillianverse.org http://scicoder.org
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sep 21, 2014, at 5:19 PM, Eric Firing efiring@hawaii.edu wrote:
I think what you are missing is that the standard Python idiom for this use case is "if self._some_array is None:". This will continue to work, regardless of whether the object being checked is an ndarray or any other Python object.
That's an alternative, but I think it's a subtle distinction that will be lost on many users. I still think that this is something that can easily trip up many people; it's not clear from looking at the code that this is the behavior; it's "hidden". At the very least, I strongly suggest that the warning point this out, e.g.
"FutureWarning: comparison to `None` will result in an elementwise object comparison in the future; use 'value is None' as an alternative."
Assume:
a = np.array([1, 2, 3, 4]) b = np.array([None, None, None, None])
What is the result of "a == None"? Is it "np.array([False, False, False, False])"?
What about the second case? Is the result of "b == None" -> np.array([True, True, True, True])? If so, then
if (b == None): ...
will always evaluate to "True" if b is "None" or *any* Numpy array, and that's clearly unexpected behavior.
On Sep 21, 2014, at 9:30 PM, Benjamin Root ben.root@ou.edu wrote:
That being said, I do wonder about related situations where the lhs of the equal sign might be an array, or it might be a None and you are comparing against another numpy array. In those situations, you aren't trying to compare against None, you are just checking if two objects are equivalent.
Right. With this change, using "==" with numpy arrays now sometimes means "are these equivalent" and other times "element-wise comparison". The potential for inadvertent bugs is far greater than what convenience this redefinition of a very basic operator might offer. Any scenario where
(a == b) != (b == a)
is asking for trouble.
Cheers, Demitri
_________________________________________ Demitri Muna
Department of Astronomy An Ohio State University
On 22 Sep 2014 03:02, "Demitri Muna" demitri.muna@gmail.com wrote:
On Sep 21, 2014, at 5:19 PM, Eric Firing efiring@hawaii.edu wrote:
I think what you are missing is that the standard Python idiom for this use case is "if self._some_array is None:". This will continue to work, regardless of whether the object being checked is an ndarray or any other Python object.
That's an alternative, but I think it's a subtle distinction that will be
lost on many users. I still think that this is something that can easily trip up many people; it's not clear from looking at the code that this is the behavior; it's "hidden". At the very least, I strongly suggest that the warning point this out, e.g.
"FutureWarning: comparison to `None` will result in an elementwise object
comparison in the future; use 'value is None' as an alternative."
Making messages clearer is always welcome, and we devs aren't always in the best position to do so because we're to close to the issues to see which parts are confusing to outsiders - perhaps you'd like to submit a pull request with this?
Assume:
a = np.array([1, 2, 3, 4]) b = np.array([None, None, None, None])
What is the result of "a == None"? Is it "np.array([False, False, False,
False])"?
After this change, yes.
What about the second case? Is the result of "b == None" ->
np.array([True, True, True, True])?
Yes again.
(Notice that this is also a subtle and confusing point for many users - how many people realize that if they want to get the latter result they have to write np.equal(b, None)?)
If so, then
if (b == None): ...
will always evaluate to "True" if b is "None" or *any* Numpy array, and
that's clearly unexpected behavior.
No, that's not how numpy arrays interact with if statements. This is independent of the handling of 'arr == None': 'if multi_element_array' is always an error, because an if statement by definition requires a single true/false decision (it can't execute both branches after all!), but a multi-element array by definition contains multiple values that might have contradictory truthiness.
Currently, 'b == x' returns an array in every situation *except* when x happens to be 'None'. After this change, 'b == x' will *always* return an array, so 'if b == x' will always raise an error.
On Sep 21, 2014, at 9:30 PM, Benjamin Root ben.root@ou.edu wrote:
That being said, I do wonder about related situations where the lhs of
the equal sign might be an array, or it might be a None and you are comparing against another numpy array. In those situations, you aren't trying to compare against None, you are just checking if two objects are equivalent.
Benjamin, can you give a more concrete example? Right now the *only* time == on arrays checks for equivalence is when the object being compared against is None, in which case == pretends to be 'is' because of this mysterious special case. In every other case it does a broadcasted ==, which is very different.
Right. With this change, using "==" with numpy arrays now sometimes means
"are these equivalent" and other times "element-wise comparison".
Err, you have this backwards :-). Right now == means element-wise comparison except in this one special case, where it doesn't. After the change, it will mean element-wise comparison consistently in all cases.
The potential for inadvertent bugs is far greater than what convenience
this redefinition of a very basic operator might offer. Any scenario where
(a == b) != (b == a)
is asking for trouble.
That would be unfortunate, yes, but fortunately it doesn't apply here :-). 'a == b' and 'b == a' currently always return the same thing, and there are no plans to change this - we'll be changing what both of them mean at the same time.
-n
Finally got off my butt and hunted down an example, and it was right under my nose in mplot3d.
lib/python2.7/site-packages/mpl_toolkits/mplot3d/axes3d.py:1094: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future. if self.button_pressed in self._rotate_btn:
self._rotate_btn is the 1d numpy array, and self.button_pressed usually will have an integer (for which mouse button), but could be None if no mouse button was pressed. I have no clue why the "in" operator is triggering this future warning. Is this intentional?
Ben Root
On Sun, Sep 21, 2014 at 10:53 PM, Nathaniel Smith njs@pobox.com wrote:
On 22 Sep 2014 03:02, "Demitri Muna" demitri.muna@gmail.com wrote:
On Sep 21, 2014, at 5:19 PM, Eric Firing efiring@hawaii.edu wrote:
I think what you are missing is that the standard Python idiom for this use case is "if self._some_array is None:". This will continue to
work,
regardless of whether the object being checked is an ndarray or any other Python object.
That's an alternative, but I think it's a subtle distinction that will
be lost on many users. I still think that this is something that can easily trip up many people; it's not clear from looking at the code that this is the behavior; it's "hidden". At the very least, I strongly suggest that the warning point this out, e.g.
"FutureWarning: comparison to `None` will result in an elementwise
object comparison in the future; use 'value is None' as an alternative."
Making messages clearer is always welcome, and we devs aren't always in the best position to do so because we're to close to the issues to see which parts are confusing to outsiders - perhaps you'd like to submit a pull request with this?
Assume:
a = np.array([1, 2, 3, 4]) b = np.array([None, None, None, None])
What is the result of "a == None"? Is it "np.array([False, False, False,
False])"?
After this change, yes.
What about the second case? Is the result of "b == None" ->
np.array([True, True, True, True])?
Yes again.
(Notice that this is also a subtle and confusing point for many users - how many people realize that if they want to get the latter result they have to write np.equal(b, None)?)
If so, then
if (b == None): ...
will always evaluate to "True" if b is "None" or *any* Numpy array, and
that's clearly unexpected behavior.
No, that's not how numpy arrays interact with if statements. This is independent of the handling of 'arr == None': 'if multi_element_array' is always an error, because an if statement by definition requires a single true/false decision (it can't execute both branches after all!), but a multi-element array by definition contains multiple values that might have contradictory truthiness.
Currently, 'b == x' returns an array in every situation *except* when x happens to be 'None'. After this change, 'b == x' will *always* return an array, so 'if b == x' will always raise an error.
On Sep 21, 2014, at 9:30 PM, Benjamin Root ben.root@ou.edu wrote:
That being said, I do wonder about related situations where the lhs of
the equal sign might be an array, or it might be a None and you are comparing against another numpy array. In those situations, you aren't trying to compare against None, you are just checking if two objects are equivalent.
Benjamin, can you give a more concrete example? Right now the *only* time == on arrays checks for equivalence is when the object being compared against is None, in which case == pretends to be 'is' because of this mysterious special case. In every other case it does a broadcasted ==, which is very different.
Right. With this change, using "==" with numpy arrays now sometimes
means "are these equivalent" and other times "element-wise comparison".
Err, you have this backwards :-). Right now == means element-wise comparison except in this one special case, where it doesn't. After the change, it will mean element-wise comparison consistently in all cases.
The potential for inadvertent bugs is far greater than what convenience
this redefinition of a very basic operator might offer. Any scenario where
(a == b) != (b == a)
is asking for trouble.
That would be unfortunate, yes, but fortunately it doesn't apply here :-). 'a == b' and 'b == a' currently always return the same thing, and there are no plans to change this - we'll be changing what both of them mean at the same time.
-n
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sa, 2015-01-31 at 09:02 -0500, Benjamin Root wrote:
Finally got off my butt and hunted down an example, and it was right under my nose in mplot3d.
lib/python2.7/site-packages/mpl_toolkits/mplot3d/axes3d.py:1094: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future. if self.button_pressed in self._rotate_btn: self._rotate_btn is the 1d numpy array, and self.button_pressed usually will have an integer (for which mouse button), but could be None if no mouse button was pressed. I have no clue why the "in" operator is triggering this future warning. Is this intentional?
If I remember right, the in operator just does `np.any(arr == other)` in C-code (which is actually a bit broken, in more ways then just this….). So in this case the FutureWarning is admittingly bogus, since the result won't actually change unless your array really *does* include None, and then I hope you want that change ;).
- Sebastian
Ben Root
On Sun, Sep 21, 2014 at 10:53 PM, Nathaniel Smith njs@pobox.com wrote: On 22 Sep 2014 03:02, "Demitri Muna" demitri.muna@gmail.com wrote: > > > On Sep 21, 2014, at 5:19 PM, Eric Firing efiring@hawaii.edu wrote: > >> I think what you are missing is that the standard Python idiom for this >> use case is "if self._some_array is None:". This will continue to work, >> regardless of whether the object being checked is an ndarray or any >> other Python object. > > > That's an alternative, but I think it's a subtle distinction that will be lost on many users. I still think that this is something that can easily trip up many people; it's not clear from looking at the code that this is the behavior; it's "hidden". At the very least, I strongly suggest that the warning point this out, e.g. > > "FutureWarning: comparison to `None` will result in an elementwise object comparison in the future; use 'value is None' as an alternative."
Making messages clearer is always welcome, and we devs aren't always in the best position to do so because we're to close to the issues to see which parts are confusing to outsiders - perhaps you'd like to submit a pull request with this? > Assume: > > a = np.array([1, 2, 3, 4]) > b = np.array([None, None, None, None]) > > What is the result of "a == None"? Is it "np.array([False, False, False, False])"? After this change, yes. > What about the second case? Is the result of "b == None" -> np.array([True, True, True, True])? Yes again. (Notice that this is also a subtle and confusing point for many users - how many people realize that if they want to get the latter result they have to write np.equal(b, None)?) > If so, then > > if (b == None): > ... > > will always evaluate to "True" if b is "None" or *any* Numpy array, and that's clearly unexpected behavior. No, that's not how numpy arrays interact with if statements. This is independent of the handling of 'arr == None': 'if multi_element_array' is always an error, because an if statement by definition requires a single true/false decision (it can't execute both branches after all!), but a multi-element array by definition contains multiple values that might have contradictory truthiness. Currently, 'b == x' returns an array in every situation *except* when x happens to be 'None'. After this change, 'b == x' will *always* return an array, so 'if b == x' will always raise an error. > > On Sep 21, 2014, at 9:30 PM, Benjamin Root <ben.root@ou.edu> wrote: > >> That being said, I do wonder about related situations where the lhs of the equal sign might be an array, or it might be a None and you are comparing against another numpy array. In those situations, you aren't trying to compare against None, you are just checking if two objects are equivalent. Benjamin, can you give a more concrete example? Right now the *only* time == on arrays checks for equivalence is when the object being compared against is None, in which case == pretends to be 'is' because of this mysterious special case. In every other case it does a broadcasted ==, which is very different. > Right. With this change, using "==" with numpy arrays now sometimes means "are these equivalent" and other times "element-wise comparison". Err, you have this backwards :-). Right now == means element-wise comparison except in this one special case, where it doesn't. After the change, it will mean element-wise comparison consistently in all cases. > The potential for inadvertent bugs is far greater than what convenience this redefinition of a very basic operator might offer. Any scenario where > > (a == b) != (b == a) > > is asking for trouble. That would be unfortunate, yes, but fortunately it doesn't apply here :-). 'a == b' and 'b == a' currently always return the same thing, and there are no plans to change this - we'll be changing what both of them mean at the same time. -n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (5)
-
Benjamin Root -
Demitri Muna -
Eric Firing -
Nathaniel Smith -
Sebastian Berg