A probably bonehead question: is it a known feature of Python that, in a logical operation, & has a higher priority than > or < ? Or is it a quirk of numarray? In [126]: a = arange(25) In [127]: v=(a>9) & (a<13) In [128]: v=a>9 & a<13 --------------------------------------------------------------------------- exceptions.RuntimeError Traceback (most recent call last) /home/swalton/research/foukal/<console> /usr/lib/python2.3/site-packages/numarray/generic.py in __nonzero__(self) 475 476 def __nonzero__(self): --> 477 raise RuntimeError("An array doesn't make sense as a truth value. Use any(a) or all(a).") 478 479 def __copy__(self): RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a).
On 07.02.2005, at 22:56, Stephen Walton wrote:
A probably bonehead question: is it a known feature of Python that, in a logical operation, & has a higher priority than > or < ? Or is it a quirk of numarray?
Operator priorities are part of the Python language and implemented in the parser. Numarray cannot interfere with that. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------
konrad.hinsen@laposte.net wrote:
Operator priorities are part of the Python language and implemented in the parser. Numarray cannot interfere with that.
In Python, of course, & is a bitwise AND, logical and being the literal word "and". & has lower operator priority than <, but "and" has a higher one. I think the real problem is that "and" doesn't work on numarray bool arrays in an element-wise fashion as I expected; perhaps it should? Otherwise constructs like v = A>10 and A<15 have to be written as v1=A>10; v2=A<15; v=v1&v2 or as v = (A>10) & (A<15)
Stephen Walton wrote:
konrad.hinsen@laposte.net wrote:
Operator priorities are part of the Python language and implemented in the parser. Numarray cannot interfere with that.
In Python, of course, & is a bitwise AND, logical and being the literal word "and". & has lower operator priority than <, but "and" has a higher one.
I think the real problem is that "and" doesn't work on numarray bool arrays in an element-wise fashion as I expected; perhaps it should? Otherwise constructs like
Currently, I don't think Python allows "over-riding" the "and" operation. It only works on the truth or falseness of the objects. Therefore, I don't think it is possible to make it work as you'd expect unless Python is changed.
Travis Oliphant wrote:
Currently, I don't think Python allows "over-riding" the "and" operation. It only works on the truth or falseness of the objects. Therefore, I don't think it is possible to make it work as you'd expect unless Python is changed.
You are correct: In [12]: dis.dis(compile('x and y','<console>','eval')) 0 0 LOAD_NAME 0 (x) 3 JUMP_IF_FALSE 4 (to 10) 6 POP_TOP 7 LOAD_NAME 1 (y) >> 10 RETURN_VALUE There is no method you can hook in there. Compare this to '&': In [13]: dis.dis(compile('x & y','<console>','eval')) 0 0 LOAD_NAME 0 (x) 3 LOAD_NAME 1 (y) 6 BINARY_AND 7 RETURN_VALUE Cheers, f
Fernando Perez wrote:
Travis Oliphant wrote:
Currently, I don't think Python allows "over-riding" the "and" operation.
You are correct:
What I want must be possible somehow, though: In [6]: a=[True,False] In [7]: b=[False,False] In [8]: a and b Out[8]: [False, False] In [9]: a or b Out[9]: [True, False] In [10]: not a Out[10]: False So, standard Python does apply logical "and" and "or", but not "not", element by element to a list or tuple of booleans. Is there no way to get access to this for a numarray bool array? Can we get Guido to change "not" to also operate element-wise on a list of booleans? I have a really common task in MATLAB I want to have work here. In MATLAB I can extract the overlapping good values from two data sets by doing f = data1 ~= flagval & data2 ~= flagval; or even more compactly, since I use IEEE NaN for flagval f = ~isnan(data1) & ~isnan(data2); Right now in numarray I have to use f = (data1 != flagval) & (data2 != flagval) with the extra parentheses and, confusingly, using bitwise AND instead of logical AND. Even if I use ieeespecial nan as flagval, there's no "notisnan(data1)" and no way to say "!isnan(data1)" Of course, masked arrays are another option for this kind of usage but they have their own difficulties: matplotlib does not at present allow one to plot from masked arrays, for example.
Stephen Walton wrote:
Fernando Perez wrote:
Travis Oliphant wrote:
Currently, I don't think Python allows "over-riding" the "and" operation.
You are correct:
What I want must be possible somehow, though:
In [6]: a=[True,False]
In [7]: b=[False,False]
In [8]: a and b Out[8]: [False, False]
In [9]: a or b Out[9]: [True, False]
In [10]: not a Out[10]: False
So, standard Python does apply logical "and" and "or", but not "not", element by element to a list or tuple of booleans. Is there no way to get access to this for a numarray bool array? Can we get Guido to change "not" to also operate element-wise on a list of booleans?
I think you are misunderstanding something. Python does not "apply logical "and" and "or", but not "not", element by element to a list or tuple of booleans", as you say. Look again at the decompiled bytecode: In [2]: dis.dis(compile('a and b','<>','eval')) 0 0 LOAD_NAME 0 (a) 3 JUMP_IF_FALSE 4 (to 10) 6 POP_TOP 7 LOAD_NAME 1 (b) >> 10 RETURN_VALUE What it does is execute the JUMP_IF_FALSE bytecode, which is mapped to the __nonzero__ special method. From http://docs.python.org/ref/customization.html: __nonzero__( self) Called to implement truth value testing, and the built-in operation bool(); should return False or True, or their integer equivalents 0 or 1. When this method is not defined, __len__() is called, if it is defined (see below). If a class defines neither __len__() nor __nonzero__(), all its instances are considered true. Now, lists don't actually have this method, so a call to __len__ is made, and any non-empty list is considered true. The return value is then whatever is on the stack, which will be either be a, or b if a tested false: In [6]: a Out[6]: [True, False] In [7]: a and 0 Out[7]: 0 In [8]: a and 1 Out[8]: 1 In [9]: a and 'hello' Out[9]: 'hello' The point is that and/or only operate on one operand at a time, with unary functions/methods. They do NOT call a binary function, the way & does: In [5]: dis.dis(compile('a & b','<>','eval')) 0 0 LOAD_NAME 0 (a) 3 LOAD_NAME 1 (b) 6 BINARY_AND 7 RETURN_VALUE This means that certain things which can be done with & are fundamentally impossible with 'and', since you never have both arguments in your hands at the same time. Bytecode analysis can be really useful to understand this kind of subtle issues. I can't think of any language where this approach is as easy to do and friendly as in python. Cheers, f
Fernando Perez wrote:
Now, lists don't actually have this method, so a call to __len__ is made, and any non-empty list is considered true. The return value is then whatever is on the stack, which will be either be a, or b if a tested false: ^^^^^^
That should have read TRUE, of course, for an 'a and b' call. For 'a or b', false is the right word. Sorry about the confusion. Cheers, f
Fernando Perez wrote a good deal clarifying how Python's logical operators work, for which I'm grateful. One brief comment:
Bytecode analysis can be really useful to understand this kind of subtle issues. I can't think of any language where this approach is as easy to do and friendly as in python.
Pardon me, but this sounds a bit like looking at the assembly language Fortran generates in order to understand the subtleties of that language :-) . I know people who used to do that, and were proud of it, but I'd rather the language itself didn't behave in surprising ways. At least in Fortran, 'a .and. b' only made sense if a and b were both of type LOGICAL; in fact, it was a compile time error if they weren't. Anyway, we've wandered afar from the main point: what is the best way to duplicate the functionality of the following MATLAB snippet: f = ~isnan(data1) & ~isnan(data2); plot(x,data1(f),x,data2(f)) ?
Stephen Walton wrote:
Fernando Perez wrote a good deal clarifying how Python's logical operators work, for which I'm grateful. One brief comment:
Bytecode analysis can be really useful to understand this kind of subtle issues. I can't think of any language where this approach is as easy to do and friendly as in python.
Pardon me, but this sounds a bit like looking at the assembly language Fortran generates in order to understand the subtleties of that language :-) . I know people who used to do that, and were proud of it, but I'd
Oh, and it is :) It's just that it's so immediate to do it in python, that it actually becomes a normal approach. I've often used it to clarify behavior or performance issues to myself. I'm not proud of it, it's just a useful trick which can come in handy in certain cases.
rather the language itself didn't behave in surprising ways. At least in Fortran, 'a .and. b' only made sense if a and b were both of type LOGICAL; in fact, it was a compile time error if they weren't.
You can argue that this is a wart (I don't think it is), but it was certainly done by design: http://docs.python.org/ref/bitwise.html#l2h-423 http://docs.python.org/ref/Booleans.html#l2h-440 'and' is meant to provie truth checking, with (critically important in many applications) short-circuit behavior. '&' is a binary operation (hence necessarily non short-circuiting, due to python's argument evaluation rules), most often used for bitwise manipulations. It seems clear to me that they both have a valid use.
Anyway, we've wandered afar from the main point: what is the best way to duplicate the functionality of the following MATLAB snippet:
f = ~isnan(data1) & ~isnan(data2); plot(x,data1(f),x,data2(f))
I don't have a clean answer for this one, sorry. Best, f
Stephen Walton wrote:
Anyway, we've wandered afar from the main point: what is the best way to duplicate the functionality of the following MATLAB snippet:
f = ~isnan(data1) & ~isnan(data2); plot(x,data1(f),x,data2(f))
Strictly speaking there is no operator equivalent (you can always use the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) ) (using isnan for the sake of illustration) I'm not sure if this is wise advice, I point out to people that generally speaking, the arguments to the and operator are typically boolean arrays and for numarray anyway (I forget if it can be different for Numeric) you can simply use the '&' bitwise operator to the same effect. No doubt that such use will eventually be misinterpreted and someone will provide a non-boolean argument resulting in confusion (perhaps a mandatory comment is needed in such cases) No doubt this is a wart from array user's perspectives. Perry
On Wed, 2005-02-09 at 13:48 -0500, Perry Greenfield wrote:
Strictly speaking there is no operator equivalent (you can always use the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) ) (using isnan for the sake of illustration)
OK, how did I miss in "Learning Python" and the numarray docs that ~ is bitwise NOT and works on numarray bool arrays? In point of fact, "~isnan(data1) & ~isnan(data2)" works fine. I kept trying !isnan(data1) which I gather makes no sense. Thanks for the patient help, all.
Anyway, we've wandered afar from the main point: what is the best way to duplicate the functionality of the following MATLAB snippet:
f = ~isnan(data1) & ~isnan(data2); plot(x,data1(f),x,data2(f))
I know this has been talked about before on this list, but only in bits and pieces. Can I summarize the way I understand the situation and hopefully others can correct me? I use MATLAB and Numeric often; I have rarely used Numarray. Summary: For people moving from Matlab to Numeric/Numarray, the recommendation is to use Numeric's "&" in place of Matlab's "&" to combine logical arrays for indexing. Different behavior results if the inputs have entries > 1, and you have to put parenthesis around any inputs that use relational operators. Also, one should use Numeric's "and" in place of Matlab's "&&". Details: In MATLAB (R14SP1) there are three logical "and" operators: "&": a binary operator which performs element-by-element logical AND. For its purposes, 0 is FALSE and every other numerical value is TRUE. "&&": a binary operator which performs logical AND, but does short-circuiting, and only works with scalar elements (length-1 arrays) "bitand()" : a function which takes two integers and does bitwise-and on their binary representations http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/ch12_n11.h... I think that these map onto the python operators in the following way: MATLAB Python/Numeric ------ -------------- && and & ufunc logical_and() bitand & Some inconsistencies in behavior (between Numeric and Matlab) occur when one uses Numeric's "&" like MATLAB "&". I can think of: * Non-logical {0,1} inputs: Numeric's output is an integer, the bitwise and of the inputs, Matlab's output is a logical array, either 1 or 0. * Precedence: Numeric's "&" operator is higher prec than its relational operators, MATLAB is the reverse And an inconsistency between MATLAB's "&&" and Numeric's "and": * MATLAB returns an error when the arguments to && are not logical scalars, Numeric returns the second argument as the expression's value. (but Numarray behaves like MATLAB) Does that seem right? Regards, Mark ---------------------------------------------------------------- Matlab example:
a = [0,1,2]; b = [1,1,63]; a & b ans = 0 1 1 a && b ??? Operands to the || and && operators must be convertible to logical scalar values. bitand(a,b) ans = 0 1 2 a > 1 & b > 1 ans = 0 0 1 (a > 1) & (b > 1) ans = 0 0 1
---------------------------------------------------------------- Numeric example: Python 2.3.4 (#2, Sep 24 2004, 08:39:09) [GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import Numeric as N a = N.array([0,1,2]); b = N.array([1,1,63]); N.logical_and(a,b) array([0, 1, 1]) a & b array([0, 1, 2]) a and b array([ 1, 1, 63]) a > 1 & b > 1 array([0, 0, 0]) (a > 1) & (b > 1) array([0, 0, 1])
---------------------------------------------------------------- Numarray example:
import numarray as na a = na.array([0,1,2]) b = na.array([1,1,63]) na.logical_and(a,b) array([0, 1, 1], type=Bool) a and b RuntimeError: An array doesn't make sense as a truth value. Use sometrue(a) or alltrue(a). a & b array([0, 1, 2])
---------------------------------------------------------------- On Wed, Feb 09, 2005 at 10:11:56AM -0800, Stephen Walton wrote:
Fernando Perez wrote a good deal clarifying how Python's logical operators work, for which I'm grateful. One brief comment:
Bytecode analysis can be really useful to understand this kind of subtle issues. I can't think of any language where this approach is as easy to do and friendly as in python.
Pardon me, but this sounds a bit like looking at the assembly language Fortran generates in order to understand the subtleties of that language :-) . I know people who used to do that, and were proud of it, but I'd rather the language itself didn't behave in surprising ways. At least in Fortran, 'a .and. b' only made sense if a and b were both of type LOGICAL; in fact, it was a compile time error if they weren't.
?
SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Travis Oliphant wrote:
Stephen Walton wrote:
konrad.hinsen@laposte.net wrote:
Operator priorities are part of the Python language and implemented in the parser. Numarray cannot interfere with that.
In Python, of course, & is a bitwise AND, logical and being the literal word "and". & has lower operator priority than <, but "and" has a higher one.
I think the real problem is that "and" doesn't work on numarray bool arrays in an element-wise fashion as I expected; perhaps it should? Otherwise constructs like
Currently, I don't think Python allows "over-riding" the "and" operation. It only works on the truth or falseness of the objects. Therefore, I don't think it is possible to make it work as you'd expect unless Python is changed.
There is __nonzero__ which could be treated as allTrue or someTrue, the former is probably the better. Colin W.
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
participants (7)
-
Colin J. Williams -
Fernando Perez -
konrad.hinsen＠laposte.net -
Mark Histed -
Perry Greenfield -
Stephen Walton -
Travis Oliphant