RE: Possible bug (was Re: numpy, overflow, inf, ieee, and rich comparison)
On the issue of whether Python should ignore over/underflow on IEEE-enabled platforms: [Tim Peters]
It can be argued that on IEEE enabled systems the proper thing to do for overflow is simply return Inf. Raising exception is WRONG. See below. [Guido van Rossum]
Incidentally, math.exp(800) returns inf in 1.5, and raises OverflowError in 2.0. So at least it's consistent.
That is not consistent at all. Suppose I'm plotting the curve f(x) where x include some singular points of f. In the first case the plot works with some portion of the curve clipped. In the second case it bombs. [Tim Peters]
[Guido van Rossum]
If using ieee is as simple as setting such a flag, there is no reason at all not to use it. Here are some more examples: Suppose you have done hours of computation on a problem. Just as you are about to get the result, you get an exception. Why? Because the residual error is too close to zero. Suppose you want to plot the curve of Gausian distribution. Oops, it fails. Because beyond a certain region the value is near zero. With these kinds of problems, vectorized numerical calculation becomes nearly impossible. How do you work in such an environment? You have to wrap every calculation in a try/except structure, and whenever there is an exception, you have to revert to elementwise operations. In practice this simply means Python would not be suitable for numerical work at all. What about the other way round? No problem. It is easy to write functions like isNaN, isInf, etc. With these one can raise exceptions in any place one want. It is even possible to raise exceptions if a matrix is singular to a certain precision, etc. The key point to observe here is that most numerical work involve more than one element. Some of them may be out of mahcine bounds but the whole thing could still be quite meaningful. Vice versa it is also quite possible that all elements are within bounds while the whole thing is meaningless. The language should never take over or subvert decisions based on numerical analysis. [Tim Peters]
Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and current code certainly relies on detecting overflows in math functions.
As Guido observed ERANGE is not generated with ieee, even for overflow. So it is the same behavior as 1.5.2. Besides, no correct numerical code should depend on exceptions like this unless the machine is incapable of handling Inf and NaN. Even in the cases where you do want to detect overflow, it is still wrong to use exceptions. Here's an example: x*log(x) approaches 0 as x approaches 0. If x==0 then log(x)==-Inf but 0*-Inf==NaN, not what one would want. But exception is the wrong tool to hangle this, because if x is an array, some of its element may be zero but other's may not. The right way to do it is something like def entropy(probability): p = max(probability, 1e-323) return p*log(p) [Tim Peters]
In no case can you expect to see overflow ignored in 2.0.
You are proposing a dramatic change from the behavior of 1.5.2. This looks like to me to need a PEP and a big debate. It would break a LOT of numerical computations. [Thomas Wouters]
The patch as described seems to be based on flawed thinking. The numbers Inf and NaN are always necessary. The -lieee could only be unnecessary if the behavior is the same as IEEE. Obviously it isn't. So I second Thomas's suggestion. [Tim Peters]
[Guido van Rossum]
What is the reason to do this? It looks like intetionally subverting ieee even when it is available. I thought Tim meant that only logistical problems prevent using ieee. If you do choose this route, please please please ignore ERANGE entirely, whether return value is zero or not. The only possible way that ERANGE could be useful at all is if it could be set independently for each element of an array, and if it behave as a warning instead of an exception, ie. the calculation would continue if it is not caught. Well, then, Inf and NaN serve this purpose perfectly. It is very reasonable to set errno in glibc for this; it is completely unreasonable to raise an exception in Python, because exceptions cannot be set to individual elements and they cannot be ignored. Huaiyu -- Huaiyu Zhu hzhu@users.sourceforge.net Matrix for Python Project http://MatPy.sourceforge.net
I don't have time to follow in detail this thread about changed behavior between versions. These observations are based on working with hundreds of code authors. I offer them as is. a. Nobody runs a serious numeric calculation without setting underflow-to-zero, in the hardware. You can't even afford the cost of software checks. Unfortunately there is no portable way to do that that I know of. b. Some people use Inf but most people want the code to STOP so they can find out where the INFS started. Otherwise, two hours later you have big arrays of Infs and no idea how it happened. Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
Thanks, Paul! This behavior has always been what I wanted Python to do (even though it's not always what Python did, depending on the platform) and also what Tim's proposed patch will implement for the specific case of math.exp() (and other math functions that may underflow or overflow), on most reasonable platforms. There are still lots of places where the platform gives Python no choice of creating NaN and Inf, and because there's no platform-independent way to test for these, they are hard to avoid in some cases; but eventually, Tim will find a way to root them out. And for people like Huaiyu, who want to see Inf, there will (eventually) be a way to select this as a run-time option; and ditto for whoever might want underflow to raise an exception. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Paul Dubois]
Amen.
$ /usr/bin/python Python 1.5.2 (#1, Sep 17 1999, 20:15:36) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
This was sane behavior. Are we saying that Python 2.0 has invented something better than IEEE 754? [Guido van Rossum]
Guido, with due respect to your decisions on Python issues, I simply have to fight this one. It is one thing to accomodate for naive users, but it is another to dumb down every one else. Case 1. Someone writes a flawed numerical routine. Two hours later he finds his array filled with Inf and NaN. Case 2. Someone writes a perfect numerical routine. Two hours later he gets an exception, because the error is near zero. Solution for case 1. Use better algorithm. Use better error control. Raise exceptions when error is too large. These are proper solutions. They are easy and efficient to implement. They are needed anyway - If something's wrong, you want to raise exceptions far earlier than Inf, certainly before you get arrays filled with elements like 1e300. Solution for case 2. Almost impossible. The division between under- and over-flow is artificial. What about 1/x or similar functions? The only way to work on such a platform is to abandon vectorized computation.
I can understand that exceptions are the only available choices if IEEE is not available. But is there a compelling reason that Python should behave "better" than IEEE when it's in fact available? If the reason is to protect naive users, I can think of several responses: 1. For people doing one-off interactive work, returning Inf is in fact more informative. 2. For users iterative numerical computations, they need to be educated about error control. Otherwise they won't get corrent results anyway. 3. For really serious work, we could provide good numerical modules so that they don't need to write themselves. To make this happen fast the fewer debacles like this one the better. Case in point: Someone asked for regession modules two weeks ago. I was trying to convert my old matlab programs, which only took a few hours. But I wasted a week of (spare) time fighting for some mysterious "overflow". Turns out that a Guassian is near zero when it's far from center, and Python does not like it. In practice, Inf may be generated more often as a proper value than by mistake. This is not an issue about whether someone "prefers" Inf or exception. It is about whether there is a choice to do proper computation. Returning Inf does not prevent someone to raise exception. Raising exception automatically prevents perfect algorithms to work properly. As Kevin has volunteered to help with IEEE implementation and made a plan, is there a strong reason to drop IEEE for Linux in 2.0? If there is insufficient time to carry out his plan, wouldn't it be prudent to keep things as they were in 1.5.2? Huaiyu
First: I haven't followed this thread from the beginning -- only the last ten or so posts. Second: One reason I didn't follow it from the start is that I'm not doing any heavy numerical stuff in Python. I've been using Lisp for that, and use Python more for string/symbolic or GUI hacking. But, if I was going to do the sort of numerical stuff I now do in Lisp in Python, I would agree with Huaiya Zhu. I do a lot of vectorized operations on what are often independent samples. If some of the numbers overflow or underflow, that just represents an outlier or bad sample. I don't want it to blow off the whole pipeline of operations on the other data points in the vector -- they are independent of the bad points. In my case, it's not that these are lengthy calculations. It's that they are interactive and tied to immediate graphical representations. If there are strange zero's or infinities in the result, there is (or should be) a way to backtrack and investigate. ( That's why people want interactive and graphical regression analysis and modeling tools! ) The idea that your calculation should blow up and you should check it and resubmit your job sounds just so ancient-20th-century- Fortran-JCL-and-punched-cards-technology! ---| Steven D. Majewski (804-982-0831) <sdm7g@Virginia.EDU> |--- ---| Department of Molecular Physiology and Biological Physics |--- ---| University of Virginia Health Sciences Center |--- ---| P.O. Box 10011 Charlottesville, VA 22906-0011 |--- "All operating systems want to be unix, All programming languages want to be lisp."
Long-running jobs are still with us, even there's neither Fortran nor JCL in them. And for these applications, stopping is better than going on with nonsense values. On the other hand, as you point out, exceptions for math errors are a bit of a pain for interactive work. So how about making this a run-time option? I'd choose exceptions by default and Infs and Nans by specifying a command-line option, but there are certainly others who would prefer it the other way round. What matters most to me is that the choice is possible somehow. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
On Thu, 12 Oct 2000 hinsen@dirac.cnrs-orleans.fr wrote:
I agree entirely! Maybe I was being a bit too glib, but I didn't mean to imply that wanting it to halt or throw an exception on errors is wrongheaded. I just wanted to make sure the counter-case to what Paul was saying also got heard: Yes-- underflows or infinities where they aren't expected are usually a sign that something is very wrong somewhere. But in the case where the vector holds independent observations or data points, then usually what it means is that there's something wrong with *that* data point -- miscallibrated or mislabeled -- but no reason not to complete the calculations for all of the other points. Scaling or doing projections for interactive graphics is another case where bad points are often better than throwing an exception. ( And it's a pain to have to remember to lambda wrap all the function calls with some sort of guard when you'ld be happy to get NaNs. ) I also mostly agree with Tim, except that I'm not sure that bad or incomplete ieee support is always better than none at all. ---| Steven D. Majewski (804-982-0831) <sdm7g@Virginia.EDU> |--- ---| Department of Molecular Physiology and Biological Physics |--- ---| University of Virginia Health Sciences Center |--- ---| P.O. Box 10011 Charlottesville, VA 22906-0011 |--- "All operating systems want to be unix, All programming languages want to be lisp."
[Steven D. Majewski]
This is true, because having Python is better than not having Python, and in having Python you indeed have bad *and* incomplete 754 support on every 754 platform it runs on. Even better, it's a subset of damaged 754 support unique to each platform whose details can't be guessed or documented in advance of trying it exhaustively to see what happens(*), and a subset that can change delightfully across releases, compiler upgrades, library upgrades and seemingly irrelevant minor config changes. So if bad or incomplete IEEE support is good, Python is-- here as elsewhere --the King of Languages <wink>. Every step of this dance is thoroughly predictable. In this case, I'm doing my darnedest to nudge Python its very first step towards *real* 754 support, and getting dumped on for it by a 754 fan. Simultaneously, the "old guard" defends their lifestyle against new-fangled ideas <wink>, asking for protections unaware that 754 *requires* they get a better form of the protections they seek than they've dreamed of so far. It appears that nobody on either side has actually read the std, and I've become the very 754 Storm Trooper I used to despise. Computer life is great <wink>. all-it's-missing-is-variety-ly y'rs - tim (*) Quiz: *if* you can manage to trick Python into creating a NaN on your particular 754 platform, does the Python expression NaN == 1.0 return true, false, or raise an exception? Answer before you try it. Then try it on enough 754 platforms until you give up trying to guess in advance. NaN == NaN is predictable in Python, and is the one 754 feature Python guarantees won't work correctly on any 754 platform (although I've heard that it loses this predictability when run using NumPy's flavor of comparisons instead of core Python's).
This is a bit off-topic, but I noticed that there are lots of experts here and I have a question about IEEE-754 that has been bugging me for some time and which I was not able to get a good answer for on the USENET newsgroups. I am hoping that one of the experts in this list will take pity on my ignorance and answer what should be a simple question. Under Microsoft Visual C++, the Standard C++ library's numeric_limits<float>::signaling_NaN() and numeric_limits<double>::signaling_NaN() both return -INF values. The numeric_limits<float>::quiet_NaN() and numeric_limits<double>::quiet_NaN() both return -IND. I have not been able to determine whether this is "proper" behavior or whether it is another example of Microsoft ignoring standards? I would have thought that one of the two should return NaN. I certainly can't for the life of me figure out why someone would call INF a NaN, but as I have said, I'm pretty ignorant. Right now, if I want to use NaN's in my C++ code (e.g., to initialize newly allocated memory blocks) I build NaNs thus:
but would prefer something a little less bit-tweaky. Microsoft's documentation that this is the way they intend their library to work may be found at: http://msdn.microsoft.com/library/devprods/vs6/visualc/vclang/sample_Members... Thanks, Jonathan ============================================================================= Jonathan M. Gilligan jonathan.gilligan@vanderbilt.edu The Robert T. Lagemann Assistant Professor Office: 615 343-6252 of Living State Physics Lab (FEL) 343-7580 Dept. of Physics and Astronomy, Box 1807-B Fax: 343-7263 6823 Stevenson Center Vanderbilt University, Nashville, TN 37235 Dep't Office: 322-2828
Incomplete vs. non-at-all IEEE 754 support is a non argument. If you have full IEEE support (which it seems everyone here thinks would be good, but difficult), it is clear what you have. If not, you are not consistent with a standard, and therefor making up your own. That being the case, it's a matter of what we want the Python standard to be. I, for one think that NaN and Inf are fantastic!!! I think the best argument for them here is that it is almost impossible to do a lot of array based calculations without them, and you can't do serious number crunching in Python without array based calculations. I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754 support, as I don't think you can change the behaviour, but you do get a consistent result across platforms (including non-iee machines like the Cray?---I have no idea). I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception. This really does render Python essentially useless for Numerics for me. I suspect all those number crunchers that want an exception rather than an Inf are NOT using array-oriented languages. My goal is to dump MATLAB for Python, but this may prevent me from doing that. Does anyone know what other array-oriented languages use? I know what MATLAB does:
Hey! I like that! Python is dynamic, why can't we just get the actual answer to sqrt(-1), without using cmath, that always returns a complex ? sorry, other subjet, not meant to be raised at the moment.
Here we get a warning, but also a result that won't crash the computation.
So comparing two NaNs yields a false, as it should, and though Python won't do it now, it could. One thing that a numerical routine should NEVER do is give an incorrect answer. No answer is often bad, but an incorrect one is much worse. NaN == NaN must return false. Does anyone know what FORTRAN 90 specifies (if anything)? What other array-oriented languages are there? I think what MATLAB does is what Tim is calling "bad *and* incomplete 754 support" Incomplete is surely true, "bad" is clearly a matter of opinion. It seems we have a variety of users of numerics in Python, that I break into three broad catagories: Casual users: mostly doing non-numeric stuff, with the occasional calculation: This group could use any non-cryptic handling of over/underflow, it just doesn't matter. Mid-level number crunchers: This is the group I put myself in. We crunch a lot of numbers, really like the array semantics (and speed) of NumPy, and are doing things like graphing functions, statistical calculations, etc. We have probably only taken one of two (at most) numerical analysis courses, and many don't know what the heck IEE 754 is. (The one Numerical Analysis course I did take, happened to be with Kahan, which is why I know what it is) For this group, the incomplete IEEE support is probably the best way to go. We're more likely to be annoyed by our whole computation stopping because of one bad data point, than we are to be pissed off that it didn't stop when it started to compute garbage. Hard Core Number crunchers: These are the folks that do things like write optimised routines for particular architectures, adn teach numerical analysis courses. These folks want either FULL IEEE, or plain old "classic" overflow and underflow errors, that they can handle themselves. Do these folks really use Python as other than a glue language? Are they really doing massive number crunching in Python itself, rather than in C (or whatever) extensions? If so, I'd be surprised is they didn't find Huaiyu's argument compelling when doing array based calculations. Tim pointed out that Python was not designed with 754 in mind, so for 2.0, what he is doing is probably our best bet, but it seems to not be the best ultimate solution, I hope using NaN and Inf will be considered for future versions, even if it is incomplete 754 compliance. Note: if the ultimate goal is REAL IEEE 754 compliance, is it possible without custom re-compiling your own interpreter? If so, is there any chance that it will come any time soon (2.1 ?) , so we don't have to have this discussion at all? Thanks for all of your work on this, Python just keeps getting better!! -Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------
Chris Barker writes:
Hey! I like that! Python is dynamic, why can't we just get the actual answer to sqrt(-1), without using cmath, that always returns a complex ?
You have to import the sqrt function from somewhere. Either you import it from math or from cmath, depending on which flavor you need. Anybody sophisticated enough to know what complex numbers are, and sophisticated enough to want to get complex values as a result of a calculation, should be sophisticated enough to be able to type "cmath".
Does anyone know what other array-oriented languages use? I know what MATLAB does:
I'm an Interactive Data Language or IDL user (the Univ of Wisconsin's Space Science Ceter is split down the middle between this and MatLab, but python/numpy is definitely on the increase). As you can see from results below, like MatLab over/under-flows in IDL are reported but do not stop execution. This is the best (only) possible scenario for an interactive arrays and visualization environment. IDL> print, exp(-777) 0.00000 % Program caused arithmetic error: Floating underflow IDL> print, exp(777) Inf % Program caused arithmetic error: Floating overflow IDL> print, sqrt(-1) -NaN % Program caused arithmetic error: Floating illegal operand IDL> print, 54/0 54 % Program caused arithmetic error: Integer divide by 0
I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754
MATLAB is fine for simple interactive data processing, and its behaviour is adapted to this task. I don't think anyone would use MATLAB for developing complex algorithms, its programming language isn't strong enough for that. Python is, so complex algorithms have to be considered as well. And for that kind of application, continuing a calculation with Infs and NaNs is a debugging nightmare.
I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception.
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
For the same reason that makes 2/3 return zero instead of a float division result. Like C or Fortran, Python treats integers, floats, and complex numbers as different data types. And the return type of a function should depend only on the types, but not the values, of its parameters (for consistency, not because of any implementational limitation). So sqrt(a) for a of type float can either always return a float (math, Numeric) and crash for negative arguments, or always return a complex (cmath). The "right" solution, in my opinion, would be to have a single "number" type of which integers, float, and complex numbers are simply different internal representations. But such a change cannot be introduced without breaking a lot of code. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
[cbarker@jps.net]
I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754 ...
[Konrad Hinsen]
A non-constructive (because futile) speculation: the first time I saw what 754 was intending to do, my immediate reaction was "hmm -- the exponent field is too narrow!". With a max (double) val in the ballpark of 1e300, you get an infinity as soon as you square something as small as 1e150, and once you get a few infinities, NaNs are sure to follow (due to Inf-Inf and Inf/Inf). The dynamic range for 754 singles is much smaller still. There's no doubt about Cray arithmetic being hard to live with, but while Mr. Cray didn't worry about proper rounding, he did devote 15 bits to Cray's exponent field (vs. 11 for 754). As a result, overflows were generally a non-issue on Cray boxes, and *nobody* complained (in my decade there) about Cray HW raising a fatal exception if one occurred. In return, you got only 48 bits of precision (vs. 53 for 754). But, for most physical problems, how accurate are the inputs? 10 bits on a good day, 20 bits on a great day? Crays worked despite their sloppy numerics because, for most problems to which they were applied, they carried more than twice the precision *and* dynamic range than the final results needed.
I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception.
I'd usually trace back and try to figure out how it got "bad" to begin with ...
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
Does NumPy use the fpectl module? I suppose not. LLNL contribued that code, but we hear very little feedback on it. It arranges to (in platform-dependent ways) convert the 754 overflow, divide-by-0 and invalid-operation signals into Python exceptions. In core Python, this is accomplished at the extraordinary expense of doing a setjmp before, and a function call + double->int conversion after, every single fp operation. A chief problem is that SIGFPE is the only handle C gives us on fp "errors", and the C std does not allow returning from a SIGFPE handler (the result of trying to is undefined, and indeed varies wildly across platforms); so if you want to regain control, you have to longjmp out of the handler. The NumPy implementation could use the PyFPE_START_PROTECT and PyFPE_END_PROTECT macros to brackete entire array operations, though, and so pay for the setjmp etc only once per array op. This is difficult stuff, but doable.
You know I'm in general agreement with you on this one, but I have to draw a distinction here: Guido thinks that 2/3 returning 0 was a design mistake, but not that math.sqrt(-1) raising an exception is a mistake. Most Python users won't know what to do with a complex number, so it's "an error" to them. I would like to view this in P3K (if not earlier) as being akin to 754 exceptions: some people are delighted to have 1e300**2 return +Inf, while others want to see OverflowError instead, and still others want to see +Inf *and* have a sticky flag set saying "an overflow occurred". We could treat f(x) (for f == sqrt and everything else) the same way wrt to a new ComplexResultFromNonComplexInputsError: define "non-stop" complex results, let the user choose whether to do nonstop or raise an exception, and a supply a sticky flag saying whether or not any complex results were generated from non-complex inputs.
The current distinction between ints and longs (unbounded ints) should also get swallowed by this. A way to get from here to there is especially irksome at the C API level, since, e.g., many C API functions pass Python ints as C longs directly. A smaller number pass Python floats as C doubles directly. it's-wonderful-that-p3k-will-solve-everything<wink>-ly y'rs - tim
No. Perhaps it should, but it doesn't make sense unless fpectl works on a large number of platforms. I confess I have never looked at it. I want my code to be portable, so I don't even consider using packages that aren't.
Well, that would be a good occasion to learn about complex numbers! I remember having learned about generalized inverses by using APL in high school (in a very similar way: I was amazed that it could invert non-square matrices), and that turned out to be very useful knowledge later on. Perhaps that's a topic for the EDU-SIG... Anyway, I don't care what math.sqrt(-1) does, but I would definitely prefer Numeric.sqrt(-1) to return a complex result. And I think that someone who uses NumPy has probably heard about complex numbers.
There is, however, one difference: in properly debugged production code, there should be no overflow situations, so it doesn't matter much how they are treated. Complex number can (do!) occur in production code, but there could also be production code relying on exceptions for sqrt(-1) (e.g. for input error checking). Therefore a program using several libraries might be impossible to run with either setting. Since this concerns only the math module, I'd prefer to keep separate module versions for the two cases, which can be used in parallel.
I don't see the problem in that direction, it's rather C API functions that *return* numbers in C data types that would be difficult to adapt. But then, why should be C API not be allowed in P3K? it's-wonderful-that-anything-may-be-changed-in-p3k'ly Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
And from Konrad Hinsen
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
Exactly. This was Huaiyu's point. The problem is that for your code to be usable for folks other than yourself, you'd have to have a lot of checks in there, and essentially revert to elementwise code, which would kill you. With the addition of the occasional "isnan" and something like Matlab's "any" ( "any(isnan(A))" returns true if any of the elements of A are NaNs) you could set up your code to raise an exception in the middle of computation. If, on the other hand the Code definiately raises an exception, than you are stuck with a lot of elementwise coding.
Exactly!!! ("the best (only) possible scenario" part)
This, however, is wierd!! 54/0 == 54 ??? I'd much rather see a NaN or Inf here!
People do some pretty complex algorith develpment with MATLAB. I'm also no so sure about "continuing a calculation with Infs and NaNs is a debugging nightmare" I'm don't think it's any more of a nighmare than having your calculation on an entire array stop because of one Inf. It's just a different trade off. Sprinkle a few "isnan"'s in there and and you can get the behaviour you want, while not forcing it on all calculations. Of course, the best option is to have it switchable, but that may prove to be very difficult. Are people doing the kind of numeric programming with "complex algorithms" using Python that Konrad refers to? More importantly, how many people?
Guido's philosophy is clearly that Python defaults should be geared to "Most Python users". I agree, and as I wrote in an earlier post, the only users for whom the "exception raising only" option is best are the users Konrad refers to as writing "complex algorithms". I would argue that those users are few, and the group most able to deal with a less-that-optimum system. Maybe we can have true 754 compliance in Py3k, and we can all be happy!! -Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------
At least one: me.
It's difficult to say what "most Python users" want; it's not a static community. I'd say the basic principle is "no bad surprises". 2/3 == 0 is a bad surprise for anyone who knows elementary math but is not familiar with certain programming languages, especially since the program goes on silently with a "wrong" intermediate result.
Maybe we can have true 754 compliance in Py3k, and we can all be happy!!
Py3k forever! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
[Huaiyu Zhu]
754 allows users to choose the behaviors they want. Any *fixed* policy is not 754. So long as we have to fix a single policy, yes, I believe Guido (& know I) would say that raising OverflowError on exp(777), and silently returning 0 on exp(-777), and raising ValueError on sqrt(-1) (*not* OverflowError, as shown above), is indeed better than the 754 default behaviors. And 2.0 will do all three of those (& I just checked in the code to make that happen). About the sqrt example above, that's neither 754 behavior nor sane behavior: default 754 behavior on sqrt(-1) is to return a NaN and keep on going. I'm pretty sure that's what glibc linked w/ -lieee actually does, too (if it doesn't, glibc -lieee is violating 754). That 1.5.2 raised an OverflowError instead is insane, and appears purely due to an accident of how gcc compiles code to compare Infs to NaNs (Python's mathmodule.c's CHECK() macro was fooled by this into setting errno to ERANGE itself). Python should raise a ValueError there instead (corresponding to libm setting errno to EDOM -- this is a domain error, not a range error) -- which it does now under CVS Python. 754-doesn't-work-unless-you've-got-all-of-it-ly y'rs - tim
To look for another way out of this, the current problem are the two different wishes for vectorized computations to have NaN/Inf or Exceptions. As the actual computations for NumPy are all done in C, isn't it possible to implement a special signal handling in the NumPy code? As an option, for the people who know that they are then possibly in trouble (different behavior for arrays than for Python scalars) __Janko Just for the record. Python 1.5.2 (#1, May 10 1999, 18:46:39) [GCC egcs-2.91.60 Debian 2.1 (egcs-1.1. on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam (IPP) Type ? for more help
# Other platform (DEC Alpha) raising domain error Python 1.5.2 (#6, Sep 20 1999, 17:44:09) [C] on osf1V4 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
[Huaiyu Zhu]
Python was not designed with IEEE-754 in mind, and *anything* that happens wrt Infs, NaNs, and all other 754 features in Python is purely an accident that can and does vary wildly across platforms. And, as you've discovered in this case, can vary wildly also across even a single library, depending on config options. We cannot consider this to be a bug since Python has had no *intended* behavior whatsoever wrt 754 gimmicks. We can and do consider gripes about these accidents to be feature requests. [Guido]
Incidentally, math.exp(800) returns inf in 1.5, and raises OverflowError in 2.0. So at least it's consistent.
[Huaiyu]
That is not consistent at all.
Please read with an assumption of good faith. Guido was pointing out that-- all in the specific case of gcc+glibc on Linux (these don't hold on other platforms) --exp(800) returning Inf in 1.5 and OverflowError in 2.0 is consistent *with* that exp(-800) returns 0 in 1.5 and raises an exception in 2.0. He's right; indeed, he is in part agreeing with you. [Guido
[Huaiyu]
If using ieee is as simple as setting such a flag, there is no reason at all not to use it.
The patch to stop setting -lieee was contributed by a Python user who claimed it fixed bugs on *their* platform. That's "the reason". We don't want to screw them either.
Here are some more examples: ...
I understand that 754 semantics can be invaluable. So does Guido. There's no argument about that. But Python doesn't yet support them, and wishing it did doesn't make it so. If linking with -lieee happens to give you the semantics you want on your platform today, you're welcome to build with that switch. It appears to be a bad choice for *most* Python users, though (more below), so we don't want to do it in the std 2.0 distro.
Your biggest obstacle in getting Python support for 754 will in fact be opposition from number-crunchers. I've been slinging fp for 21 years, 15 of those writing compilers and math libraries for "supercomputer" vendors. 754 is a Very Good Thing, but is Evil in subset form (see Kahan's (justified!) vilification of Java on this point); ironically, 754 is hardest to sell to those who could benefit from it the most.
What about the other way round? No problem. It is easy to write functions like isNaN, isInf, etc.
It's easy for a platform expert to write such functions for their specific platform of expertise, but it's impossible to write them in a portable way before C99 is implemented (C99 requires that library suppliers provide them, rendering the question moot).
Which is why a subset of 754 is evil. 754 requires that the user be able to *choose* whether or not, e.g., overflow signals an exception. Your crusade to insist that it never raise an exception is as least as bad (I think it's much worse) as Python's most common accidental behavior (where overflow from libm usually raises an exception). One size does not fit all. [Tim]
Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and current code certainly relies on detecting overflows in math functions.
[Huaiyu]
As Guido observed ERANGE is not generated with ieee, even for overflow. So it is the same behavior as 1.5.2.
You've got a bit of a case of tunnel vision here, Huaiyu. Yes, in the specific case of gcc+glibc+Linux, ignoring ERANGE returned from exp() is what 1.5.2 acted like. But you have not proposed to ignore it merely from ERANGE returned from exp() in the specific case of gcc+glibc+Linux. This code runs on many dozens of platforms, and, e.g., as I suggested before, it looks like HPUX has always set errno on both overflow and underflow. MS Windows sets it on overflow but not underflow. Etc. We have to worry about the effects on all platforms.
Besides, no correct numerical code should depend on exceptions like this unless the machine is incapable of handling Inf and NaN.
Nonsense. 754 provides the option to raise an exception on overflow, or not, at the user's discretion, precisely because exceptions are sometimes more appropriate than Infs of NaNs. Kahan himself isn't happy with Infs and NaNs because they're too often too gross a clue (see his writings on "presubstitution" for a more useful approach). [Tim]
In no case can you expect to see overflow ignored in 2.0.
[Huaiyu]
I personally doubt that, but realize it may be true. That's why he have beta releases. So far yours is the only feedback we've heard (thanks!), and as a result we're going to change 2.0 to stop griping about underflow, and do so in a way that will actually work across all platforms. We're probably going to break some HPUX programs as a result; but they were relying on accidents too. [Guido]
[Huaiyu]
Python does not support 754 today. Period. I would like it to, but not in any half-assed, still platform-dependent, still random crap-shoot, still random subset of 754 features, still rigidly inflexible, way. When it does support it, Guido & I will argue that it should enable (what 754 calls) the overflow, divide-by-0 and invalid operation exceptions by default, and disable the underflow and inexact exceptions by default. This ensures that, under the default, an infinity or NaN can never be created from non-exceptional inputs without triggering an exception. Not only is that best for newbies, you'll find it's the *only* scheme for a default that can be sold to working number-crunchers (been there, done that, at Kendall Square Research). It also matches Guido's original, pre-754, *intent* for how Python numerics should work by default (it is, e.g., no accident that Python has always had an OverflowError exception but never an UnderflowError one). And that corresponds to the change Guido says <wink> I'm going to make in mathmodule.c: suppress complaints about underflow, but let complaints about overflow go through. This is not a solution, it's a step on a path towards a solution. The next step (which will not happen for 2.0!) is to provide an explicit way to, from Python, disable overflow checking, and that's simply part of providing the control and inquiry functions mandated by 754. Then code that would rather deal with Infs than exceptions can, at its explicit discretion.
If you do choose this route, please please please ignore ERANGE entirely, whether return value is zero or not.
It should be clear that isn't going to happen. I realize that triggering overflow is going to piss you off, but you have to realize that not triggering overflow is going to piss more people off, and *especially* your fellow number-crunchers. Short of serious 754 support, picking "a winner" is the best we can do for now. You have the -lieee flag on your platform du jour if you can't bear it. [Paul Dubois]
I don't have time to follow in detail this thread about changed behavior between versions.
What makes you think we do <0.5 wink>?
These observations are based on working with hundreds of code authors. I offer them as is.
FWIW, they exactly match my observations from 15 years on the HW vendor side.
C allows libm implementations a lot of discretion in whether to set errno to ERANGE in case of underflow. The change we're going to make is to ignore ERANGE in case of underflow, ensuring that math.* functions will *stop* raising underflow exceptions on all platforms (they'll return a zero instead; whether +0 or -0 will remain a platform-dependent crap shoot for now). Nothing here addresses underflow exceptions that may be raised by fp hardware, though; this has solely to do with the platform libm's treatment of errno. So in this respect, we're removing Python's current unpredictability, and in the way you favor.
Apparently Python on libc+glibc+Linux never raised OverflowError from math.* functions in 1.5.2 (although it did on most other platforms I know about). A patch checked in to fix some other complaint on some other Unixish platform had the side-effect of making Python on libc+glibc+Linux start raising OverflowError from math.* functions in 2.0, but in both overflow and underflow cases. We intend to (as above) suppress the underflow exceptions, but let the overflow cases continue to raise OverflowError. Huaiyu's original complaint was about the underflow cases, but (as such things often do) expanded beyond that when it became clear he would get what he asked for <wink>. Again we're removing Python's current unpredictability, and again in the way you favor -- although this one is still at the mercy of whether the platform libm sets errno correctly on overflow (but it seems that most do these days).
Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
Nobody has proposed changing anything about libm domain (as opposed to range) errors (although Huaiyu probably should if he's serious about his flavor of 754 subsetism -- I have no idea what gcc+glibc+Linux did here on 1.5.2, but it *should* have silently returned a NaN (not a zero) without setting errno if it was *self*-consistent -- anyone care to check that under -lieee?: import math math.sqrt(-1) NaN or ValueError? 2.0 should raise ValueError regardless of what 1.5.2 did here.). just-another-day-of-universal-python-harmony-ly y'rs - tim
On Wed, Oct 11, 2000 at 10:44:20PM -0400, Tim Peters wrote:
Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
import math math.sqrt(-1)
The same under both 1.5.2 and 2.0c1 with -lieee. Without -lieee, both do:
Consistency-conschmistency-ly y'rs, -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
I don't have time to follow in detail this thread about changed behavior between versions. These observations are based on working with hundreds of code authors. I offer them as is. a. Nobody runs a serious numeric calculation without setting underflow-to-zero, in the hardware. You can't even afford the cost of software checks. Unfortunately there is no portable way to do that that I know of. b. Some people use Inf but most people want the code to STOP so they can find out where the INFS started. Otherwise, two hours later you have big arrays of Infs and no idea how it happened. Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
Thanks, Paul! This behavior has always been what I wanted Python to do (even though it's not always what Python did, depending on the platform) and also what Tim's proposed patch will implement for the specific case of math.exp() (and other math functions that may underflow or overflow), on most reasonable platforms. There are still lots of places where the platform gives Python no choice of creating NaN and Inf, and because there's no platform-independent way to test for these, they are hard to avoid in some cases; but eventually, Tim will find a way to root them out. And for people like Huaiyu, who want to see Inf, there will (eventually) be a way to select this as a run-time option; and ditto for whoever might want underflow to raise an exception. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Paul Dubois]
Amen.
$ /usr/bin/python Python 1.5.2 (#1, Sep 17 1999, 20:15:36) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
This was sane behavior. Are we saying that Python 2.0 has invented something better than IEEE 754? [Guido van Rossum]
Guido, with due respect to your decisions on Python issues, I simply have to fight this one. It is one thing to accomodate for naive users, but it is another to dumb down every one else. Case 1. Someone writes a flawed numerical routine. Two hours later he finds his array filled with Inf and NaN. Case 2. Someone writes a perfect numerical routine. Two hours later he gets an exception, because the error is near zero. Solution for case 1. Use better algorithm. Use better error control. Raise exceptions when error is too large. These are proper solutions. They are easy and efficient to implement. They are needed anyway - If something's wrong, you want to raise exceptions far earlier than Inf, certainly before you get arrays filled with elements like 1e300. Solution for case 2. Almost impossible. The division between under- and over-flow is artificial. What about 1/x or similar functions? The only way to work on such a platform is to abandon vectorized computation.
I can understand that exceptions are the only available choices if IEEE is not available. But is there a compelling reason that Python should behave "better" than IEEE when it's in fact available? If the reason is to protect naive users, I can think of several responses: 1. For people doing one-off interactive work, returning Inf is in fact more informative. 2. For users iterative numerical computations, they need to be educated about error control. Otherwise they won't get corrent results anyway. 3. For really serious work, we could provide good numerical modules so that they don't need to write themselves. To make this happen fast the fewer debacles like this one the better. Case in point: Someone asked for regession modules two weeks ago. I was trying to convert my old matlab programs, which only took a few hours. But I wasted a week of (spare) time fighting for some mysterious "overflow". Turns out that a Guassian is near zero when it's far from center, and Python does not like it. In practice, Inf may be generated more often as a proper value than by mistake. This is not an issue about whether someone "prefers" Inf or exception. It is about whether there is a choice to do proper computation. Returning Inf does not prevent someone to raise exception. Raising exception automatically prevents perfect algorithms to work properly. As Kevin has volunteered to help with IEEE implementation and made a plan, is there a strong reason to drop IEEE for Linux in 2.0? If there is insufficient time to carry out his plan, wouldn't it be prudent to keep things as they were in 1.5.2? Huaiyu
First: I haven't followed this thread from the beginning -- only the last ten or so posts. Second: One reason I didn't follow it from the start is that I'm not doing any heavy numerical stuff in Python. I've been using Lisp for that, and use Python more for string/symbolic or GUI hacking. But, if I was going to do the sort of numerical stuff I now do in Lisp in Python, I would agree with Huaiya Zhu. I do a lot of vectorized operations on what are often independent samples. If some of the numbers overflow or underflow, that just represents an outlier or bad sample. I don't want it to blow off the whole pipeline of operations on the other data points in the vector -- they are independent of the bad points. In my case, it's not that these are lengthy calculations. It's that they are interactive and tied to immediate graphical representations. If there are strange zero's or infinities in the result, there is (or should be) a way to backtrack and investigate. ( That's why people want interactive and graphical regression analysis and modeling tools! ) The idea that your calculation should blow up and you should check it and resubmit your job sounds just so ancient-20th-century- Fortran-JCL-and-punched-cards-technology! ---| Steven D. Majewski (804-982-0831) <sdm7g@Virginia.EDU> |--- ---| Department of Molecular Physiology and Biological Physics |--- ---| University of Virginia Health Sciences Center |--- ---| P.O. Box 10011 Charlottesville, VA 22906-0011 |--- "All operating systems want to be unix, All programming languages want to be lisp."
Long-running jobs are still with us, even there's neither Fortran nor JCL in them. And for these applications, stopping is better than going on with nonsense values. On the other hand, as you point out, exceptions for math errors are a bit of a pain for interactive work. So how about making this a run-time option? I'd choose exceptions by default and Infs and Nans by specifying a command-line option, but there are certainly others who would prefer it the other way round. What matters most to me is that the choice is possible somehow. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
On Thu, 12 Oct 2000 hinsen@dirac.cnrs-orleans.fr wrote:
I agree entirely! Maybe I was being a bit too glib, but I didn't mean to imply that wanting it to halt or throw an exception on errors is wrongheaded. I just wanted to make sure the counter-case to what Paul was saying also got heard: Yes-- underflows or infinities where they aren't expected are usually a sign that something is very wrong somewhere. But in the case where the vector holds independent observations or data points, then usually what it means is that there's something wrong with *that* data point -- miscallibrated or mislabeled -- but no reason not to complete the calculations for all of the other points. Scaling or doing projections for interactive graphics is another case where bad points are often better than throwing an exception. ( And it's a pain to have to remember to lambda wrap all the function calls with some sort of guard when you'ld be happy to get NaNs. ) I also mostly agree with Tim, except that I'm not sure that bad or incomplete ieee support is always better than none at all. ---| Steven D. Majewski (804-982-0831) <sdm7g@Virginia.EDU> |--- ---| Department of Molecular Physiology and Biological Physics |--- ---| University of Virginia Health Sciences Center |--- ---| P.O. Box 10011 Charlottesville, VA 22906-0011 |--- "All operating systems want to be unix, All programming languages want to be lisp."
[Steven D. Majewski]
This is true, because having Python is better than not having Python, and in having Python you indeed have bad *and* incomplete 754 support on every 754 platform it runs on. Even better, it's a subset of damaged 754 support unique to each platform whose details can't be guessed or documented in advance of trying it exhaustively to see what happens(*), and a subset that can change delightfully across releases, compiler upgrades, library upgrades and seemingly irrelevant minor config changes. So if bad or incomplete IEEE support is good, Python is-- here as elsewhere --the King of Languages <wink>. Every step of this dance is thoroughly predictable. In this case, I'm doing my darnedest to nudge Python its very first step towards *real* 754 support, and getting dumped on for it by a 754 fan. Simultaneously, the "old guard" defends their lifestyle against new-fangled ideas <wink>, asking for protections unaware that 754 *requires* they get a better form of the protections they seek than they've dreamed of so far. It appears that nobody on either side has actually read the std, and I've become the very 754 Storm Trooper I used to despise. Computer life is great <wink>. all-it's-missing-is-variety-ly y'rs - tim (*) Quiz: *if* you can manage to trick Python into creating a NaN on your particular 754 platform, does the Python expression NaN == 1.0 return true, false, or raise an exception? Answer before you try it. Then try it on enough 754 platforms until you give up trying to guess in advance. NaN == NaN is predictable in Python, and is the one 754 feature Python guarantees won't work correctly on any 754 platform (although I've heard that it loses this predictability when run using NumPy's flavor of comparisons instead of core Python's).
This is a bit off-topic, but I noticed that there are lots of experts here and I have a question about IEEE-754 that has been bugging me for some time and which I was not able to get a good answer for on the USENET newsgroups. I am hoping that one of the experts in this list will take pity on my ignorance and answer what should be a simple question. Under Microsoft Visual C++, the Standard C++ library's numeric_limits<float>::signaling_NaN() and numeric_limits<double>::signaling_NaN() both return -INF values. The numeric_limits<float>::quiet_NaN() and numeric_limits<double>::quiet_NaN() both return -IND. I have not been able to determine whether this is "proper" behavior or whether it is another example of Microsoft ignoring standards? I would have thought that one of the two should return NaN. I certainly can't for the life of me figure out why someone would call INF a NaN, but as I have said, I'm pretty ignorant. Right now, if I want to use NaN's in my C++ code (e.g., to initialize newly allocated memory blocks) I build NaNs thus:
but would prefer something a little less bit-tweaky. Microsoft's documentation that this is the way they intend their library to work may be found at: http://msdn.microsoft.com/library/devprods/vs6/visualc/vclang/sample_Members... Thanks, Jonathan ============================================================================= Jonathan M. Gilligan jonathan.gilligan@vanderbilt.edu The Robert T. Lagemann Assistant Professor Office: 615 343-6252 of Living State Physics Lab (FEL) 343-7580 Dept. of Physics and Astronomy, Box 1807-B Fax: 343-7263 6823 Stevenson Center Vanderbilt University, Nashville, TN 37235 Dep't Office: 322-2828
Incomplete vs. non-at-all IEEE 754 support is a non argument. If you have full IEEE support (which it seems everyone here thinks would be good, but difficult), it is clear what you have. If not, you are not consistent with a standard, and therefor making up your own. That being the case, it's a matter of what we want the Python standard to be. I, for one think that NaN and Inf are fantastic!!! I think the best argument for them here is that it is almost impossible to do a lot of array based calculations without them, and you can't do serious number crunching in Python without array based calculations. I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754 support, as I don't think you can change the behaviour, but you do get a consistent result across platforms (including non-iee machines like the Cray?---I have no idea). I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception. This really does render Python essentially useless for Numerics for me. I suspect all those number crunchers that want an exception rather than an Inf are NOT using array-oriented languages. My goal is to dump MATLAB for Python, but this may prevent me from doing that. Does anyone know what other array-oriented languages use? I know what MATLAB does:
Hey! I like that! Python is dynamic, why can't we just get the actual answer to sqrt(-1), without using cmath, that always returns a complex ? sorry, other subjet, not meant to be raised at the moment.
Here we get a warning, but also a result that won't crash the computation.
So comparing two NaNs yields a false, as it should, and though Python won't do it now, it could. One thing that a numerical routine should NEVER do is give an incorrect answer. No answer is often bad, but an incorrect one is much worse. NaN == NaN must return false. Does anyone know what FORTRAN 90 specifies (if anything)? What other array-oriented languages are there? I think what MATLAB does is what Tim is calling "bad *and* incomplete 754 support" Incomplete is surely true, "bad" is clearly a matter of opinion. It seems we have a variety of users of numerics in Python, that I break into three broad catagories: Casual users: mostly doing non-numeric stuff, with the occasional calculation: This group could use any non-cryptic handling of over/underflow, it just doesn't matter. Mid-level number crunchers: This is the group I put myself in. We crunch a lot of numbers, really like the array semantics (and speed) of NumPy, and are doing things like graphing functions, statistical calculations, etc. We have probably only taken one of two (at most) numerical analysis courses, and many don't know what the heck IEE 754 is. (The one Numerical Analysis course I did take, happened to be with Kahan, which is why I know what it is) For this group, the incomplete IEEE support is probably the best way to go. We're more likely to be annoyed by our whole computation stopping because of one bad data point, than we are to be pissed off that it didn't stop when it started to compute garbage. Hard Core Number crunchers: These are the folks that do things like write optimised routines for particular architectures, adn teach numerical analysis courses. These folks want either FULL IEEE, or plain old "classic" overflow and underflow errors, that they can handle themselves. Do these folks really use Python as other than a glue language? Are they really doing massive number crunching in Python itself, rather than in C (or whatever) extensions? If so, I'd be surprised is they didn't find Huaiyu's argument compelling when doing array based calculations. Tim pointed out that Python was not designed with 754 in mind, so for 2.0, what he is doing is probably our best bet, but it seems to not be the best ultimate solution, I hope using NaN and Inf will be considered for future versions, even if it is incomplete 754 compliance. Note: if the ultimate goal is REAL IEEE 754 compliance, is it possible without custom re-compiling your own interpreter? If so, is there any chance that it will come any time soon (2.1 ?) , so we don't have to have this discussion at all? Thanks for all of your work on this, Python just keeps getting better!! -Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------
Chris Barker writes:
Hey! I like that! Python is dynamic, why can't we just get the actual answer to sqrt(-1), without using cmath, that always returns a complex ?
You have to import the sqrt function from somewhere. Either you import it from math or from cmath, depending on which flavor you need. Anybody sophisticated enough to know what complex numbers are, and sophisticated enough to want to get complex values as a result of a calculation, should be sophisticated enough to be able to type "cmath".
Does anyone know what other array-oriented languages use? I know what MATLAB does:
I'm an Interactive Data Language or IDL user (the Univ of Wisconsin's Space Science Ceter is split down the middle between this and MatLab, but python/numpy is definitely on the increase). As you can see from results below, like MatLab over/under-flows in IDL are reported but do not stop execution. This is the best (only) possible scenario for an interactive arrays and visualization environment. IDL> print, exp(-777) 0.00000 % Program caused arithmetic error: Floating underflow IDL> print, exp(777) Inf % Program caused arithmetic error: Floating overflow IDL> print, sqrt(-1) -NaN % Program caused arithmetic error: Floating illegal operand IDL> print, 54/0 54 % Program caused arithmetic error: Integer divide by 0
I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754
MATLAB is fine for simple interactive data processing, and its behaviour is adapted to this task. I don't think anyone would use MATLAB for developing complex algorithms, its programming language isn't strong enough for that. Python is, so complex algorithms have to be considered as well. And for that kind of application, continuing a calculation with Infs and NaNs is a debugging nightmare.
I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception.
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
For the same reason that makes 2/3 return zero instead of a float division result. Like C or Fortran, Python treats integers, floats, and complex numbers as different data types. And the return type of a function should depend only on the types, but not the values, of its parameters (for consistency, not because of any implementational limitation). So sqrt(a) for a of type float can either always return a float (math, Numeric) and crash for negative arguments, or always return a complex (cmath). The "right" solution, in my opinion, would be to have a single "number" type of which integers, float, and complex numbers are simply different internal representations. But such a change cannot be introduced without breaking a lot of code. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
[cbarker@jps.net]
I've come to Python from MATLAB for numerics, and I really appreciated MATLAB's way of handling all this. I don't think MATLAB has true 754 ...
[Konrad Hinsen]
A non-constructive (because futile) speculation: the first time I saw what 754 was intending to do, my immediate reaction was "hmm -- the exponent field is too narrow!". With a max (double) val in the ballpark of 1e300, you get an infinity as soon as you square something as small as 1e150, and once you get a few infinities, NaNs are sure to follow (due to Inf-Inf and Inf/Inf). The dynamic range for 754 singles is much smaller still. There's no doubt about Cray arithmetic being hard to live with, but while Mr. Cray didn't worry about proper rounding, he did devote 15 bits to Cray's exponent field (vs. 11 for 754). As a result, overflows were generally a non-issue on Cray boxes, and *nobody* complained (in my decade there) about Cray HW raising a fatal exception if one occurred. In return, you got only 48 bits of precision (vs. 53 for 754). But, for most physical problems, how accurate are the inputs? 10 bits on a good day, 20 bits on a great day? Crays worked despite their sloppy numerics because, for most problems to which they were applied, they carried more than twice the precision *and* dynamic range than the final results needed.
I have not yet heard a decent response to the question of what to do when a single value in a large array is bad, and causes an exception.
I'd usually trace back and try to figure out how it got "bad" to begin with ...
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
Does NumPy use the fpectl module? I suppose not. LLNL contribued that code, but we hear very little feedback on it. It arranges to (in platform-dependent ways) convert the 754 overflow, divide-by-0 and invalid-operation signals into Python exceptions. In core Python, this is accomplished at the extraordinary expense of doing a setjmp before, and a function call + double->int conversion after, every single fp operation. A chief problem is that SIGFPE is the only handle C gives us on fp "errors", and the C std does not allow returning from a SIGFPE handler (the result of trying to is undefined, and indeed varies wildly across platforms); so if you want to regain control, you have to longjmp out of the handler. The NumPy implementation could use the PyFPE_START_PROTECT and PyFPE_END_PROTECT macros to brackete entire array operations, though, and so pay for the setjmp etc only once per array op. This is difficult stuff, but doable.
You know I'm in general agreement with you on this one, but I have to draw a distinction here: Guido thinks that 2/3 returning 0 was a design mistake, but not that math.sqrt(-1) raising an exception is a mistake. Most Python users won't know what to do with a complex number, so it's "an error" to them. I would like to view this in P3K (if not earlier) as being akin to 754 exceptions: some people are delighted to have 1e300**2 return +Inf, while others want to see OverflowError instead, and still others want to see +Inf *and* have a sticky flag set saying "an overflow occurred". We could treat f(x) (for f == sqrt and everything else) the same way wrt to a new ComplexResultFromNonComplexInputsError: define "non-stop" complex results, let the user choose whether to do nonstop or raise an exception, and a supply a sticky flag saying whether or not any complex results were generated from non-complex inputs.
The current distinction between ints and longs (unbounded ints) should also get swallowed by this. A way to get from here to there is especially irksome at the C API level, since, e.g., many C API functions pass Python ints as C longs directly. A smaller number pass Python floats as C doubles directly. it's-wonderful-that-p3k-will-solve-everything<wink>-ly y'rs - tim
No. Perhaps it should, but it doesn't make sense unless fpectl works on a large number of platforms. I confess I have never looked at it. I want my code to be portable, so I don't even consider using packages that aren't.
Well, that would be a good occasion to learn about complex numbers! I remember having learned about generalized inverses by using APL in high school (in a very similar way: I was amazed that it could invert non-square matrices), and that turned out to be very useful knowledge later on. Perhaps that's a topic for the EDU-SIG... Anyway, I don't care what math.sqrt(-1) does, but I would definitely prefer Numeric.sqrt(-1) to return a complex result. And I think that someone who uses NumPy has probably heard about complex numbers.
There is, however, one difference: in properly debugged production code, there should be no overflow situations, so it doesn't matter much how they are treated. Complex number can (do!) occur in production code, but there could also be production code relying on exceptions for sqrt(-1) (e.g. for input error checking). Therefore a program using several libraries might be impossible to run with either setting. Since this concerns only the math module, I'd prefer to keep separate module versions for the two cases, which can be used in parallel.
I don't see the problem in that direction, it's rather C API functions that *return* numbers in C data types that would be difficult to adapt. But then, why should be C API not be allowed in P3K? it's-wonderful-that-anything-may-be-changed-in-p3k'ly Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
And from Konrad Hinsen
I'd like to have at least the option of raising an exception in that case. Note that this is not what NumPy does today.
Exactly. This was Huaiyu's point. The problem is that for your code to be usable for folks other than yourself, you'd have to have a lot of checks in there, and essentially revert to elementwise code, which would kill you. With the addition of the occasional "isnan" and something like Matlab's "any" ( "any(isnan(A))" returns true if any of the elements of A are NaNs) you could set up your code to raise an exception in the middle of computation. If, on the other hand the Code definiately raises an exception, than you are stuck with a lot of elementwise coding.
Exactly!!! ("the best (only) possible scenario" part)
This, however, is wierd!! 54/0 == 54 ??? I'd much rather see a NaN or Inf here!
People do some pretty complex algorith develpment with MATLAB. I'm also no so sure about "continuing a calculation with Infs and NaNs is a debugging nightmare" I'm don't think it's any more of a nighmare than having your calculation on an entire array stop because of one Inf. It's just a different trade off. Sprinkle a few "isnan"'s in there and and you can get the behaviour you want, while not forcing it on all calculations. Of course, the best option is to have it switchable, but that may prove to be very difficult. Are people doing the kind of numeric programming with "complex algorithms" using Python that Konrad refers to? More importantly, how many people?
Guido's philosophy is clearly that Python defaults should be geared to "Most Python users". I agree, and as I wrote in an earlier post, the only users for whom the "exception raising only" option is best are the users Konrad refers to as writing "complex algorithms". I would argue that those users are few, and the group most able to deal with a less-that-optimum system. Maybe we can have true 754 compliance in Py3k, and we can all be happy!! -Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------
At least one: me.
It's difficult to say what "most Python users" want; it's not a static community. I'd say the basic principle is "no bad surprises". 2/3 == 0 is a bad surprise for anyone who knows elementary math but is not familiar with certain programming languages, especially since the program goes on silently with a "wrong" intermediate result.
Maybe we can have true 754 compliance in Py3k, and we can all be happy!!
Py3k forever! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
[Huaiyu Zhu]
754 allows users to choose the behaviors they want. Any *fixed* policy is not 754. So long as we have to fix a single policy, yes, I believe Guido (& know I) would say that raising OverflowError on exp(777), and silently returning 0 on exp(-777), and raising ValueError on sqrt(-1) (*not* OverflowError, as shown above), is indeed better than the 754 default behaviors. And 2.0 will do all three of those (& I just checked in the code to make that happen). About the sqrt example above, that's neither 754 behavior nor sane behavior: default 754 behavior on sqrt(-1) is to return a NaN and keep on going. I'm pretty sure that's what glibc linked w/ -lieee actually does, too (if it doesn't, glibc -lieee is violating 754). That 1.5.2 raised an OverflowError instead is insane, and appears purely due to an accident of how gcc compiles code to compare Infs to NaNs (Python's mathmodule.c's CHECK() macro was fooled by this into setting errno to ERANGE itself). Python should raise a ValueError there instead (corresponding to libm setting errno to EDOM -- this is a domain error, not a range error) -- which it does now under CVS Python. 754-doesn't-work-unless-you've-got-all-of-it-ly y'rs - tim
To look for another way out of this, the current problem are the two different wishes for vectorized computations to have NaN/Inf or Exceptions. As the actual computations for NumPy are all done in C, isn't it possible to implement a special signal handling in the NumPy code? As an option, for the people who know that they are then possibly in trouble (different behavior for arrays than for Python scalars) __Janko Just for the record. Python 1.5.2 (#1, May 10 1999, 18:46:39) [GCC egcs-2.91.60 Debian 2.1 (egcs-1.1. on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam (IPP) Type ? for more help
# Other platform (DEC Alpha) raising domain error Python 1.5.2 (#6, Sep 20 1999, 17:44:09) [C] on osf1V4 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
[Huaiyu Zhu]
Python was not designed with IEEE-754 in mind, and *anything* that happens wrt Infs, NaNs, and all other 754 features in Python is purely an accident that can and does vary wildly across platforms. And, as you've discovered in this case, can vary wildly also across even a single library, depending on config options. We cannot consider this to be a bug since Python has had no *intended* behavior whatsoever wrt 754 gimmicks. We can and do consider gripes about these accidents to be feature requests. [Guido]
Incidentally, math.exp(800) returns inf in 1.5, and raises OverflowError in 2.0. So at least it's consistent.
[Huaiyu]
That is not consistent at all.
Please read with an assumption of good faith. Guido was pointing out that-- all in the specific case of gcc+glibc on Linux (these don't hold on other platforms) --exp(800) returning Inf in 1.5 and OverflowError in 2.0 is consistent *with* that exp(-800) returns 0 in 1.5 and raises an exception in 2.0. He's right; indeed, he is in part agreeing with you. [Guido
[Huaiyu]
If using ieee is as simple as setting such a flag, there is no reason at all not to use it.
The patch to stop setting -lieee was contributed by a Python user who claimed it fixed bugs on *their* platform. That's "the reason". We don't want to screw them either.
Here are some more examples: ...
I understand that 754 semantics can be invaluable. So does Guido. There's no argument about that. But Python doesn't yet support them, and wishing it did doesn't make it so. If linking with -lieee happens to give you the semantics you want on your platform today, you're welcome to build with that switch. It appears to be a bad choice for *most* Python users, though (more below), so we don't want to do it in the std 2.0 distro.
Your biggest obstacle in getting Python support for 754 will in fact be opposition from number-crunchers. I've been slinging fp for 21 years, 15 of those writing compilers and math libraries for "supercomputer" vendors. 754 is a Very Good Thing, but is Evil in subset form (see Kahan's (justified!) vilification of Java on this point); ironically, 754 is hardest to sell to those who could benefit from it the most.
What about the other way round? No problem. It is easy to write functions like isNaN, isInf, etc.
It's easy for a platform expert to write such functions for their specific platform of expertise, but it's impossible to write them in a portable way before C99 is implemented (C99 requires that library suppliers provide them, rendering the question moot).
Which is why a subset of 754 is evil. 754 requires that the user be able to *choose* whether or not, e.g., overflow signals an exception. Your crusade to insist that it never raise an exception is as least as bad (I think it's much worse) as Python's most common accidental behavior (where overflow from libm usually raises an exception). One size does not fit all. [Tim]
Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and current code certainly relies on detecting overflows in math functions.
[Huaiyu]
As Guido observed ERANGE is not generated with ieee, even for overflow. So it is the same behavior as 1.5.2.
You've got a bit of a case of tunnel vision here, Huaiyu. Yes, in the specific case of gcc+glibc+Linux, ignoring ERANGE returned from exp() is what 1.5.2 acted like. But you have not proposed to ignore it merely from ERANGE returned from exp() in the specific case of gcc+glibc+Linux. This code runs on many dozens of platforms, and, e.g., as I suggested before, it looks like HPUX has always set errno on both overflow and underflow. MS Windows sets it on overflow but not underflow. Etc. We have to worry about the effects on all platforms.
Besides, no correct numerical code should depend on exceptions like this unless the machine is incapable of handling Inf and NaN.
Nonsense. 754 provides the option to raise an exception on overflow, or not, at the user's discretion, precisely because exceptions are sometimes more appropriate than Infs of NaNs. Kahan himself isn't happy with Infs and NaNs because they're too often too gross a clue (see his writings on "presubstitution" for a more useful approach). [Tim]
In no case can you expect to see overflow ignored in 2.0.
[Huaiyu]
I personally doubt that, but realize it may be true. That's why he have beta releases. So far yours is the only feedback we've heard (thanks!), and as a result we're going to change 2.0 to stop griping about underflow, and do so in a way that will actually work across all platforms. We're probably going to break some HPUX programs as a result; but they were relying on accidents too. [Guido]
[Huaiyu]
Python does not support 754 today. Period. I would like it to, but not in any half-assed, still platform-dependent, still random crap-shoot, still random subset of 754 features, still rigidly inflexible, way. When it does support it, Guido & I will argue that it should enable (what 754 calls) the overflow, divide-by-0 and invalid operation exceptions by default, and disable the underflow and inexact exceptions by default. This ensures that, under the default, an infinity or NaN can never be created from non-exceptional inputs without triggering an exception. Not only is that best for newbies, you'll find it's the *only* scheme for a default that can be sold to working number-crunchers (been there, done that, at Kendall Square Research). It also matches Guido's original, pre-754, *intent* for how Python numerics should work by default (it is, e.g., no accident that Python has always had an OverflowError exception but never an UnderflowError one). And that corresponds to the change Guido says <wink> I'm going to make in mathmodule.c: suppress complaints about underflow, but let complaints about overflow go through. This is not a solution, it's a step on a path towards a solution. The next step (which will not happen for 2.0!) is to provide an explicit way to, from Python, disable overflow checking, and that's simply part of providing the control and inquiry functions mandated by 754. Then code that would rather deal with Infs than exceptions can, at its explicit discretion.
If you do choose this route, please please please ignore ERANGE entirely, whether return value is zero or not.
It should be clear that isn't going to happen. I realize that triggering overflow is going to piss you off, but you have to realize that not triggering overflow is going to piss more people off, and *especially* your fellow number-crunchers. Short of serious 754 support, picking "a winner" is the best we can do for now. You have the -lieee flag on your platform du jour if you can't bear it. [Paul Dubois]
I don't have time to follow in detail this thread about changed behavior between versions.
What makes you think we do <0.5 wink>?
These observations are based on working with hundreds of code authors. I offer them as is.
FWIW, they exactly match my observations from 15 years on the HW vendor side.
C allows libm implementations a lot of discretion in whether to set errno to ERANGE in case of underflow. The change we're going to make is to ignore ERANGE in case of underflow, ensuring that math.* functions will *stop* raising underflow exceptions on all platforms (they'll return a zero instead; whether +0 or -0 will remain a platform-dependent crap shoot for now). Nothing here addresses underflow exceptions that may be raised by fp hardware, though; this has solely to do with the platform libm's treatment of errno. So in this respect, we're removing Python's current unpredictability, and in the way you favor.
Apparently Python on libc+glibc+Linux never raised OverflowError from math.* functions in 1.5.2 (although it did on most other platforms I know about). A patch checked in to fix some other complaint on some other Unixish platform had the side-effect of making Python on libc+glibc+Linux start raising OverflowError from math.* functions in 2.0, but in both overflow and underflow cases. We intend to (as above) suppress the underflow exceptions, but let the overflow cases continue to raise OverflowError. Huaiyu's original complaint was about the underflow cases, but (as such things often do) expanded beyond that when it became clear he would get what he asked for <wink>. Again we're removing Python's current unpredictability, and again in the way you favor -- although this one is still at the mercy of whether the platform libm sets errno correctly on overflow (but it seems that most do these days).
Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
Nobody has proposed changing anything about libm domain (as opposed to range) errors (although Huaiyu probably should if he's serious about his flavor of 754 subsetism -- I have no idea what gcc+glibc+Linux did here on 1.5.2, but it *should* have silently returned a NaN (not a zero) without setting errno if it was *self*-consistent -- anyone care to check that under -lieee?: import math math.sqrt(-1) NaN or ValueError? 2.0 should raise ValueError regardless of what 1.5.2 did here.). just-another-day-of-universal-python-harmony-ly y'rs - tim
On Wed, Oct 11, 2000 at 10:44:20PM -0400, Tim Peters wrote:
Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
import math math.sqrt(-1)
The same under both 1.5.2 and 2.0c1 with -lieee. Without -lieee, both do:
Consistency-conschmistency-ly y'rs, -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
participants (13)
-
Charles G Waldman -
Chris Barker -
Guido van Rossum -
hinsen＠dirac.cnrs-orleans.fr -
Huaiyu Zhu -
Janko Hauser -
Jonathan M. Gilligan -
Konrad Hinsen -
Nick Bower -
Paul F. Dubois -
Steven D. Majewski -
Thomas Wouters -
Tim Peters