floating point in 2.0

[Tim Peters]

[Tim]
>> Ack -- if you folks haven't learned anything else <wink> from the 16
>> years of half-baked 754 support so far, believe me when I tell you
>> that when it comes to f.p., most vendors see "strongly recommended"
>> and think "OK, we can skip this one -- next.".

[Edward Jason Riedy]
> Hell, they do that if it's required.

HW is key, because without near-universal HW support for a thing that thing
will never become widely accepted.  HW vendors used to punt on denorms
because of the expense, but despite the expense everyone plays along with
denorms now, and not because of user demand but simply because it's
required.  Very few play along with double extended, simply because it's not
required.

Getting language support is a different battle (or endless sequence of
decade-long battles <0.5 wink>).

> One of the good-and-bad points of 754 is that it's a _system_ standard.
> It's good in that it can't be undercut by some portion of the system.
> It's bad because each part simply points to the other parts for
> compliance.
>
> For example, 754 requires single precision if double precision
> is supported, but a certain relevant language doesn't provide
> it.  (Well, ok, you can use Numpy with a one-element array.)

Python has no 754 story -- it doesn't try.  This is a reflection of that std
C classic has no 754 story, and C99's 754 story isn't implemented yet (and
sorry to say I'm not sure it ever will be -- there too it's an optional
thing).  Without reliable semantics in the implementation language, it takes
a battalion of platform experts to "fake it" via #ifdef tricks instead;
there aren't enough platform experts to go around, and to the extent that
users have been motivated enough to contribute #ifdef'ed f.p. code to
Python, we ended up with the fpectl module (whose purpose is to fight 754
semantics by raising exceptions instead of returning NaNs and Infs).

> ...
> We also have to consider how we're getting this through the
> IEEE process.  If we come out and say C99 ain't 754 (it mandates
> 6 digit output by default), well, I think some people will be
> upset.

You can be more creative than that <0.9 wink>.

>> Vendors don't compete on mere "quality of implementation" f.p issues
>> because their customers overwhelmingly can't tell the difference so
>> don't care.

> That's not entirely true.  Consider IBM.  They added fused
> multipy-accumulate (being added to 754) and a ton of diagnostics
> that are not required.  It wasn't just out of the goodness of
> their hearts.

It certainly wasn't.  Fusing the mul and add HW wasn't even suggested by
754, it was something they did to reduce chip real estate, and because they
had clear ways in mind to speed math libraries by exploiting the resulting
ability to get at the low-order 53-bits of f.p. products.  Vendors do
compete on chip cost and math library speed, and this was a brilliant way to
improve both.

> Similarly, Sun has a host of easy-to-use exception handling
> libraries.  So far, I'm the only confirmed user outside Sun.  The
> libraries aren't going away because the internal support folks use
> them to diagnose client problems.

Sun is an exception, and I'd guess more due to David Hough's influence and
persistence than to anything else -- and I expect you just confirmed my
impression of how much this is driven by user demand <wink>.

>> That's why you have near-universal support for denorms and directed
>> rounding modes today, [...]

> Denorms, yes, but who supports directed rounding modes?  Currently
> only C99 is the only major language to give reasonable (arguable)
> access to them.  System standard, not hardware standard.

Virtually all HW FPUs support them, and because they're required.  Without
that C99 wouldn't have even considered it (part of X3J11's mandate is not to
require things that can't be done w/ reasonable efficiency, so w/o
near-universal HW support already in place, they would have punted on this
too).

> Oh, and actually, _no one_ supports denorms as intended.  There's
> a clause (don't have it handy) that can be read a few different
> ways.  The author intended one reading, but all implementors took
> another.  ;)  And it looks like there's actually a bug in one
> reading where you can have an inexact answer without an inexact
> flag.  I seem to have lost my notes on it, which is rather upsetting.
> Denorms will see some simplification as a result of all this; we're
> not sure what yet.

Bingo:  They implement exactly what's written (to the best of their ability
to read it), because it's required.  Even if it doesn't make sense, even if
nobody asks for it, just because it's required.  Tossing "recommended" out
the window is the best tool a std has.

> ...
> All decimal->binary conversions are correctly rounded.

No matter how many digits are in the decimal string?  For example,

    "0.5" + "0" * 1000000 + "1"

In the +inf rounding mode, stop anywhere short of the full million and four
digits and you get a wrong answer.  This kind of thing isn't worth it to
library writers, so go *too* far off the deep end and the standard will be
ignored in favor of the old one.  You said you're introducing a quad format,
and I assume that's "the usual" 15-bit exponent + 113-bit mantissa one, so
stop at required correct rounding to and from no more than 36 decimal
digits, and everyone will be happy (this can be done with code much simpler
than Gay's, and also without bigint arithmetic).

> ...
> Binary to decimal is constrained so that binary -> decimal -> binary
> produces the same number if both binary numbers are in the same
> precision.

Users want that, but IIRC 754 already requires it.

> Dealing with NaNs and infinities is the real problem.  I want to
> avoid specifying a textual representation to avoid the character
> encoding mess.  We'll see.

C99 didn't back down from this, so I expect it's safe to leave this up to
other stds (e.g., Fortran and C spellings may differ, but that's on their
heads then -- more, each language has its own way of dealing with character
encoding issues, and *can't* bend to anyone else's view).

> ...
> Python's default really should be single precision with at least
> double precision for all intermediates.

I expect there's no chance of single default, or of using more than double
for intermediates, the latter simply because C doesn't support it
x-platform.  WRT single default, it doesn't have enough precision to avoid
gross surprises -- newbies aren't forecasting the weather with one-digit
inputs, they're calculating their retirement benefits based on 20% compound
growth for 50 years <wink>.  Seriously, single carries less precision than a
one-dollar hand calculator, and the latter are the source of newbies'
expectations.  Single is a storage-efficiency gimmick for big problems with
low-precision data.

> ...
> A widest-feasible evaluation scheme could get rid of the whole
> right-method v. left-method v. coerce nonsense, too,

"Mumber-like types" include things like Dates and Times in Python; there is
no widest numeric type, and the set of numeric types isn't even known at
compile-time (users can add new ones on the fly).  Even restricted to
floating types, at least two Python GMP wrappers exist now, at least one of
which exposes GMP's unbounded (but dynamically specified) precision binary
fp type.  "Widest feasible" is something Python simply can't know -- that's
a strategy from the world of statically typed languages, and it doesn't
carry over cleanly.