[Python-Dev] Adventures with Decimal

[Tim Peters]

Sorry, I simply can't make more time for this.  Shotgun mode:

[Raymond]
> I have no such thoughts but do strongly prefer the current
> design.

How can you strongly prefer it?  You asked me whether I typed floats
with more than 28 significant digits.  Not usually <wink>.  Do you? 
If you don't either, how can you strongly prefer a change that makes
no difference to what you do?

> ...
> The overall design of the module and the spec is to apply
> context to the results of operations, not their inputs.

But string->float is an _operation_ in the spec, as it has been since
1985 in IEEE-754 too.  The float you get is the result of that
operation, and is consistent with normal numeric practice going back
to the first time Fortran grew a distinction between double and single
precision.  There too the common practice was to write all literals as
double-precision, and leave it to the compiler to round off excess
bits if the assignment target was of single precision.  That made it
easy to change working precision via fiddling a single "implicit" (a
kind of type declaration) line.  The same kind of thing would be
pleasantly applicable for decimal too -- if the constructor followed
the rules.

> In particular, the spec recognizes that contexts can change
|> and rather than specifying automatic or implicit context
> application to all existing values, it provides the unary plus
> operation so that such an application is explicit.  The use
> of extra digits in a calculation is not invisible as the
> calculation will signal Rounded and Inexact (if non-zero digits
> are thrown away).

Doesn't change that the standard rigorously specifies how strings are
to be converted to decimal floats, or that our constructor
implementation doesn't do that.

> One of the original motivating examples was "schoolbook"
> arithmetic where the input string precision is incorporated
> into the calculation.

Sorry, doesn't ring a bell to me.  Whose example was this?

> IMO, input truncation/rounding is inconsistent with that
> motivation.

Try keying more digits into your hand calculator than it can hold <0.5 wink>.

> Likewise, input rounding runs contrary to the basic goal of
> eliminating representation error.

It's no surprise that an exact value containing more digits than
current precision gets rounded.  What _is_ surprising is that the
decimal constructor doesn't follow that rule, instead making up its
own rule.  It's an ugly inconsistency at best.

> With respect to integration with the rest of Python (everything
> beyond that spec but needed to work with it), I suspect that
> altering the Decimal constructor is fraught with issues such
> as the string-to-decimal-to-string roundtrip becoming context
> dependent.

Nobody can have a reasonable expectation that string -> float ->
string is an identity for any fixed-precision type across all strings.
 That's just unrealistic.  You can expect string -> float -> string to
be an identity if the string carries no more digits than current
precision.  That's how a bounded type works.  Trying to pretend it's
not bounded in this one case is a conceptual mess.

> I haven't thought it through yet but suspect that it does not
> bode well for repr(), pickling, shelving, etc.

The spirit of the standard is always to deliver the best possible
approximation consistent with current context.  Unpickling and
unshelving should play that game too.  repr() has a special desire for
round-trip fidelity.

> Likewise, I suspect that traps await multi-threaded or multi-
> context apps that need to share data.

Like what?  Thread-local context precision is a reality here, going
far beyond just string->float.

> Also, adding another step to the constructor is not going to
> help the already disasterous performance.

(1) I haven't found it to be a disaster.  (2) Over the long term, the
truly speedy implementations of this standard will be limited to a
fixed set of relatively small precisions (relative to, say, 1000000,
not to 28 <wink>).  In that world it would be unboundedly more
expensive to require the constructor to save every bit of every input:
 rounding string->float is a necessity for speedy operation over the
long term.

> I appreciate efforts to make the module as idiot-proof as
> possible.

That's not my interest here.  My interest is in a consistent,
std-conforming arithmetic, and all fp standards since IEEE-754
recognized that string->float is "an operation" much like every other
fp operation.  Consistency helps by reducing complexity.  Most users
will never bump into this, and experts have a hard enough job without
gratuitous deviations from a well-defined spec.  What's the _use case_
for carrying an unbounded amount of information into a decimal
instance?  It's going to get lost upon the first operation anyway.

> However, that is a pipe dream.  By adopting and exposing the
> full standard instead of the simpler X3.274 subset, using the
> module is a non-trivial exercise and, even for experts, is a
> complete PITA.

Rigorous numeric programming is a difficult art.  That's life.  The
many exacting details in the standard aren't the cause of that,
they're a distillation of decades of numeric experience by bona fide
numeric experts.  These are the tools you need to do a rigorous job --
and most users can ignore them completely, or at worst set precision
once at the start and forget it.  _Most_ of the stuff (by count) in
the standard is for the benefit of expert library authors, facing a
wide variety of externally imposed requirements.

> Even a simple fixed-point application (money, for example)
> requires dealing with quantize(), normalize(), rounding modes,
> signals, etc.

I don't know why you'd characterize a monetary application as
"simple".  To the contrary, they're as demanding as they come.  For
example, requirements for bizarre rounding come with that territory,
and the standard exposes tools to _help_ deal with that.  The standard
didn't invent rounding modes, it recognizes that needing to deal with
them is a fact of life, and that it's much more difficult to do
without any help from the core arithmetic.  So is needing to deal with
many kinds of exceptional conditions, and in different ways depending
on the app -- that's why all that machinery is there.

> By default, outputs are not normalized so it is difficult even
> to recognize what a zero looks like.

You're complaining about a feature there <wink>.  That is, the lack of
normalization is what makes 1.10 the result of 2.21 - 1.11, rather
than 1.1 or 1.100000000000000000000000000.  1.10 is what most people
expect.

> Just getting output without exponential notation is difficult.

That's a gripe I have with the std too.  Its output formats are too
simple-minded and few.  I had the same frustration using REXX. 
Someday the %f/%g/%e format codes should learn how to deal with
decimals, and that would be pleasant enough for me.

> If someone wants to craft another module to wrap around and
> candy-coat the Decimal API, I would be all for it.

For example, Facundo is doing that with a money class, yes?  That's
fine.  The standard tries to support many common arithmetic needs, but
big as it is, it's just a start.

> Just recognize that the full spec doesn't have a beginner
> mode -- for better or worse, we've simulated a hardware FPU.

I haven't seen a HW FPU with unbounded precision, or one that does
decimal arithmetic.  Apart from the limited output modes, I have no
reason to suspect that a beginner will have any particular difficulty
with decimal.  They don't have to know anything about signals and
traps, rounding modes or threads, etc etc -- right out of the box,
except for output fomat it acts very much like a high-end hand
calculator.

> Lastly, I think it is a mistake to make a change at this point.

It's a worse mistake to let a poor decision slide indefinitely -- it
gets harder & harder to change it over time.  Heck, to listen to you,
decimal is so bloody complicated nobody could possibly be using it now
anyway <wink>.

> The design of the constructor survived all drafts of the PEP,
> comp.lang.python discussion, python-dev discussion, all early
> implementations, sandboxing, the Py2.4 alpha/beta, cookbook
> contributions, and several months in the field.

So did every other aspect of Python you dislike now <0.3 wink>.  It
never occurred to me that the implementation _wouldn't_ follow the
spec in its treatment of string->float.  I whined about that when I
discovered it, late in the game.  A new, conforming string->float
method was added then, but for some reason (or no reason) I don't
recall, the constructor wasn't changed.  That was a mistake.

>  I say we document a recommendation to use
> Context.create_decimal() and get on with life.

....

> P.S.  With 28 digit default precision, the odds of this coming
> up in practice are slim (when was the last time you typed in a
> floating point value with more than 28 digits; further, if you had,
> would it have ruined your day if your 40 digits were not first
> rounded to 28 before being used).

Depends on the app, of course.  More interesting is the day when
someone ports an app from a conforming implementation of the standard,
sets precision to (say) 8, and gets different results in Python
despite that the app stuck solely to standard operations.  Of course
that can be a genuine disaster for a monetary application -- extending
standards in non-obvious ways imposes many costs of its own, but they
fall on users, and aren't apparent at first.  I want to treat the
decimal module as if the standard it purports to implement will
succeed.