Re:[Python-Dev] Fixed Decimal types
Andy Robinson wrote: For the work I do, simply taking the precision of the most precise input (4.00L)would do the trick, but your answer (4.0000L) is purer. We should provide a rounding function, and in practice anyone using such a function would round (or floor, or ceiling) to get to the desired precision immediately. I'm not sure on division either but I'm sure there are precedents to look at. The AS400 provides a useful example of the right way to do scaled decimals. In the RPG programming language, all internal calculations (i.e. multiplication, division) are performed to the maximum precision of the intermediate result (in the multiplication example below), the intermediate result would be 4.0000L. When the intermediate result is assigned to the target scaled decimal number, the decimal precision is automatically extended or truncated to fit the target precision. One extra wrinkle in all of this is the option to "half-adjust" the intermediate value on assignment; that is to apply automatic 5/4 rounding to the precision of the target. So, if the target field is defined as numeric(4,2), the result will be 4.00L. These are probably the kind of semantics that a scaled decimal type would require in Python also; i.e. allow unlimited precision in intermediate calculations, with a sensible set of rules for assignment to a variable of different scale and precision. However, unlike RPG, we should probably ensure that attempts to overflow or underflow the scale result in NaN or Overflow conditions, rather than assuming the user is right and losing the significant digits. Regards, Mike da Silva
participants (3)
-
Da Silva, Mike
-
Gordon McMillan
-
Jim Fulton