"Da Silva, Mike" wrote:

Andy Robinson wrote: For the work I do, simply taking the precision of the most precise input (4.00L)would do the trick, but your answer (4.0000L) is purer. We should provide a rounding function, and in practice anyone using such a function would round (or floor, or ceiling) to get to the desired precision immediately.

I'm not sure on division either but I'm sure there are precedents to look at.

The AS400 provides a useful example of the right way to do scaled decimals.

In the RPG programming language, all internal calculations (i.e. multiplication, division) are performed to the maximum precision of the intermediate result (in the multiplication example below), the intermediate result would be 4.0000L. When the intermediate result is assigned to the target scaled decimal number, the decimal precision is automatically extended or truncated to fit the target precision. One extra wrinkle in all of this is the option to "half-adjust" the intermediate value on assignment; that is to apply automatic 5/4 rounding to the precision of the target.

Yee ha! This is great input. Anyone have any other examples of what any other systems do? Anyone got a PL/I manual handy. ;)

So, if the target field is defined as numeric(4,2), the result will be 4.00L.

Since Python doesn't have types values, this is not an issue internally, but would be an issue when binding to external databases.

These are probably the kind of semantics that a scaled decimal type would require in Python also; i.e. allow unlimited precision in intermediate calculations, with a sensible set of rules for assignment to a variable of different scale and precision.

However, unlike RPG, we should probably ensure that attempts to overflow or underflow the scale result in NaN or Overflow conditions, rather than assuming the user is right and losing the significant digits.

Since this would be based on infinite-precision numbers, I don't think that this would be an issue. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

"Da Silva, Mike" wrote:

Andy Robinson wrote: For the work I do, simply taking the precision of the most precise input (4.00L)would do the trick, but your answer (4.0000L) is purer. We should provide a rounding function, and in practice anyone using such a function would round (or floor, or ceiling) to get to the desired precision immediately.

I'm not sure on division either but I'm sure there are precedents to look at.

The AS400 provides a useful example of the right way to do scaled decimals.

In the RPG programming language, all internal calculations (i.e. multiplication, division) are performed to the maximum precision of the intermediate result (in the multiplication example below), the intermediate result would be 4.0000L. When the intermediate result is assigned to the target scaled decimal number, the decimal precision is automatically extended or truncated to fit the target precision. One extra wrinkle in all of this is the option to "half-adjust" the intermediate value on assignment; that is to apply automatic 5/4 rounding to the precision of the target.

Yee ha! This is great input. Anyone have any other examples of what any other systems do? Anyone got a PL/I manual handy. ;)

So, if the target field is defined as numeric(4,2), the result will be 4.00L.

Since Python doesn't have types values, this is not an issue internally, but would be an issue when binding to external databases.

These are probably the kind of semantics that a scaled decimal type would require in Python also; i.e. allow unlimited precision in intermediate calculations, with a sensible set of rules for assignment to a variable of different scale and precision.

However, unlike RPG, we should probably ensure that attempts to overflow or underflow the scale result in NaN or Overflow conditions, rather than assuming the user is right and losing the significant digits.

Since this would be based on infinite-precision numbers, I don't think that this would be an issue. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.