# [Tutor] the binary math "wall"

Dave Angel davea at ieee.org
Wed Apr 21 18:31:25 CEST 2010

```Lowell Tackett wrote:
> From the virtual desk of Lowell Tackett
>
>
>
> --- On Wed, 4/21/10, Dave Angel <davea at ieee.org> wrote:
>
>
>> From: Dave Angel <davea at ieee.org>
>> Subject: Re: [Tutor] the binary math "wall"
>> To: "Lowell Tackett" <lowelltackett at yahoo.com>
>> Cc: tutor at python.org, "Steven D'Aprano" <steve at pearwood.info>
>> Date: Wednesday, April 21, 2010, 6:46 AM
>>
>>
>> Lowell Tackett wrote:
>>
>>> --- On Tue, 4/20/10, Steven D'Aprano <steve at pearwood.info>
>>>
>> wrote:
>>
>>>
>>>
>>>> From: Steven D'Aprano <steve at pearwood.info>
>>>>     <snip>
>>>>
>>>> The simplest, roughest way...hit them with a
>>>> hammer:
>>>>
>>>>
>>>>>>> round(18.15*100) == 1815
>>>>>>>
>>>>>>>
>>
>>
>>>> True
>>>>
>>>>
>>> ...when I tried...:
>>>
>>> Python 2.5.1 (r251:54863, Oct 14 2007, 12:51:35)
>>> [GCC 3.4.1 (Mandrakelinux 10.1 3.4.1-4mdk)] on linux2
>>>
>>
>>>
>>>
>>>>>> round(18.15)*100 == 1815
>>>>>>
>>>>>>
>>> False
>>>
>>> <snip>
>>>
>> But you typed it differently than Steven.  He
>> had   round(18.15*100), and you used
>> round(18.15)*100
>>
>
> As soon as I'd posted my answer I realized this mistake.
>
>
>> Very different.   His point boils down to
>> comparing integers, and when you have dubious values, round
>> them to an integer before comparing.  I have my doubts,
>> since in this case it would lead to bigger sloppiness than
>> necessary.
>>
>> round(18.154 *100) == 1815
>>
>> probably isn't what you'd want.
>>
>> So let me ask again, are all angles a whole number of
>> seconds?  Or can you make some assumption about how
>> accurate they need to be when first input (like tenths of a
>> second, or whatever)?  If so use an integer as
>> follows:
>>
>> val =  round((((degrees*60)+minutes)*60) +
>> seconds)*10)
>>
>> The 10 above is assuming that tenths of a second are your
>> quantization.
>>
>> HTH
>> DaveA
>>
>>
>>
>
> Recalling (from a brief foray into college Chem.) that a result could not be displayed with precision greater than the least precise component that bore [the result].  So, yes, I could accept my input as the arbitrator of accuracy.
>
> A scenario:
>
> Calculating the coordinates of a forward station from a given base station would require [perhaps] the bearing (an angle from north, say) and distance from hither to there.  Calculating the north coordinate would set up this relationship, e.g.:
>
> cos(3° 22' 49.6") x 415.9207'(Hyp) = adjacent side(North)
>
> My first requirement, and this is the struggle I (we) are now engaged in, is to convert my bearing angle (3° 22' 49.6") to decimal degrees, such that I can assign its' proper cosine value.  Now, I am multiplying these two very refined values (yes, the distance really is honed down to 10,000'ths of a foot-that's normal in surveying data); within the bowels of the computer's blackboard scratch-pad, I cannot allow errors to evolve and emerge.
>
> Were I to accumulate many of these "legs" into perhaps a 15 mile traverse-accumulating little computer errors along the way-the end result could be catastrophically wrong.
>
> (Recall that in the great India Survey in the 1800's, Waugh got the elevation of Mt. Everest wrong by almost 30' feet for just this exact same reason.)  In surveying, we have a saying, "Measure with a micrometer, mark with chalk, cut with an axe".  Accuracy [in math] is a sacred tenet.
>
> So, I am setting my self very high standards of accuracy, simply because those are the standards imposed by the project I am adapting, and I can require nothing less of my finished project.
>
>
If you're trying to be accurate when calling cos, why are you using
degrees?  The cosine function takes an angle in radians.  So what you
need is a method to convert from deg/min/sec to radians.  And once you
have to call trig, you can throw out all the other nonsense about
getting exact values.  Trig functions don't take arbitrary number
units.  They don't take decimals, and they don't take fractions.  They
take double-precision floats.

Perhaps you don't realize the amount of this quantization error we've
been talking about.  The double type is 64bits in size, and contains the
equivalent of about 18 decimal digits of precision.  (Assuming common
modern architectures, of course)

Your angle is specified to about 5 digits of precision, and the distance
to 7.  So it would take a VERY large number of typical calculations for
errors in the 18th place to accumulate far enough to affect those.

The real problem, and one that we can't solve for you, and neither can
Python, is that it's easy to do calculations starting with 8 digits of
accuracy, and the result be only useful to 3 or 4.  For example, simply
subtract two very close numbers, and use the result as though it were
meaningful.

precision of a calculation he was doing.  I had written the math
microcode of the machine he was using (from add and subtract, up to
trigs and logs, I wrote it all).  So I analyzed his problem, pointed out
his errors, and sent him away happy.

He was trying to calculate the difference between a perfectly flat
machine table, and one that was leval at all points, following the
curvature of the earth.  With a table of 200 feet, the distinction was a
measurable one, and he was trying to calculate it.

His effective math was calculating the distance from the center of the
earth to each end of the table, and subtracting.  So he was subtracting
two very close numbers.  I did some simple geometry (using similar
triangles), and showed him another way to do the calculation.

Anyway, in your case binary floating point is irrelevant.  You've got to
do your own error analysis of every step in the calculation to
understand how much you can trust the result.  And it's unlikely the
"errors" in the math package will be the limiting factor.

DaveA

```