# [Tutor] the binary math "wall"

Steven D'Aprano steve at pearwood.info
Wed Apr 21 01:39:30 CEST 2010

```On Wed, 21 Apr 2010 02:58:06 am Lowell Tackett wrote:
> I'm running headlong into the dilemma of binary math representation,
with game-ending consequences, e.g.:
> >>> 0.15
>
> 0.14999999999999999
>
> Obviously, any attempts to manipulate this value, under the misguided
> assumption that it is truly "0.15" are ill-advised, with inevitable

That really depends on what sort of manipulation you are doing.

>>> x = 0.15
>>> x
0.14999999999999999
>>> x*100 == 15
True

Seems pretty accurate to me.

However:

>>> 18.15*100 == 1815
False

The simplest, roughest way to fix these sorts of problems (at the risk
of creating *other* problems!) is to hit them with a hammer:

>>> round(18.15*100) == 1815
True

[...]
> What I'm shooting for, by the way, is an algorithm that converts a
> deg/min/sec formatted number to decimal degrees.  It [mostly] worked,
> until I stumbled upon the peculiar cases of 15 minutes and/or 45
> minutes, which exposed the flaw.

I'm afraid that due to the nature of floating point, this is a hard
problem. Even the professionals at Hewlett-Packard's scientific
calculator division don't always get it right, and they are *extremely*
careful:

The best result I can suggest is, change the problem! Don't pass
degrees-minutes-seconds around using a floating point value, but as a
tuple with distinct (DEG, MIN, SEC) integer values. Or create a custom
class.

But if you really need D.MMSS floats, then something like this should be
a good start.:

def dms2deg(f):
"""Convert a floating point number formatted as D.MMSS
into degrees.
"""
mmss, d = math.modf(f)
assert d == int(f)
if mmss >= 0.60:
raise ValueError(
'bad fractional part, expected < .60 but got %f' % mmss)
mmss *= 100
m = round(mmss)
if m >= 60:
raise ValueError('bad minutes, expected < 60 but got %d' % m)
s = round((mmss - m)*100, 8)
if not 0 <= s < 60.0:
raise ValueError('bad seconds, expected < 60.0 but got %f' % s)
return d + m/60.0 + s/3600.0

>>> dms2deg(18.15)
18.25
>>> dms2deg(18.1515)
18.254166666666666

which compares well to my HP-48GX:

18.15 HMS->

gives 18.25, and:

18.1515 HMS->

gives 18.2541666667.

Note though that this still fails with some valid input. I will leave
fixing it as an exercise (or I might work on it later, time
permitting).

--
Steven D'Aprano
```