[New-bugs-announce] [issue25604] [Minor] bug in integer true division algorithm

[Mark Dickinson]

New submission from Mark Dickinson:

There's a harmless but annoying (for code readers) bug in the code for true division of (long) integers.  In `long_true_divide` in `Objects/longobject.c`, we have the following code for manipulating the bits of a 55- or 56-bit long to get something that will be exactly representable as a float:

    /* The number of extra bits that have to be rounded away. */
    extra_bits = MAX(x_bits, DBL_MIN_EXP - shift) - DBL_MANT_DIG;
    assert(extra_bits == 2 || extra_bits == 3);

    /* Round by directly modifying the low digit of x. */
    mask = (digit)1 << (extra_bits - 1);
    low = x->ob_digit[0] | inexact;
    if (low & mask && low & (3*mask-1))
        low += mask;
    x->ob_digit[0] = low & ~(mask-1U);

    /* Convert x to a double dx; the conversion is exact. */
    [...]

The last code line above is supposed to be masking out all the bits that we don't want to keep. Instead, it's masking out all but one of those bits.  The line

    x->ob_digit[0] = low & ~(mask-1U);

should be replaced with

    x->ob_digit[0] = low & ~(2*mask-1U);

As it stands, the comment about the conversion to double being exact is false. (I've verified this by converting x both to a 64-bit unsigned integer and to a double and checking that the integer and double don't always match in value; with the fix above, they do.)

I don't *think* this bug actually affects the output on any platform whose arithmetic and ldexp functions do correct rounding with round-ties-to-even: the extra bit will always get rounded away (in the correct direction) by either the conversion to float (for the non-subnormal case) or by the ldexp operation (for the subnormal case). Still, the bug makes the code a bit more susceptible to platform arithmetic quirks.

----------
assignee: mark.dickinson
components: Interpreter Core
messages: 254523
nosy: mark.dickinson
priority: low
severity: normal
status: open
title: [Minor] bug in integer true division algorithm
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25604>
_______________________________________