[New-bugs-announce] [issue41540] Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX

[Tony Reix]

New submission from Tony Reix <tony.reix at atos.net>:

Python master of 2020/08/11

Test test_maxcontext_exact_arith (test.test_decimal.CWhitebox) checks that Python correctly handles a case where an object of size 421052631578947376 is created.

maxcontext = Context(prec=C.MAX_PREC, Emin=C.MIN_EMIN, Emax=C.MAX_EMAX)

Both on Linux and AIX, we have:
Context(prec=999999999999999999,
        rounding=ROUND_HALF_EVEN,
        Emin=-999999999999999999,
         Emax=999999999999999999, capitals=1, clamp=0, flags=[], traps=[InvalidOperation, DivisionByZero, Overflow])

The test appears in:
  Lib/test/test_decimal.py
    5665     def test_maxcontext_exact_arith(self):
and the issue (on AIX) exactly appears at:
                self.assertEqual(Decimal(4) / 2, 2)

The issue is due to code in: Objects/obmalloc.c :
void *
PyMem_RawMalloc(size_t size)
{
    /*
     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
     * Most python internals blindly use a signed Py_ssize_t to track
     * things without checking for overflows or negatives.
     * As size_t is unsigned, checking for size < 0 is not required.
     */
    if (size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);

Both on Fedora/x86_64 and AIX, we have:
 size:            421052631578947376
 PY_SSIZE_T_MAX: 9223372036854775807
thus: size < PY_SSIZE_T_MAX and _PyMem_Raw.malloc() is called.

However, on Linux, the malloc() returns a NULL pointer in that case, and then Python handles this and correctly runs the test.
However, on AIX, the malloc() tries to allocate the requested memory, and the OS gets stucked till the Python process is killed by the OS.

Either size is too small, or PY_SSIZE_T_MAX is not correctly computed:
./Include/pyport.h :
  /* Largest positive value of type Py_ssize_t. */
  #define PY_SSIZE_T_MAX ((Py_ssize_t)(((size_t)-1)>>1))

Anyway, the following code added in PyMem_RawMalloc() before the call to _PyMem_Raw.malloc() , which in turns calls malloc() :
    if (size == 421052631578947376)
        {
                printf("TONY: 421052631578947376: --> PY_SSIZE_T_MAX: %ld \n", PY_SSIZE_T_MAX);
                return NULL;
        }
does fix the issue on AIX.
However, it is simply a way to show where the issue can be fixed.
Another solution (fix size < PY_SSIZE_T_MAX) is needed.

----------
components: C API
messages: 375302
nosy: T.Rex
priority: normal
severity: normal
status: open
title: Test test_maxcontext_exact_arith (_decimal) consumes all memory on AIX
type: crash
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41540>
_______________________________________