[New-bugs-announce] [issue9687] dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.py fails) when built against gdbm (int vs Py_ssize_t)

[Dave Malcolm]

New submission from Dave Malcolm <dmalcolm at redhat.com>:

With a clean build of release27-maint (r84317), test_dbm.py fails on ppc64 with this error:
  File "test_dbm.py", line 24, in test_keys
    self.assert_(k in self.d)
AssertionError

I'm building gainst gdbm-1.8.0 (specifically, on a prerelease of RHEL6, with gdbm-devel-1.8.0-36.el6.ppc64)

All of the headers define datum as:
      typedef struct {
        char *dptr;
        int   dsize;
      } datum;

Note the use of "int" for dsize.

This fragment of code in python's Modules/dbmmodule.c:dbm_contains:
          if (PyString_AsStringAndSize(v, (char **)&key.dptr,
                                       (Py_ssize_t *)&key.dsize)) {
                  return -1;
          }
 appears to assume that
  sizeof(datum.dsize) == sizeof(Py_ssize_t)
which is not correct on these architectures:

(gdb) p sizeof(key.dsize)
$25 = 4
(gdb) p sizeof(Py_ssize_t)
$26 = 8

On ppc64, when PyString_AsStringAndSize writes the 0x00000000000000001 value for the ob_size of "a" to &key.dsize, I believe the 0x00000000 part is written to &key.size, and the 0x00000001 part is written to the 4 bytes following it, due to the incorrect cast from (int*) to (Py_ssize_t*)

Thankfully
(gdb) p sizeof(key)
$28 = 16
so it writes this value to padding within the "datum key", rather than corrupting the stack.

The dbm_fetch() invocation is thus passed a 0 dsize, and doesn't find the key, hence the test fails.

The various other uses with that source file appear correct:
(i) there are various PyArg_Parse* calls using s#, with int, which is correct, given the absence of the PY_SSIZE_T_CLEAN macro.
(ii) there are various calls of PyString_FromStringAndSize(, datum.dsize), which I believe is correct: I believe the compiler will coerce this int to the wider Py_ssize_t type.

I'm attaching a patch which (I hope) correctly coerces the size of the key from Py_ssize_t to "int" within gdb_contains.

----------
components: Extension Modules
files: fix-dbm_contains-on-64bit-bigendian.patch
keywords: patch, patch
messages: 114934
nosy: dmalcolm
priority: normal
severity: normal
stage: patch review
status: open
title: dbmmodule.c:dbm_contains fails on 64bit big-endian (test_dbm.py fails) when built against gdbm (int vs Py_ssize_t)
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file18644/fix-dbm_contains-on-64bit-bigendian.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9687>
_______________________________________