[pypy-dev] swig + pypy - object reference counting

[Alex Pyattaev]

Well,
the point is that first I make an owned copy of the object:

%typemap(in) void* {
   Py_XINCREF($input);
   $1 = $input;
}

Here is the storage struct:
struct event{
    int code;
    void* node_tx;
    void* node_rx;
    void* packet;
    double signal_power;
    double noise_power;
    double BER;
    struct event* next;
};

For the C-code python objects are just void*, so they are perfectly safe.
Now, when I fetch the objects by my own get function i have the following 
typemap:

%typemap (out) event_t* {
    if ($1 == NULL)
    {
        $result = Py_None;
        Py_XINCREF($result);
    }
    else
    {
        $result = PyTuple_New(7);
        PyTuple_SetItem($result, 0 , PyInt_FromLong($1->code));
        PyTuple_SetItem($result, 1 , $1->node_tx);
        PyTuple_SetItem($result, 2 , $1->node_rx);
        PyTuple_SetItem($result, 3 , $1->packet);
        #ifdef PYTHON_PYPY
            Py_XINCREF($1->node_tx);
            Py_XINCREF($1->node_rx);
            Py_XINCREF($1->packet);
        #endif
        PyTuple_SetItem($result, 4 , PyFloat_FromDouble($1->signal_power));
        PyTuple_SetItem($result, 5 , PyFloat_FromDouble($1->noise_power));
        PyTuple_SetItem($result, 6 , PyFloat_FromDouble($1->BER));
        free($1);
        Py_XINCREF($result);
    }
}
As you can see, in Python I do not need to INCREF object references, but in 
PYPY I do, otherwise it crashes.

In the wrapper function it looks like this:
SWIGINTERN PyObject *_wrap_fetch_event(PyObject *SWIGUNUSEDPARM(self), 
PyObject *args) {
  PyObject *resultobj = 0;
  event_t *result = 0 ;

  if (!SWIG_Python_UnpackTuple(args,"fetch_event",0,0,0)) SWIG_fail;
  result = (event_t *)fetch_event();
  {
    if (result == NULL)
    {
      resultobj = Py_None;
      Py_XINCREF(resultobj);
    }
    else
    {
      resultobj = PyTuple_New(7);
      #ifdef PYTHON_PYPY
            Py_XINCREF($1->node_tx);
            Py_XINCREF($1->node_rx);
            Py_XINCREF($1->packet);
      #endif
      PyTuple_SetItem(resultobj, 0 , PyInt_FromLong(result->code));
      PyTuple_SetItem(resultobj, 1 , result->node_tx);
      PyTuple_SetItem(resultobj, 2 , result->node_rx);
      PyTuple_SetItem(resultobj, 3 , result->packet);

      PyTuple_SetItem(resultobj, 4 , PyFloat_FromDouble(result-
>signal_power));
      PyTuple_SetItem(resultobj, 5 , PyFloat_FromDouble(result->noise_power));
      PyTuple_SetItem(resultobj, 6 , PyFloat_FromDouble(result->BER));
      free(result);
      Py_XINCREF(resultobj);
    }
  }
  return resultobj;
fail:
  return NULL;
}

So essentially the same code works in different ways for python and pypy. IMHO 
there is a bug somewhere, but I have not time ATM to find it. And yes, it 
leaks memory like hell due to extra ref=(
On Wednesday 28 September 2011 11:11:07 Amaury Forgeot d'Arc wrote:
> 2011/9/28 Alex Pyattaev <alex.pyattaev at gmail.com>
> 
> > Hi!
> > I have a quite sophisticated program that can be summarized as follows:
> > 1. Save a pypy object pointer inside C program. Here I call Py_XINCREF
> > so
> > that
> > it does not get deleted.
> > 2. Do some logic, move this reference around C code.
> > 3. Return a python tuple via typemap, here I am probably supposed to
> > return a
> > borrowed reference. And in Python2 it works just fine. BUT. In pypy, for
> > some
> > reason, it causes segfault with following message:
> > """
> > Fatal error in cpyext, CPython compatibility layer, calling
> > PyTuple_SetItem Either report a bug or consider not using this
> > particular extension <InvalidPointerException object at 0x14a87a8>
> > 
> > RPython traceback:
> >  File "module_cpyext_api_1.c", line 28965, in PyTuple_SetItem
> >  File "module_cpyext_pyobject.c", line 1018, in CpyTypedescr_realize
> > 
> > Segmentation fault
> > """
> > If I call Py_XINCREF before returning the object, the crash does not
> > happen and the memory does not seem to be leaking (at least not
> > noticeably massive amounts of it). So it seems that PyPy is somewhat
> > incompatible with Python2 in
> > that matter.
> > If you want I could send the code example that triggers the bug (it IS
> > quite
> > large app, which might have many more bugs apart from this, but still).
> 
> Isn't PyTuple_SetItem supposed to "steal" the reference?
> In this case you'd better INCREF the object if it is globally shared.