Problem with embedding - can only make about 150 calls...

[Mads Bondo Dydensborg]

Hi there

I am in the process of embedding Python in a program. I am running into
some weird problems with my program/Python crashing.

I have constructed a minimal program to show my problem (the real problem 
shows up in a larger project, but is similar):


------------------- test.c --------------------
#include <Python.h>
#include <stdio.h>

static PyObject * MyTest(PyObject * self, PyObject *args) {
  return Py_None;
}

static PyMethodDef mytest_methods[] = {
  {"MyTest",  MyTest,      METH_VARARGS},
  {NULL,          NULL}           /* sentinel */
};

int main(int argc, char ** argv) {
  if (argc != 2) {
    fprintf(stderr, "Usage: ./test <count>\n");
    exit(1);
  }
  
  int count = atoi(argv[1]);
  
  Py_Initialize();
  PyImport_AddModule("MyTest");
  Py_InitModule("MyTest", mytest_methods);
  PyRun_SimpleString("import MyTest");
  
  FILE * tmpf = fopen("MyTest.py", "r");
  bool result = false;
  if (tmpf) {
    result = (0 == PyRun_SimpleFile(tmpf, "MyTest.py"));
    fclose(tmpf);
  } 
  if (result) {
    printf("init succes\n");
  } else {
    printf("init failure\n");
  }

  char buf[256];
  sprintf(buf, "PyMyTest(%i)", count);
  
  printf("Executing %s\n", buf);

  if (!PyRun_SimpleString(buf)) {
    printf("Surpringsly, no error running PyMyTest\n");
  }

}
-----------------------------------------------

And, here is the script file that goes with it:

------------------- MyTest.py -----------------
def PyMyTest(n):
    if (n > 0):
        print "n = ", n, " - calling MyTest.MyTest recursivly"
        MyTest.MyTest()
        PyMyTest(n-1)
-----------------------------------------------

I compile it like this:

gcc -Wall -o test test.cc -I/usr/include/python2.0 /usr/lib/python2.0/config/libpython2.0.a -ldl -lpthread -lutil

And, when I run it, with parameters 100, it works great:

[madsdyd at amigos48 src]$ ./test 100
init succes
Executing PyMyTest(100)
n =  100  - calling MyTest.MyTest recursivly
n =  99  - calling MyTest.MyTest recursivly
....
n =  2  - calling MyTest.MyTest recursivly
n =  1  - calling MyTest.MyTest recursivly
Surpringsly, no error running PyMyTest

BUT, when I do a 1000, I get:

[madsdyd at amigos48 src]$ ./test 1000
init succes
Executing PyMyTest(1000)
n =  1000  - calling MyTest.MyTest recursivly
n =  999  - calling MyTest.MyTest recursivly
....
n =  848  - calling MyTest.MyTest recursivly
n =  847  - calling MyTest.MyTest recursivly
n =  846  - calling MyTest.MyTest recursivly
Lagersegmentfejl (core dumped)
[madsdyd at amigos48 src]$ 
(The last line is danish for segmentation error).

Now, what gives?
It is not the recursion depth - if you remove the MyTest.MyTest call, and
call with 1000, you will exceed the recursion depth, but in more than 900
calls, not 154.

But, what is it? Do I need to do something differently? I have looked at
about 3 different versions of the extending and embedding document, and I
am unsure how to proceed. It is a real showstopper.

Note; The above program does not show it, but it does not need to be 150
calls at once - calling 25 will work, the next 125 as well, and then it
will fail at about 154, 155.

This problem occurs for me with a Mandrake 8.0 installation, which means
[madsdyd at amigos48 src]$ gcc -v
Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux/2.96/specs
gcc version 2.96 20000731 (Linux-Mandrake 8.0 2.96-0.48mdk)
and Python 2.0.

All help, information, flames, sarcasm in my general direction will be
appriciated. I am stuck in an area I thought I understood, which is
extremely annoying. :-(

Thanks,

Mads

-- 
Mads Bondo Dydensborg.                               madsdyd at challenge.dk
64 GB PAE mode works just fine on my 8GB RAM, 8-way Xeon box [..]
                      
                                 - Ingo Molnar, Linux Kernel List, 1999-11-03