[capi-sig] Embedded python, adding a path to sys.path before import, strange observations

[Todd Lyons]

I was well on my way to sending a help request about "why can I not
add a path to sys.path" in an embedded python instance (that I'm
adding to a c app).  I have test code from *three* different methods
that should add a path to the sys.path object, but none of them
_seemed_ to work.  I highlight that word because the way I was testing
is what seems to be flawed.  All three methods turn out to work, but
the way I was testing appears to be flawed.

I was testing by comparing the output of Py_GetPath() before I
adjusted the path, and then after I adjusted the path.  It always
shows the same.

On a whim, I added a call to load a quickie python script in that
extra subdirectory after I adjusted the path and I was FLOORED when
loaded.  I've been working on this for days thinking that it wasn't
working because the output of Py_GetPath() did not indicate the new
path I was adding.

Question A: Is it normal that Py_GetPath() doesn't show the adjusted path?

Method 1: PySys_GetObject("path"), PyList_Insert(), PySys_SetObject()
with the new object

Method 2: PyRun_SimpleString("import sys" "sys.path.append('blah')")

Method 3: Py_GetPath(),append the syspath and new path together,
PySys_SetPath() with the new string

Question B: All three seem to work equally well, is there a reason to
choose one over the other?

All positive guidance and suggestions are welcome.  Testing C code is
quoted below, as well as the sample output from running it and an
strace trimmed output to show that it does appear to be loading the
module.  Thanks for any comments, observations, or explanations.

...Todd


py_testc:

#include <Python.h>

int main (int argc, char *argv[])
{
  char *path, *eximpath = "/etc/exim";
  PyObject *syspath, *pName, *pModule;

  Py_SetProgramName("application");
  Py_Initialize();
  if (!Py_IsInitialized())
  {
    printf("Error initializing Python interpreter\n");
    return 1;
  }
  path = Py_GetPath();
  printf("Python search path is:\n%s\n", path);

  syspath = PySys_GetObject("path");
  if (syspath == 0)
    printf("Python failed to import sys\n");

  printf("Adding %s\n", eximpath);
  pName = PyString_FromString(eximpath);
  if (PyList_Insert(syspath, 0, pName))
    printf("Error inserting extra path into sys.path list\n");
  if (PySys_SetObject("path", syspath))
    printf("Error setting sys.path object\n");

  pName = PyString_FromString("mod_python");
  pModule = PyImport_Import(pName);
  Py_DECREF(pName);
  if (pModule == NULL)
    printf("Unable to import module\n");
  else
    printf("Loaded module\n");

  /* Still shows wrong path! But module still loads above. */
  path = Py_GetPath();
  printf("Modified Python search path is:\n%s\n", path);

  Py_DECREF(syspath);
  Py_Finalize();
  return 0;
}
// vim:sw=2 ts=2 expandtab

[todd at tlyons ~/python]$ ./py_test
Python search path is:
/usr/lib/python2.7/:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload
Adding /etc/exim
Loaded module
Modified Python search path is:
/usr/lib/python2.7/:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload

[todd at tlyons ~/python]$ strace ./py_test 2>&1 | grep mod_python
stat64("/etc/exim/mod_python", 0xbf9dfa58) = -1 ENOENT (No such file
or directory)
open("/etc/exim/mod_python.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No
such file or directory)
open("/etc/exim/mod_pythonmodule.so", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
open("/etc/exim/mod_python.py", O_RDONLY|O_LARGEFILE) = 3
open("/etc/exim/mod_python.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No
such file or directory)
unlink("/etc/exim/mod_python.pyc")      = -1 ENOENT (No such file or directory)
open("/etc/exim/mod_python.pyc",
O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0100644) = -1 EACCES
(Permission denied)

--
The total budget at all receivers for solving senders' problems is $0.
 If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine