Problems with mod_pyapache and cPickle

[Andrew Csillag]

When using mod_pyapache, an interpreter is initialized and finalized for
each request.  The problem is when using cPickle for serialization,
cPickle, upon unpickling, creates a cache of objects that it loads out
of modules so that it doesn't have to repeatedly import modules to get
at classes it has already de-serialized, which is cool.  The problem is
that when the second time (in a different interpreter than the first)
that cPickle is loaded, it's class cache still references classes from
the first interpreter, which when used, causes all sorts of problems
since their __builtins__ module has been wrecked and all it's external
module references are now wrecked too, making them virtually unusable.

I don't believe that this problem is specific to the cPickle module, but
is problematic for any C module that contains references to python
objects (C objects appear to be immune).  

For the cPickle module, I've written a patch to make it work correctly,
but it requires a python wrapper module to make it "transparent".  What
I did is this:  made a module function that destroys the class cache
(called class_map in the code), and creates a new, empty one.  Then, I
have a wrapper module that imports cPickle and calls the function to
clear the cache.

--
Added this function to cPickle.c:
--
static PyObject*
clear_map(PyObject *self, PyObject *args) {

  PyObject *new_map;
  PyObject *old_map;

  /* check arguments */
  if (!PyArg_ParseTuple(args, ""))
    return NULL;

  /* try to create a new dict */
  new_map=PyDict_New();
  if (!new_map)
    return NULL;

  old_map=class_map;
  class_map=new_map;
  Py_DECREF(old_map);
  Py_INCREF(Py_None);
  return Py_None;
}

--

and added this to the cPickle_methods structure in cPickle.c

--
{"clear_map", (PyCFunction)clear_cache,   1,
 "clear_map() -- clear the class map\n"
},
--

Then the simple python wrapper module

--
import cPickle
cPickle.clear_map()
--

This alleviates the problem and makes it work.  Outside of not
finalizing interpreters (which alleviates it somewhat, but makes the
code run in restricted mode since __builtins__ of the unpickled objects
isn't the same as the "current" __builtins__), I don't know of any other
way to fix this.

This is likely an issue in other apps that use multiple interpreters and
use C extensions which maintain references to python objects also. 
Might there be a way (although likely not trivial) to fix this for all
cases?


-- 
"There are two major products that come out of Berkeley: 
LSD and UNIX. We don't believe this to be a coincidence." 
- Jeremy S. Anderson