Parent module not loaded error

[Graham Dumpleton]

On Jan 14, 9:41 pm, Ståle Undheim <staa... at gmail.com> wrote:
> On Jan 14, 11:31 am, Graham Dumpleton <Graham.Dumple... at gmail.com>
> wrote:
>
>
>
> > On Jan 14, 9:20 pm, Ståle Undheim <staa... at gmail.com> wrote:
>
> > > I have a pretty strange error that I can't figure out the cause off.
> > > This is in a Django app.
>
> > > I am using berkelydb, with secondary databases for indexing. The
> > > secondary databases are associated with a callback that uses cPickle
> > > to serialize index values. The problem is that cPickle.dumps(value)
> > > fails when I run it throughmod_wsgior mod_python on the deployment
> > > server (Ubuntu Linux 8.10), but works fine when when I use Djangos
> > > runserver on the same computer. Also, on my own computer (Ubuntu Linux
> > > 8.10), it works fine through mod_python.
>
> > > Each time I call on cPickle.dumps(value) I get:
> > > SystemError("Parent module 'd4' not loaded",)
>
> > > Using sys.exc_info() I get no type or traceback, just that exception.
> > > The callstack as far as I can figure is:
>
> > > django
> > > view
> > > bsddb.put
> > > indexer callback
> > > cPickle.dumps
>
> > > cPickle.loads works fine within the callback, I can also use
> > > cPickle.dumps() outside the indexer callback. But inside the callback,
> > > I need to use normal pickle instead. I just want to know why. I am
> > > assuming it has something to do with the fact that I go from python to
> > > C (bsddb) back to python (indexer callback) and back to C (cPickle).
> > > It is still strange then that cPickle.loads() work, but not
> > > cPickle.dumps(). I am also only storing integer values when I get this
> > > error, so no fancy list, dictionaries or objects.
>
> > Where is module 'd4'? Is that one of yours defined inside of your
> > Django project?
>
> d4 is the root project name. This message occurs in d4/views.py.
>
> Here is how I set up my wsgi.py - which is in the d4/ folder:
>   sys.path.append(path.join(path.dirname(__file__), '..'))
>
>   os.environ['DJANGO_SETTINGS_MODULE'] = 'd4.settings'
>   application = django.core.handlers.wsgi.WSGIHandler()
>
> > One of the problems with Django runserver is that the parent of the
> > site directory and the site directory are effectively starting points
> > for searching for modules, even though neither is explicitly listed in
> > sys.path. The parent is used because runserver does momentarily add
> > parent to sys.path to import site package root, after which it removes
> > directory from sys.path. The site directory itself is used because for
> > runserver that is the current working directory and imports look there
> > automatically.
>
> > End result is that modules inside of site can be imported either via
> > site package root, or direct. For example, if d4.py was at same level
> > as settings.py file and site was called mysite, you could import it as
> > mysite.d4 or just d4. This is bad because means you could end up with
> > two copies of a module imported under the different name references.
>
> Everywhere in the code, the imports refer to d4.something.something.
> All urls.py files refer to the full package names. The test code where
> I am running this is actually quit minimal.
>
> > If they way the pickle was done in application was such that reference
> > was via d4, and then when you deployed to mod_python or mod_wsgi you
> > only added parent directory of site to Python path, then it will not
> > be able to find d4, since at that point would only be found as
> > mysite.d4.
>
> I use "import cPickle as pickle" at the top off my views file. So I
> don't reference imports from other files.
>
> > If this is the case, add the site directory to Python path by setting
> > PythonPath for mod_python or sys.path in WSGI script file for
> > mod_wsgi. Better still, fix up your code so module references in code,
> > ie., urls.py, or anywhere else, are always done by site package root
> > reference and not direct. Ie., such that it uses mysite.d4 and not
> > just d4.
>
> > Would that seem about right?
>
> Unfortunatly, at least to me, it seems that all code is allready
> correctly using packagenames. The views.py file where I am testing
> this is a simple file that doesn't depend on anything else in the
> project. It only imports a set of system modules for testing and info.
> The only "dependency" is really the urls.py file, which does refer to
> all the views as "d4.views.somemethod".

As a test, just prior to where unpickle is done, do:

  import sys
  import d4

  print >> sys.stderr, "d4.__name__", dr.__name__
  print >> sys.stderr, "d4.__file__", dr.__file__

This will just confirm that manual import works and what values of
those attributes are. The messages will end up in Apache error log for
that virtual host or main server error log as appropriate.

Graham