Pyrex / py2exe import interaction

[Thomas Heller]

From: "Clark C . Evans" <cce at clarkevans.com>
> Hello.  I'm having difficulty with using a combination
> of pyrex and py2exe; the problem occurs when the module
> compiled with pyrex has an "import".  The program then
> works just fine until the calling script is packaged
> up with py2exe, then you get the following:
> 
>     Traceback (most recent call last):
>       File "<string>", line 1, in ?
>       File "imputil.pyc", line 103, in _import_hook
>       File "<string>", line 52, in _import_top_module
>       File "imputil.pyc", line 216, in import_top
>       File "imputil.pyc", line 267, in _import_one
>       File "<string>", line 163, in get_code
>       File "imputil.pyc", line 89, in _import_hook
>       File "imputil.pyc", line 152, in _determine_import_context
>     AttributeError: 'module' object has no attribute 'get'
> 
> Anyway, both of these programs play all sorts of tricks to
> get importing to work, so I figure that interaction of the
> two importing strategies is causing difficulty.
> 
> Following is the source code for the various files that 
> I'm using.  Further, these files are packged up at
> http://clarkevans.com/tmp/prob.tgz ; the build was performed
> with ActiveState 2.2.1 on Win2K using py2exe 0.3.3 and 
> pyrex 0.4 compiled using mingw32, gcc 2.95.3-6
> 
> Any help would be very cool.  My actual setup is much more
> complicated than this, but I tried to make the smallest 
> example which exhibits the behavior.   Also, I've had to 
> split "setup.py" into two distinct files since I didn't 
> know how to make them both work together.
Here's a combined setup script you could use:
-----
from distutils.core import setup, Extension
import py2exe
from Pyrex.Distutils import build_ext

setup(
    name = "Demos",
    scripts=["test.py"],
    ext_modules=[Extension("testmod", ["testmod.pyx"])],
    cmdclass = {"build_ext": build_ext}
    )
-----

but now to the more interesting questions:

It seems to be a problem (as you also have discovered in later posts)
in the way Pyrex does the importing.
A nice way to debug this (which also works without py2exe, and
without windows) is to check it with using imputil's own import
hook in the interactive Python interpreter:

C:\evans\prob>py22
Python 2.2.1 (#34, Apr  9 2002, 19:34:33) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imputil
>>> imputil._test_revamp() # install the hook
>>> import testmod
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\python22\lib\imputil.py", line 103, in _import_hook
    top_module = self._import_top_module(parts[0])
  File "C:\python22\lib\imputil.py", line 187, in _import_top_module
    module = self.fs_imp.import_from_dir(item, name)
  File "C:\python22\lib\imputil.py", line 536, in import_from_dir
    result = self._import_pathname(_os_path_join(dir, fqname), fqname)
  File "C:\python22\lib\imputil.py", line 569, in _import_pathname
    return importFunc(filename, finfo, fqname)
  File "C:\python22\lib\imputil.py", line 602, in import_file
    module = imp.load_module(fqname, fp, filename, self.desc)
  File "C:\python22\lib\imputil.py", line 89, in _import_hook
    parent = self._determine_import_context(globals)
  File "C:\python22\lib\imputil.py", line 152, in _determine_import_context
    if not globals or not globals.get('__importer__'):
AttributeError: 'module' object has no attribute 'get'
>>> import pdb
>>> pdb.pm()
> c:\python22\lib\imputil.py(152)_determine_import_context()
-> if not globals or not globals.get('__importer__'):
(Pdb) l
147             will be looked for within that package. The context could also be None,
148             meaning there is no context -- the module should be looked for as a
149             "top-level" module.
150             """
151
152  ->         if not globals or not globals.get('__importer__'):
153                 # globals does not refer to one of our modules or packages. That
154                 # implies there is no relative import context (as far as we are
155                 # concerned), and it should just pick it off the standard path.
156                 return None
157
(Pdb) print globals
<module 'testmod' (built-in)>
(Pdb)

So we see 'globals' is a module, not a dictionary (as it should be).

IMO Pyrex should call PyImport_ImportModuleEx(), which AFAIK uses the import hook,
and is also easier to use than to retrieving the __import__ from builtins,
and it should probably call it with the result from PyEval_GetGlobals(),
but I'll leave this to Greg.

HTH,

Thomas