[Hack] Import binary extensions from zipfiles, windows only

[Thomas Heller]

"PJDM" <PeterMayne at ap.spherion.com> writes:

> I'm trying to make ZopeX3 start faster by zipping up the "zope"
> directory. (Because this will be stored on a CD, having less data to
> read will make it quicker to start.)
>
> The standard python zipimporter won't work with the ZopeX3 .pyd files,
> so zipextimporter came along at just the right time. However, it
> doesn't seem to work for me (Python 2.3.4 (#53, May 25 2004, 21:17:02)
> [MSC v.1200 32 bit (Intel)] on win32, Windows XP SP2).
>
> Sample script:
> ----
> import zipextimporter
> zipextimporter.install()
>
> import sys
> print 'Hooks:\n', sys.path_hooks
>
> sys.path.append(r'C:\tmp\zope.zip')
> print 'Path:\n', sys.path
>
> # Try an innocuous import first to make sure we have the path correct.
> import zope.server
> print 'Server file:\n', zope.server.__file__
>
> # Now try a .pyd import.
> import zope.thread._zope_thread
> print 'Thread file:\n', zope.thread._zope_thread.__file__
> ----
>
> Output:
> ----
> Hooks:
> [<class 'zipextimporter.ZipExtensionImporter'>, <type
> 'zipimport.zipimporter'>]
> Path:
> ['C:\\tmp', 'C:\\WINDOWS\\system32\\python23.zip', 'C:\\tmp',
> 'C:\\opt\\Python23\\DLLs', 'C:\\opt\\Python23\\lib',
> 'C:\\opt\\Python23\\lib\\plat-win', 'C:\\opt\\Python23\\lib\\lib-tk',
> 'C:\\opt\\Python23', 'C:\\opt\\Python23\\lib\\site-packages',
> 'C:\\tmp\\zope.zip']
> Server file:
> C:\tmp\zope.zip\zope\server\__init__.py
> Traceback (most recent call last):
> File "zei.py", line 15, in ?
> import zope.thread._zope_thread
> ImportError: No module named _zope_thread
> ----
>
> The zope.zip file contains zope.thread:
> ----
> $ unzip -l zope.zip |grep zope/thread
> 0  11-08-04 11:00   zope/thread/
> 0  07-03-04 04:34   zope/thread/DEPENDENCIES.cfg
> 65  07-28-04 22:52   zope/thread/SETUP.cfg
> 994  07-03-04 04:34   zope/thread/tests.py
> 748  11-08-04 11:00   zope/thread/tests.pyc
> 748  11-08-04 11:00   zope/thread/tests.pyo
> 20480  11-07-04 00:54   zope/thread/_zope_thread.pyd
> 7838  08-17-04 17:20   zope/thread/__init__.py
> 7808  11-08-04 11:00   zope/thread/__init__.pyc
> 7808  11-08-04 11:00   zope/thread/__init__.pyo
> ----

zipextimporter, as published, doesn't handle extensions in packages.
The patch I attached at the end, may fix this - I tested it with PIL.

It may work with zope, or may not - depends on how the extensions are
structured.  If there are extensions/pyds that link to other
extensions/pyds, then it will most certainly not work.

If it doesn't work, you could still try the py2exe approach:
py2exe bundles the .py modules in a zip file, and also inserts loaders
for extensions into the zip file.  The loaders load the .pyd extensions
from the file system.

Thomas

<patch>
Index: zipextimporter.py
===================================================================
RCS file: /cvsroot/py2exe/py2exe/hacks/memimp/zipextimporter.py,v
retrieving revision 1.2
diff -c -r1.2 zipextimporter.py
*** zipextimporter.py	16 Dec 2004 09:29:42 -0000	1.2
--- zipextimporter.py	4 Jan 2005 20:08:08 -0000
***************
*** 50,55 ****
--- 50,56 ----
          if result:
              return result
  
+         fullname = fullname.replace(".", "\\")
          for s in self._suffixes:
              if (fullname + s) in self._files:
                  return self
***************
*** 60,72 ****
              return zipimport.zipimporter.load_module(self, fullname)
          except zipimport.ZipImportError:
              pass
          for s in self._suffixes:
              path = fullname + s
              if path in self._files:
                  # XXX should check sys.modules first ? See PEP302 on reload
                  # XXX maybe in C code...
                  code = self.get_data(path)
!                 mod = _memimporter.import_module(code, "init" + fullname)
                  mod.__file__ = "%s\\%s" % (self.archive, path)
                  mod.__loader__ = self
                  if _memimporter.get_verbose_flag():
--- 61,75 ----
              return zipimport.zipimporter.load_module(self, fullname)
          except zipimport.ZipImportError:
              pass
+         initname = fullname.split(".")[-1]
+         fullname = fullname.replace(".", "\\")
          for s in self._suffixes:
              path = fullname + s
              if path in self._files:
                  # XXX should check sys.modules first ? See PEP302 on reload
                  # XXX maybe in C code...
                  code = self.get_data(path)
!                 mod = _memimporter.import_module(code, "init" + initname)
                  mod.__file__ = "%s\\%s" % (self.archive, path)
                  mod.__loader__ = self
                  if _memimporter.get_verbose_flag():
<patch/>