[Python-Dev] Safely importing zip files with C extensions

[Vinay Sajip]

> This quote is here to stop GMane complaining that I'm top-posting. Ignore.

I've already posted this to distutils-sig, but thought that it might be of
interest to readers here as it relates to importing C extensions ...

zipimport is great, but there can be issues importing software that contains
C extensions. But the new wheel format (PEP 427) may give us a better way of
importing zip files containing C extensions. Since wheels are .zip files, they
can sometimes be used to provide functionality without needing to be installed.
But whereas .zip files contain no convention for indicating compatibility with
a particular Python, wheels do contain this compatibility information. Thus, it
is possible to check if a wheel can be directly imported from, and the wheel
support in distlib allows you to take advantage of this using the mount() and
unmount() methods. When you mount a wheel, its absolute path name is added to
sys.path, allowing the Python code in it to be imported. (A DistlibException is
raised if the wheel isn't compatible with the Python which calls the mount()
method.)

You don't need mount() just to add the wheel's name to sys.path, or to import
pure-Python wheels, of course. But the mount() method goes further than just
enabling Python imports - any C extensions in the wheel are also made available
for import. For this to be possible, the wheel has to be built with additional
metadata about extensions - a JSON file called EXTENSIONS which serialises an
extension mapping dictionary. This maps extension module names to the names in
the wheel of the shared libraries which implement those modules.

Running unmount() on the wheel removes its absolute pathname from sys.path and
makes its C extensions, if any, also unavailable for import.

Wheels built with the new "distil" tool contain the EXTENSIONS metadata, so can
be mounted complete with C extensions:

$ distil download -d /tmp simplejson
Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
    63KB @  73 KB/s 100 % Done: 00:00:00
Unpacking ... done.
$ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
The following packages were built:
  /tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distlib.wheel import Wheel
>>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>>> w.mount()
>>> import simplejson._speedups
>>> dir(simplejson._speedups)
['__doc__', '__file__', '__loader__', '__name__', '__package__',
 'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
>>> simplejson._speedups.__file__
'/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>>> 

Does anyone see any problems with this approach to importing C extensions from
zip files?

Regards,

Vinay Sajip