[Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))

[Thomas Heller]

Am 30.01.2014 14:23, schrieb Paul Moore:
>
> First of all, it is not possible to load a DLL into a process' memory
> [2, 3] unless it is stored as a file in the filesystem.
>
[...]
>
> [2] I'm talking from a Windows perspective here. I do not have
> sufficient low-level knowledge of Unix to comment on that case. I
> suspect that the issues are similar but I defer to the platform
> experts.
>
> [3] There is, I believe, code "out there" on the internet to map a DLL
> image into a process based purely in memory, but I think it's a fairly
> gross hack. I have a suspicion that someone - possibly Thomas Heller -
> experimented with it at one time, but never came up with a viable
> implementation. There's also TCL's tclkit technology, which *may*
> support binary extensions, and may be worth a look, but TCL has
> virtual filesystem support built in quite deep in the core, so how it
> works may not be applicable to Python.

Well, py2exe uses code that loads dlls from zip-files directly into
process memory and executes it.  This is used for 'single-file'
applications that py2exe creates; it is triggered by the appropriate
'bundle_files' option.

It works quite well for some applications (wx apps for example), but
not for others.
Applications that use numpy for example crashes, but the reason is
that py2exe *also* puts the mkl-library (or maybe even more stuff like 
windows dlls) that numpy needs into the zip.
So, the trick is to only load python-extensions and not arbitrary
dlls from the zip.

Changing the packing process so that py2exe leave these
non-python dlls in the file system makes the exe work again. At least
now the range of apps that work reliably is much larger.

Anyway; the py2exe installer installs the python-extension and 
correspinding py-module that implements this magic, so anyone can
experiment with it; it is called 'zipextimporter.py'.
Here's the start of the docstring:

r"""zipextimporter - an importer which can import extension modules from 
zipfiles

This file and also _memimporter.pyd is part of the py2exe package.

Overview
========

zipextimporter.py contains the ZipExtImporter class which allows to
load Python binary extension modules contained in a zip.archive,
without unpacking them to the file system.

Call the zipextimporter.install() function to install the import hook,
add a zip-file containing .pyd or .dll extension modules to sys.path,
and import them.

It uses the _memimporter extension which uses code from Joachim
Bauch's MemoryModule library.  This library emulates the win32 api
function LoadLibrary.
"""


I wanted to post this to clarify the current state.


Thomas