[Distutils] Re: Add multiple frozen modules

[James C. Ahlstrom]

I think I finally understand the work Greg, Gordon and the distutils
sig folks have done with building the Python library into the
Python binary.  But I am still having problems.  I still think
we may need to be able to have multiple frozen module arrays.

When Python starts up (Py_Initialize()) it:
  Performs an import of exceptions.py.
  Calls initsigs().
  The implication is that someday something_else.py may be imported
here.
  Calls initmain().
  Performs an import of site.py.

Greg's (wonderful) imputil.py must be "turned on" in site.py.  So
the conclusion is that the default Python import logic can not
be replaced until after it has been used to find exceptions.py and
site.py.  This seems to be unfortunate.  It would be nice
to replace it for all modules.

It would be nice if all this were identical on Unix and Windows,
and that no C compiler were required.  And that Python versions
could be changed by replacing python15.dll/.so.  And that having
a frozen "__main__" still worked...

Here are a few ideas:
1)  Use freeze to include exceptions.py, site.py and imputil.py
    in the binary.  Frozen modules are always (???) found before
    identical modules on PYTHONPATH/Registry.  There is no
    chicken-and-egg problem because site.py has already been
    hacked to turn on imputil.py and to provide for custom
    imports.

    This works now.  But there can be only one frozen module array,
    thus my suggestion to allow multiple arrays.  OTOH, maybe this
    is OK since imports are now customized.  But we are taking away
    the freeze feature including frozen "__main__".

2)  Declare that imputil.py is part of the distribution, and hack
    Py_Initialize() to turn it on before anything is imported.
    I am not sure how to solve the chicken-and-egg problem.
    Perhaps sys.executable could be used as the initial search
    path in Py_Initialize(), and the regular path be used iff
    sys.executable fails.  At least on Windows 95 and later,
    sys.executable is the highly reliable path to the Python
    interpreter binary.  I am not sure how reliable it is on Unix.
    The idea is that site.py (etc.) is always in the same directory
    as the binary.

    For this, we need to guarantee that *.py in the directory of
    sys.executable is always found first.  This is not currently
    the case.

3)  Somehow have the user create special built-in modules which
    actually contain the *.pyc code, and hack Py_Initialize() to
    load and initialize them first.

4)  Use pickle to create the magic file "python.py0" which contains
    site.py etc. in a standard format, I guess a dictionary.
    Hack Py_Initialize() to load its modules if it is located in
    the directory of sys.executable.  So the start-up Python code
    lives in "python.py0" with its binary.  Maybe we could
    automagically load python.py1, 2, ... too thus providing
    a hook for the user to add frozen code without a compiler.

    sys.executable must be bullet-proof.  Does the executable
    mean python.exe or python15.dll??  Both??

Another problem is that tools/Freeze/*.py seems to require one of
the frozen modules to be named "__main__" and thus be executed on
start up.  This means we can't put all this into python15.dll.  It
seems that (on Windows) exceptions.py and imputils.py should go into
python15.dll, and not into python.exe.  Site.py might go into either
python15.dll or python.exe.

FWIW, I am currently using three frozen module arrays.  Site.py,
exceptions.py and the Python library goes in python15.dll.  My
WPY modules go in python.exe, and the application main modules
go in another DLL.  This enables replacing python15.dll to update
Python.  But I am not that happy with this scheme.

Jim Ahlstrom