[Distutils] Re: Add multiple frozen modules

[James C. Ahlstrom]

I got a little time to think about this over the weekend
and propose this design.  It is a way to package *.pyc files
into a single-file portable archive for distribution with
commercial, CGI (like Marc is doing) or otherwise simplified
distributions.  It is dumb-stupid-simple, my personal favorite and
a requirement for commercial software.  This is not concerned
with an "installer", which is a separate problem.  Few of these
ideas are mine.

1.  There is a Python library file format (PYL) which can
hold *.pyc files.  The format is:
  a.pyc, b.pyc, marshalled dict, offset of dict, magic number.

The a.pyc, b.pyc,... are the bytes of the *.pyc files including
the eight byte header.  The dict has keys "name", values are the
seek offset of the start of the *.pyc files.  The "offset
of dict" is the seek offset for finding the dict.  The magic
number identifies the file as a PYL file.

The "names" can be normal names such as "os", or dotted names
such as "foo.bar".  Although initially devoted to *.pyc files,
we note that it is possible to include other resources with
non-module names such as "more*data".

A PYL file is portable to any computer, Unix, Windows, etc.
Compression is not included, and should be done at the
"installer" level if desired.

2.  The PYL file always has the same directory and file name
as the binary module containing import.c. but with a .pyl ending:

        Python Binary                PYL File Name
    /usr/local/bin/python      /usr/local/bin/python.pyl
    /my/so/dir/python.so       /my/so/dir/python.pyl
    C:/python/python15.dll     C:/python/python15.pyl

These Python binary names are available as sys.executable (the main)
and sys.dllfullpath (the shared library or DLL).

3)  Since the PYL file can be efficiently read backwards, it
can, if desired, be appended the the Python binary itself:
    cat python15.pyl >> python15.dll

4)  The PYL file is created with the Python program makepyl.py
and no C compiler is necessary.

5)  There is a new optional built-in module "importer" which may
be included by editing Setup.  It is imported in Py_Initialize()
after "sys" is set up, but before any other imports such as
"exceptions".  It is not an error if it is absent.  If present, it
replaces __import__ just like imputils.py does.  The replacement
importer searches for PYL files as a .pyl file, in the current
sys.executable, and in the current sys.dllfullpath (name of DLL).
Note that importer can use multiple PYL files.  Importer is able
to import the modules exceptions, site and sitecustomize, and
probably most other modules.  Importer has methods to print out
the names of PYL modules available.  You could still override
importer using sitecustomize and imputils if desired, in which
case it may be convenient to use importer's methods.

6)  Alternative to (5):  Modules exceptions, site, sitecustomize
and imputils are frozen (using frozen modules) into the interpreter
binary, and sitecustomize boots imputils.  Thereafter the Python
logic in sitecustomize and imputils implements the logic of (5).
Sitecustomize has methods to print out the names of PYL modules
available.

7)  The Python main program is enhanced to start "__main__" as
the main program if it can be imported, in which case all command
line arguments are sent to __main__.  This enables you to create
a Python program "myprog", and start it with the command
"myprog -a arg1 ..." just like a normal program.

8)  The Python main can start any module "foo" (which may be in a PYL
file) as the main program in response to a new command line argument.
This enables you to ship multiple main programs in the PYL file.

9)  The current frozenmain.c is eliminated and the enhanced main
is used instead.  This (I hope) results in a net decrease in code.

Where this is going:

It enables simplified distribution of Python programs.

Greg has demonstrated that import.c can be replaced with a
Python module, and that in general at least some of Python
should be written in itself.  This heads that way.

Please send in your "hell no" comments now.

Jim Ahlstrom