I got a little time to think about this over the weekend and propose this design. It is a way to package *.pyc files into a single-file portable archive for distribution with commercial, CGI (like Marc is doing) or otherwise simplified distributions. It is dumb-stupid-simple, my personal favorite and a requirement for commercial software. This is not concerned with an "installer", which is a separate problem. Few of these ideas are mine.
1. There is a Python library file format (PYL) which can hold *.pyc files. The format is: a.pyc, b.pyc, marshalled dict, offset of dict, magic number.
The a.pyc, b.pyc,... are the bytes of the *.pyc files including the eight byte header. The dict has keys "name", values are the seek offset of the start of the *.pyc files. The "offset of dict" is the seek offset for finding the dict. The magic number identifies the file as a PYL file.
The "names" can be normal names such as "os", or dotted names such as "foo.bar". Although initially devoted to *.pyc files, we note that it is possible to include other resources with non-module names such as "more*data".
A PYL file is portable to any computer, Unix, Windows, etc. Compression is not included, and should be done at the "installer" level if desired.
2. The PYL file always has the same directory and file name as the binary module containing import.c. but with a .pyl ending:
Python Binary PYL File Name /usr/local/bin/python /usr/local/bin/python.pyl /my/so/dir/python.so /my/so/dir/python.pyl C:/python/python15.dll C:/python/python15.pyl
These Python binary names are available as sys.executable (the main) and sys.dllfullpath (the shared library or DLL).
3) Since the PYL file can be efficiently read backwards, it can, if desired, be appended the the Python binary itself: cat python15.pyl >> python15.dll
4) The PYL file is created with the Python program makepyl.py and no C compiler is necessary.
5) There is a new optional built-in module "importer" which may be included by editing Setup. It is imported in Py_Initialize() after "sys" is set up, but before any other imports such as "exceptions". It is not an error if it is absent. If present, it replaces __import__ just like imputils.py does. The replacement importer searches for PYL files as a .pyl file, in the current sys.executable, and in the current sys.dllfullpath (name of DLL). Note that importer can use multiple PYL files. Importer is able to import the modules exceptions, site and sitecustomize, and probably most other modules. Importer has methods to print out the names of PYL modules available. You could still override importer using sitecustomize and imputils if desired, in which case it may be convenient to use importer's methods.
6) Alternative to (5): Modules exceptions, site, sitecustomize and imputils are frozen (using frozen modules) into the interpreter binary, and sitecustomize boots imputils. Thereafter the Python logic in sitecustomize and imputils implements the logic of (5). Sitecustomize has methods to print out the names of PYL modules available.
7) The Python main program is enhanced to start "__main__" as the main program if it can be imported, in which case all command line arguments are sent to __main__. This enables you to create a Python program "myprog", and start it with the command "myprog -a arg1 ..." just like a normal program.
8) The Python main can start any module "foo" (which may be in a PYL file) as the main program in response to a new command line argument. This enables you to ship multiple main programs in the PYL file.
9) The current frozenmain.c is eliminated and the enhanced main is used instead. This (I hope) results in a net decrease in code.
Where this is going:
It enables simplified distribution of Python programs.
Greg has demonstrated that import.c can be replaced with a Python module, and that in general at least some of Python should be written in itself. This heads that way.
Please send in your "hell no" comments now.