[Distutils] Re: Add multiple frozen modules

[Gordon McMillan]

Jim Ahlstrom wrote:

> I got a little time to think about this over the weekend
> and propose this design.  It is a way to package *.pyc files
> into a single-file portable archive for distribution with
> commercial, CGI (like Marc is doing) or otherwise simplified
> distributions.  It is dumb-stupid-simple, my personal favorite and a
> requirement for commercial software.  This is not concerned with an
> "installer", which is a separate problem.  Few of these ideas are
> mine.

Sigh. Most of this is in my installer.

> 1.  There is a Python library file format (PYL) which can
> hold *.pyc files.  The format is:
>   a.pyc, b.pyc, marshalled dict, offset of dict, magic number.

Right (except I put magic before offset). I've got 2. One is just like
this, except everything is zlib-ed. There's another where you've got
aribtrary chunks of bytes
 chunk1, chunk2, ... table-of-contents, magic, offset

where table-of-contents is fairly easily read and written in either
Python or C, and contains additional information (like whether the
chunk needs compressing / decompressing, and a filename...). 

> The a.pyc, b.pyc,... are the bytes of the *.pyc files including the
> eight byte header.  The dict has keys "name", values are the seek
> offset of the start of the *.pyc files.  The "offset of dict" is the
> seek offset for finding the dict.  The magic number identifies the
> file as a PYL file.
> 
> The "names" can be normal names such as "os", or dotted names
> such as "foo.bar".  Although initially devoted to *.pyc files,
> we note that it is possible to include other resources with
> non-module names such as "more*data".

My first kind is confined to compiled python. The 2nd can have 
anything.

> A PYL file is portable to any computer, Unix, Windows, etc.

Yup. Both kinds.

> Compression is not included, and should be done at the
> "installer" level if desired.

Actually, because the .pyz file is always open and you don't have to
stat anything, I find that it's faster even with decompression than
the normal import stuff.

> 2.  The PYL file always has the same directory and file name
> as the binary module containing import.c. but with a .pyl ending:
> 
>         Python Binary                PYL File Name
>     /usr/local/bin/python      /usr/local/bin/python.pyl
>     /my/so/dir/python.so       /my/so/dir/python.pyl
>     C:/python/python15.dll     C:/python/python15.pyl

You put anything in one of my .pyz's, (packages or modules) and they
just look like they're on your pythonpath.

> These Python binary names are available as sys.executable (the main)
> and sys.dllfullpath (the shared library or DLL).

Not sure what you're getting at. You can find the name of the .pyz by
asking the importer (an attribute stuck onto the imported module).

> 3)  Since the PYL file can be efficiently read backwards, it
> can, if desired, be appended the the Python binary itself:
>     cat python15.pyl >> python15.dll

Never tried it on the dll.  I do it with the .exe.

> 4)  The PYL file is created with the Python program makepyl.py
> and no C compiler is necessary.

Right. It's called archivebuilder. Pass it module names, package
names, directory names...

> 5)  There is a new optional built-in module "importer" which may be
> included by editing Setup.  It is imported in Py_Initialize() after
> "sys" is set up, but before any other imports such as "exceptions". 
> It is not an error if it is absent.  If present, it replaces
> __import__ just like imputils.py does.  The replacement importer
> searches for PYL files as a .pyl file, in the current
> sys.executable, and in the current sys.dllfullpath (name of DLL).
> Note that importer can use multiple PYL files.  Importer is able to
> import the modules exceptions, site and sitecustomize, and probably
> most other modules.  Importer has methods to print out the names of
> PYL modules available.  You could still override importer using
> sitecustomize and imputils if desired, in which case it may be
> convenient to use importer's methods.
> 
> 6)  Alternative to (5):  Modules exceptions, site, sitecustomize and
> imputils are frozen (using frozen modules) into the interpreter
> binary, and sitecustomize boots imputils.  Thereafter the Python
> logic in sitecustomize and imputils implements the logic of (5).
> Sitecustomize has methods to print out the names of PYL modules
> available.

I don't muck with core python at all. That means you need exceptions
and site, with site using imputil to load the .pyzs. I'm hoping
imputil attains sanctified status, so this happens in only one step.

> 7)  The Python main program is enhanced to start "__main__" as
> the main program if it can be imported, in which case all command
> line arguments are sent to __main__.  This enables you to create a
> Python program "myprog", and start it with the command "myprog -a
> arg1 ..." just like a normal program.

Yup. For Windows, I include a bunch of different .exes (all from one
source) which are just python.exe / pythonw.exe with some added smarts
about archives (the 2nd kind). Also not linked against python's import
lib, so they don't have to have python.dll in place before they start.

> 8)  The Python main can start any module "foo" (which may be in a
> PYL file) as the main program in response to a new command line
> argument. This enables you to ship multiple main programs in the PYL
> file.

Not really necessary to build that in. Your __main__ script can do it.

> 9)  The current frozenmain.c is eliminated and the enhanced main is
> used instead.  This (I hope) results in a net decrease in code.

A full python/Lib .pyz occupies less than 500K. Incidentally, I 
packed up your demo05.py (from wpy). For some reason, on my NT 
system, I end up using the Tk version of wpy. At any rate, with all
the Tcl/Tk, it still comes out to about 1.1Meg. Runs on a system
without any Python / Tcl / Tk, but not perfectly - you seem to do some
funny things in wpy.


- Gordon