[Python-Dev] Towards a Python based import scheme

[Greg Stein]

James C. Ahlstrom wrote:
>...
> As a result of all this import discussion I am a bit worried that
> the python library *.pyl file format may not be powerful enough.

Background for the readers:

.pyl is an extension that I used in my "small" distribution. I think
Gordon uses it, too. In any case, it is effectively a concatenation of
.pyc files along with a TOC mapping fully-qualified dotted module names
to seek-positions within the file.

[speaking of stat() calls: using a .pyl eliminates them quite nicely --
this may be part of Gordon's observed speed increase when using an
archive]

The .pyl format was discussed a bit on the distutils-sig list and "sort
of" accepted as an okay format for jamming a bunch of modules into a
single file.
[by "sort of", I mean that the small group who participated in the
discussion were okay with it :-); it is a great, minimalist format, so
it probably won't please people who like a ton of features in a file
format :-) ]

>...
> How about if the *.pyl file format is exactly a directory structure?
> I mean that the table of contents is limited to paths starting with
> a directory name only, and that the seperator is '/' instead of '.'.
> So a listing would be identical to the output of 'ls -R'.  So:
>   Lib/string.pyc
>   Lib/exceptions.pyc
>   Lib/plat-sunos4/...
>   mx/__init__.pyc
>   mx/...
>   package2/...
>   dir3/...
>   ...
> 
> The implied PYTHONPATH for this file is ["Lib", "."].  Since the
> format is exactly a directory tree, it is guaranteed that whatever
> PYTHONPATH or imports can do now or in the future with a directory
> tree, it can still do it with a *.pyl file.

People import things using a dotted name. Therefore, I think it makes
the most sense to map that straight to the resulting .pyc file. No
reason to put directories into the file... they make no sense to the end
user. During construction of the .pyl, you would walk the tree finding
all the available modules (and their corresponding dotted name) and
insert them.

Note that you can distribute multiple .pyl files. There could be the
Python standard lib in one file, the mx package in another, etc. As a
module is searched for, the system just peeks into each .pyl in turn,
looking for the module.

Search order is currently defined by order of install() on the Importer
instances. I believe the Right Way to do things is to create
sys.importers (as a list of Importers) and deprecate the sys.path
variable. Python could start up with an Importer than simply scanned
sys.path as a backwards compat measure; it could also leave sys.path
empty and create DirectoryImporters for each path component (this could
cause problems, though, for some apps that believe sys.path shouldn't be
empty, or that use it for magic-munging). I've search the standard lib
in the past -- there are only a couple real uses of sys.path if I
remember rightly (test package and the traceback module).

Cheers,
-g

--
Greg Stein, http://www.lyra.org/