[Python-Dev] New relative import issue

Phillip J. Eby pje at telecommunity.com
Fri Sep 22 22:25:49 CEST 2006

At 12:42 PM 9/22/2006 -0700, Josiah Carlson wrote:
> > You might as well suggest that each environment
> > consist of a single large zipfile containing the packages in question: 
> this
> > would actually be *more* practical (and fast!) in terms of Python startup,
> > and is no different from having a database with respect to the need for
> > installation and uninstallation to modify a central file!
>We should remember that the sizes of databases that (I expect) will be
>common, we are talking about maybe 30k if a user has installed every
>package in pypi.  And after the initial query, everything will be stored
>in a dictionary or dictionary-like object, offering faster query times
>than even a zip file

Measure it.  Be sure to include the time to import SQLite vs. the time to 
import the zipimport module.

>SQLite is pretty fast.  And for startup, we are really only performing a
>single query per database "SELECT * FROM package_registry".  It will end
>up reading the entire database, but these databases will be generally
>small, perhaps a few dozen rows, maybe a few thousand if we have set up
>a bunch of installation-time application environments.

Again, seriously, compare this against a zipfile.  You'll find that there's 
absolutely no comparison between reading this and reading a zipfile central 
directory -- which also results in an in-memory cache that can then be used 
to seek() directly to the module.

>Actually, I'm offering a way of *registering* a package with the
>repository from the command line.  I'm of the opinion that setting the
>environment via command line for the subsequent Python runs is a bad
>idea, but then again, I have been using wxPython's wxversion method for
>a while to select which wxPython installation I want to use, and find
>things like:
>     import wxversion
>     wxversion.ensureMinimal('2.6-unicode', optionsRequired=True)
>To be exactly the amount of control I want, where I want it.

Well, that's already easy to do for arbitrary packages and arbitrary 
versions with setuptools.  Eggs installed in "multi-version" mode are added 
to sys.path at runtime if/when they are requested.

>With a package registry (perhaps as I have been describing, perhaps
>something different), all of the disparate ways of choosing a version of
>a library during import can be removed in favor of a single mechanism.
>This single mechanism could handle things like the wxPython
>'ensureMinimal', perhaps even 'ensure exact' or 'use latest'.

This discussion is mostly making me realize that sys.path is exactly the 
right thing to have, and that the only thing that actually need fixing is 
universal .pth support, and maybe some utility functions for better 
sys.path manipulation within .pth files.  I suggest that there is no way an 
arbitrary "registry" implementation is going to be faster than reading 
lines from a text file.

> > Setuptools works around this by installing an enhancement for the 'site'
> > module that extends .pth support to include all PYTHONPATH
> > directories.  The enhancement delegates to the original site module after
> > recording data about sys.path that the site module destroys at startup.
>But wasn't there a recent discussion describing how keeping persistant
>environment variables is a PITA both during install and runtime?

Yes, exactly.

>Extending .pth files to PYTHONPATH seems to me like a hack meant to work
>around the fact that Python doesn't have a package registry.  And really,
>all of the current sys.path + .pth + PYTHONPATH stuff could be subsumed
>into a *single* mechanism.

Sure -- I suggest that the single mechanism is none other than 
*sys.path*.  The .pth files, PYTHONPATH, and a new command-line option 
merely being ways to set it.

All of the discussion that's taken place here has sufficed at this point to 
convince me that sys.path isn't broken at all, and doesn't need 
fixing.  Some tweaks to 'site' and maybe a new command-line option will 
suffice to clean everything up quite nicely.

I say this because all of the version and dependency management things that 
people are asking about can already be achieved by setuptools, so clearly 
the underlying machinery is fine.  It wasn't until this message of yours 
that I realized that you are trying to solve a bunch of problems that are 
quite solvable within the existing machinery.  I was mainly interested in 
cleaning up the final awkwardness that's effectively caused by lack of .pth 
support for the startup script directory.

> > I'm not sure of that, since I don't yet know how your approach would deal
> > with namespace packages, which are distributed in pieces and assembled
> > later.  For example, many PEAK and Zope distributions live in the peak.*
> > and zope.* package namespaces, but are installed separately, and glued
> > together via __path__ changes (see the pkgutil docs).
>     packages.register('zope', '/path/to/zope')
>And if the installation path is different:
>     packages.register('zope.subpackage', '/different/path/to/subpackage/')
>Otherwise the importer will know where the zope (or peak) package exists
>in the filesystem (or otherwise), and search it whenever 'from zope
>import ...' is performed.

If you're talking about replacing the current import machinery, you would 
have to leave this to Py3K, otherwise all you've done is add a *new* import 
hook, i.e. a "sys.package_loaders" dictionary or some such.

If you wanted something like that now, of course, you could slap an 
importer into sys.meta_path that then did a lookup in 
sys.package_loaders.  Getting this mechanism bootstrapped, however, is left 
as an exercise for the reader.  ;)

Note, by the way, that it might be quite possible to do away with 
everything but sys.meta_path in Py3K, prepopulated with such an importer 
(along with ones to support builtin and frozen modules).  You could then 
import a backward-compatibility module that would add support for sys.path 
and for package __path__ attributes, by adding a new entry to 
sys.meta_path.  But this is strictly a pipe dream where Python 2.x is 

More information about the Python-Dev mailing list