[Python-Dev] New relative import issue

Josiah Carlson jcarlson at uci.edu
Fri Sep 22 09:08:03 CEST 2006

"Phillip J. Eby" <pje at telecommunity.com> wrote:
> At 08:44 PM 9/21/2006 -0700, Josiah Carlson wrote:
> >This can be implemented with a fairly simple package registry, contained
> >within a (small) SQLite database (which is conveniently shipped in
> >Python 2.5).  There can be a system-wide database that all users use as
> >a base, with a user-defined package registry (per user) where the
> >system-wide packages can be augmented.
> As far as I can tell, you're ignoring that per-user must *also* be 
> per-version, and per-application.  Each application or runtime environment 
> needs its own private set of information like this.

Having a different database per Python version is not significantly
different than having a different Python binary for each Python version. 
About the only (annoying) nit is that the systemwide database needs to
be easily accessable to the Python runtime, and is possibly volatile. 
Maybe a symlink in the same path as the actual Python binary on *nix,
and the file located next to the binary on Windows.

I didn't mention the following because I thought it would be superfluous,
but it seems that I should have stated it right out.  My thoughts were
that on startup, Python would first query the 'system' database, caching
its results in a dictionary, then query the user's listing, updating the
dictionary as necessary, then unload the databases.  On demand, when
code runs packages.register(), if both persist and systemwide are False,
it just updates the dictionary. If either are true, it opens up and
updates the relevant database.

With such a semantic, every time Python gets run, every instance gets
its own private set of paths, derived from the system database, user
database, and runtime-defined packages.

> Next, putting the installation data inside a database instead of 
> per-installation-unit files presents problems of its own.  While some 
> system packaging tools allow install/uninstall scripts to run, they are 
> often frowned upon, and can be unintentionally bypassed.

This is easily remedied with a proper 'packages' implementation:

    python -Mpackages name path

Note that Python could auto-insert standard library and site-packages
'packages' on startup (creating the initial dictionary, then the
systemwide, then the user, ...).

> These are just a few of the issues that come to mind.  Realistically 
> speaking, .pth files are currently the most effective mechanism we have, 
> and there actually isn't much that can be done to improve upon them.

Except that .pth files are only usable in certain (likely) system paths,
that the user may not have write access to.  There have previously been
proposals to add support for .pth files in the path of the run .py file,
but they don't seem to have gotten any support.

> What's more needed are better mechanisms for creating and managing Python 
> "environments" (to use a term coined by Ian Bicking and Jim Fulton over on 
> the distutils-sig), which are individual contexts in which Python 
> applications run.  Some current tools in development by Ian and Jim include:
> Anyway, system-wide and per-user environment information isn't nearly 
> sufficient to address the issues that people have when developing and 
> deploying multiple applications on a server, or even using multiple 
> applications on a client installation (e.g. somebody using both the 
> Enthought Python IDE and Chandler on the same machine).  These relatively 
> simple use cases rapidly demonstrate the inadequacy of system-wide or 
> per-user configuration of what packages are available.

It wouldn't be terribly difficult to add environment switching and
environment derivation (copying or linked, though copying would be

    packages.register(name, path, env=environment)

It also wouldn't be terribly difficult to set up environments that
required certain packages...

    packages.new_environment(environment, *required_packages, test=True)

To verify that the Python installation has the required packages, then

    packages.new_environment(environment, *required_packages, persist=True)

I believe that most of the concerns that you have brought up can be
addressed, and I think that it could be far nicer to deal with than the
current sys.path hackery. The system database location is a bit annoying,
but I lack the *nix experience to say where such a database could or
should be located.

 - Josiah

More information about the Python-Dev mailing list