[Distutils] wishlist for setuptools

Christopher Dunn cdunn2001 at gmail.com
Sun Aug 10 15:19:42 CEST 2008

In a nutshell, here is what I want to be able to do:

   1. Download several Python eggs into a directory, including multiple
   versions of some eggs.
   2. Add that directory to sys.path at runtime.
   3. Use pkg_resources.require() to select versions of the eggs.
   4. 'import Foo' for each package "Foo".

Python Eggs are great, but I have a major problem with setuptools: Where I
work, I am not allowed to alter PYTHONPATH or to install into site-packages.
 I can add paths to sys.path at runtime, but normally we load modules via
sys.meta_path.  We have a large database of modules by many different
developers, and we have per-project configuration files which select
versions of these modules to be accessible via sys.meta_path.  I'd like to
continue to select our own modules via the configuration files, but I'd like
to load Eggs from PYPI via the procedure above.

This may seem impossible, but I have a fairly simple solution: Put an Egg
importer on sys.meta_path.  I'll outline the solution:

First, realize that no site.py or .pth files will be processed in this usage

Second, when a version range is specified via pkg_resources.require(),
register that specification into a global Egg importer (attached to the
setuptools module).  Also, append that importer to the end of sys.meta_path,
if that has not already been done.  This Egg importer should follow the
protocol of PEP 302.

Third, when a module registered as an Egg is imported, Python will first
look for an importer on sys.meta_path.  If not intercepted by another
meta_path importer, eventually our new Egg importer will be tried.  The Egg
importer should search for the Egg by scanning sys.path, in order.  It
should look not only for Eggs, but also for directories which might contain
Eggs.  It should accept the first Egg discovered which matches the
registered criteria and load it. Otherwise, it should throw an ImportError

Some things to note about this approach:

   1. sys.path_hooks is ignored.  This is because the call to require()
   implied a change in import semantics.
   2. If an egg is not registered via 'require', then the Egg importer on
   sys.meta_path will not look for it, but the egg can still be loaded if it
   was 'easy_installed' and if the site.py and .pth files were properly read.
    In other words, we have complete backward-compatibility.
   3. If an appropriate egg cannot be found, the ImportError is raised at
   'import' time, not when 'require()' is called.  For this reason, we may need
   a different 'require' function, maybe 'eggs.require()'.  Or, sys.path could
   be scanned inside 'require()'.
   4. We have eliminated the O(n^2) look-up cost of a large number of eggs
   listed on sys.path directly.  In fact, we don't have to put any eggs on
   sys.path, but if we do, they will take precedence over later eggs or
   directories containing eggs, as a user would expect.

I hope that everyone takes this request seriously.  The current system is
almost completely broken for me.  The only way I can use eggs is to add each
one to sys.path explicitly at runtime.  That's a huge PITA and a big
maintenance problem.  I'm very disappointed by the whole PYPI system at the
moment.  Something that could be extremely easy is instead very difficult.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20080810/5e8b31e7/attachment.htm>

More information about the Distutils-SIG mailing list