In a nutshell, here is what I want to be able to do:

1. Download several Python eggs into a directory, including multiple versions of some eggs.
2. Add that directory to sys.path at runtime.
3. Use pkg_resources.require() to select versions of the eggs.
4. 'import Foo' for each package "Foo".

Python Eggs are great, but I have a major problem with setuptools: Where I work, I am not allowed to alter PYTHONPATH or to install into site-packages. I can add paths to sys.path at runtime, but normally we load modules via sys.meta_path. We have a large database of modules by many different developers, and we have per-project configuration files which select versions of these modules to be accessible via sys.meta_path. I'd like to continue to select our own modules via the configuration files, but I'd like to load Eggs from PYPI via the procedure above.

This may seem impossible, but I have a fairly simple solution: Put an Egg importer on sys.meta_path. I'll outline the solution:

First, realize that no site.py or .pth files will be processed in this usage model.

Second, when a version range is specified via pkg_resources.require(), register that specification into a global Egg importer (attached to the setuptools module). Also, append that importer to the end of sys.meta_path, if that has not already been done. This Egg importer should follow the protocol of PEP 302.

Third, when a module registered as an Egg is imported, Python will first look for an importer on sys.meta_path. If not intercepted by another meta_path importer, eventually our new Egg importer will be tried. The Egg importer should search for the Egg by scanning sys.path, in order. It should look not only for Eggs, but also for directories which might contain Eggs. It should accept the first Egg discovered which matches the registered criteria and load it. Otherwise, it should throw an ImportError exception.

Some things to note about this approach:

1. sys.path_hooks is ignored. This is because the call to require() implied a change in import semantics.
2. If an egg is not registered via 'require', then the Egg importer on sys.meta_path will not look for it, but the egg can still be loaded if it was 'easy_installed' and if the site.py and .pth files were properly read. In other words, we have complete backward-compatibility.
3. If an appropriate egg cannot be found, the ImportError is raised at 'import' time, not when 'require()' is called. For this reason, we may need a different 'require' function, maybe 'eggs.require()'. Or, sys.path could be scanned inside 'require()'.
4. We have eliminated the O(n^2) look-up cost of a large number of eggs listed on sys.path directly. In fact, we don't have to put any eggs on sys.path, but if we do, they will take precedence over later eggs or directories containing eggs, as a user would expect.

I hope that everyone takes this request seriously. The current system is almost completely broken for me. The only way I can use eggs is to add each one to sys.path explicitly at runtime. That's a huge PITA and a big maintenance problem. I'm very disappointed by the whole PYPI system at the moment. Something that could be extremely easy is instead very difficult.