[Distutils] Disabling --single-version-externally-managed

[Toshio Kuratomi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>> Just to illustrate what I'm trying to achieve.  I've updated the Fedora
>> Packaging Guidelines[1]_ to allow two versions of a package to coexist.
>>  I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and
>> the output of the test-sql.sh script using this procedure.  The end
>> result is what we want but the build step to get there seem a tad
>> fragile and kludgey.  Since these are going to become guidelines for all
>> of our python packages, I'd like to know if either: 1) there's a better
>> way to do this or 2) the results I'm achieving are not expected and
>> could disappear with a random upgrade of setuptools.
>>
>> .. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs
> 
> Here's the thing: if you want multiple installed versions of a package,
> *and* you want there to be a default version, then you *have* to have
> easy_install (or zc.buildout, or some similar tool) generate the startup
> scripts for anything that wants to use a non-default version.
> 
You'll have to explain why that is because all my experiments show that
to be false.  Installing one package via easy_install -m and another via
- --single-version-externally-managed gets us 75% of the way to this
working.  Using easy_install -m for both and then copying/symlinking
gets us 100% of the way.

> This is true irrespective of the formats of the versions involved,
> whether they're zipfiles, directories, single-version, or whatever. 
> It's just the nature of the beast, due to the fact that there is a
> global 'working_set' that lists the projects that are currently on
> sys.path, and it is initialized when pkg_resources is imported.  (And
> currently, there is no way to *remove* a distribution from the working
> set.)
> 
Since the working_set doesn't explicitly list eggs unless they are
specified in a project's requires.txt it seems like
pkg_resources.require() can override a module which is the default
because it is installed in site-packages.  In fact, I understood this to
be a feature of setuptools: allowing someone to override the vendor
installed packages in site-packages with your own eggs.

> Thus, if you want multiple versions *and* want to be able to select a
> version after pkg_resources has been imported, you *cannot* have a
> default version.  In other words, the egg must not be on sys.path when
> pkg_resources is imported.  Then, pkg_resources can locate the desired
> version and add it.
>
Not quite.  In the single-version case, the egg is on sys.path because
the module directory is in site-packages.  Therefore pkg_resources make
a different version importable by placing a different egg's path before
site-packages in sys.path.

The goal is to have all of these things work:
1) import MODULE should import whatever the vendor decided should be the
default.
2) requires.txt in a project's egginfo should work to select a specific
version from the installed eggs.
3) In a simple script (without requires.txt), pkg_resources.requires()
should work to select a specific version from the installed eggs.

I think the basic way to enable this is:
For 1) Either install a module that is not an egg into site-packages or
install an egg as single-version-externally-managed into site-packages.
 When the user does import MODULE, python looks in site-packages, finds
the MODULE, and imports it without touching any setuptools code.

For 2) All installed modules that we want to be selectable must be eggs.
 This needs to work with both single-version and multiple-version eggs
when determining the best version match then, if necessary, modify
sys.path to place the best match in front of the other eggs.

For 3) No eggs for this module can be on sys.path before the
pkg_resources.require() call.  This does not count modules brought in
via site-packages as those are going to be overridden when we place egg
paths before site-packages.

This seems to mostly work right now with the procedure I outlined in my
previous message.  I have to fix the bug found in my other message to
see if I can get it to work all the time (and perhaps eliminate the
kludginess in my original procedures.)

You seem to think there's something wrong with this so there's obviously
something you're seeing that I don't.  Can you give an example of where
this will fail?

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2dUKX6yAic2E7kgRAiMnAJ40guBFKrh0V3Yl+kzJqRlZ6V5TJgCgrA/j
fnHPHB9Q6533xSAzELAV6Dc=
=swaR
-----END PGP SIGNATURE-----