[Distutils] Smart conflict management/resolution for setuptools

[Phillip J. Eby]

Okay, so I've been looking over the various remaining issues that people 
have encountered with setuptools installation, and there's a kind of theme 
emerging: conflict management and resolution.

First, let me give some of the problems.  First off, the conflict detection 
system at installation time isn't very bright.  It'll happily report 
conflicts with system-installed packages, even ones that are installed 
using --single-version-externally-managed.  It doesn't pay attention to 
what order stuff is installed in on sys.path, either; it just says, "hey, 
this *might* conflict" and kicks up a fuss.  The "develop" command 
currently doesn't check for conflicts at all.  And finally, the 
--delete-conflicting flag will happily attempt to delete system-installed 
packages, even if they're .egg-info ones.  Ouch.

And those are just the problems with install-time conflict 
detection.  Runtime conflict resolution's not that great, either.  Once an 
egg gets in the working set (e.g., due to being the default or 
system-installed version of something), it never comes out, meaning that 
you can't really override the default version.  There's a special trick 
that's used to bypass this issue for scripts, that for all practical 
purposes just throws out sys.path and starts over if there's a conflict, 
but it's a little drastic and in any case can't be generalized.

Finally, Guido has proposed allowing setuptools to get into the stdlib for 
Python 2.5, but because it's a moving target, we need a way (ala PyXML) to 
allow users to safely install newer versions of setuptools in such a way as 
to replace the stdlib-supplied version.

After a fair amount of head-scratching, I think I have some ideas that can 
fix all of these problems, but I want to air them out here before starting 
implementation, to see if anybody can find any holes in the plan or 
improvements to it.  The runtime part is pretty easy and less likely to 
have any controversial impacts, so I'll save that part for last, and deal 
with the installation conflict issues first.


Installation Conflicts
======================

The original reason for having installation conflict detection was that 
eggs added using .pth files normally ended up at the tail end of sys.path, 
after any site-packages or other locally-installed packages.  In effect, 
eggs were always the lowest man on the totem pole, so you *had* to get rid 
of any other installed version in order for it to work.

But in setuptools 0.6a6, I added logic to pkg_resources that automatically 
placed activated eggs *before* their containing directory in 
sys.path.  Thus, an egg in site-packages would be placed (at runtime) 
*before* the site-packages directory in sys.path, thereby overriding any 
"legacy" or "system-installed" packages there.  This is also true of any 
other directory on sys.path: if it contains an egg, and you activate the 
egg, the egg will be placed just before that directory in sys.path.

So in theory, conflict detection shouldn't have been needed since 
0.6a6.  In practice, however, people need to be able to use eggs *without* 
importing pkg_resources and munging sys.path, so this is only a partial fix.

However, it does point the way to a full fix.  If we could get .pth files 
to do essentially the same trick, we'd be all set.  Better still, if we 
could do it in such a way that the added entries were before the stdlib on 
sys.path, then we would at the same time accomplish the other need: 
upgrading a stdlib-provided version of setuptools!

At PyCon, I had a few hallway conversations that touched on this issue, and 
as I mentioned then, the simplest possible hack is something like adding 
this line to a .pth file (e.g. setuptools.pth):

     import sys; sys.path.insert(0,"/path/to/setuptools.egg")

But this isn't very helpful in the case where you might have more than one 
sys.path directory with an easy-install.pth file in it.  Why?  Because the 
later the directory is on sys.path, the *earlier* on sys.path the egg will 
be inserted.  Thus, the eggs end up in reverse order, with eggs installed 
in site-packages would take precedence over even those on PYTHONPATH!


A Half-baked Solution
---------------------

So, I've been racking my brain for a solution to this.  Mostly, I was 
thinking in terms of tracking what egg version is selected, but today I 
realized there is a simpler solution.  We just need to ensure that the 
insertion point *moves forward* each time we insert an egg, e.g.:

     import sys; sys.__egginsert=p=getattr(sys,'__egginsert',-1)+1; \
         sys.path.insert(p,"/path/to/the.egg")

I've used a \ continuation for clarity here, but of course the real .pth 
file couldn't do this; the whole thing has to be on one line.

I've considered a few other ways to do this, like inserting a marker value 
directly into sys.path:

     import sys; '@@@' not in sys.path and sys.path.insert(0,'@@@'); \
         sys.path.insert(sys.path.index('@@@'),"/path/to/the.egg")

But this has the problem of needing to choose a suitable marker value that 
won't be a valid import string, and doesn't introduce security problems, 
etc.  So I came up with the __egginsert approach as a way to fix that.

The main (potential) problems I see remaining with __egginsert are:

1. Change in semantics of current install (can't be helped, but needs 
investigation)

2. If somebody else is crazy enough to do something similar, hilarity may 
ensue as hacks in different .pth files use insert points that don't go 
where they think.  (Not that likely; few people have even heard of .pth 
files, and in any case the insertions will retain a consistent order 
*within* each set, putting an upper limit on the possible chaos.)

3. EasyInstall will have to be able to *read* these lines as well as write 
them, and parsing one correctly looks a bit daunting to do.

4. The 'site' module has to 'exec' lines beginning with 'import', so 
there's a compile-and-exec delay imposed by each line, thus increasing the 
incremental startup cost for each installed egg.


A More Complete Solution
------------------------

So here's a new idea, made up fresh while writing this:

         import sys; sys.__oldpath=sys.path[:]; sys.path[:]=[]

         ... normal .pth lines go here ...

         import sys; new=sys.path[:]; sys.path[:]=sys.__oldpath; \
            p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \
            sys.__egginsert = p+len(new)

(Again, the \ continuations won't be in the real file.)

Anyway, what this does is temporarily nuke sys.path, then let the normal 
.pth processing logic run until we get to the end of the file, where we 
restore it and slap in the list of directories that were added by the 
'site' module.

Or, perhaps more simply:

         import sys; sys.__plen = len(sys.path)

         ... normal .pth lines go here ...

         import sys; new=sys.path[sys.__plen:]; \
            del sys.path[sys.__plen:]; \
            p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \
            sys.__egginsert = p+len(new)

This doesn't wipe out sys.path during the interim processing, although I 
suppose it is really just as complex as the previous version.  Somehow, it 
just feels "safer" not to ditch sys.path during the processing.

Anyway, either of these will keep the number of 'exec's capped at a maximum 
of 2 lines, *and* they'll make easy_install's parsing job easy, because it 
can just ignore the "import" lines when reading, and just regenerate them 
when writing.

This change would also thus affect only easy_install'ed eggs; it would have 
no effect on normal .pth processing.  But I would have to slightly alter 
the existing 'site.py' hack that setuptools installs in PYTHONPATH 
directories, so that it resets the sys.__egginsert position to zero before 
processing the .pth files on PYTHONPATH, and then restores its original 
value afterward.  (That way, eggs installed on PYTHONPATH will come before 
those installed in site-packages, but any subsequent addsitedir() 
operations will add to the right spot, more or less.)

That does suggest that there's some fragility involved in this process, if 
you use addsitedir() at runtime.  This is probably the biggest issue with 
this approach.

I did a bit of googling, however, and was only able to find 3 or 4 projects 
that do any kind of addsitedir stuff, including the usual suspects such as 
Twisted and Zope.  :)  I didn't find anything that looked like it would be 
negatively affected, although I admit I didn't study the surrounding code 
too deeply.  Mostly, they were doing things that were basically the same as 
what we're trying to do here: i.e., support .pth files in PYTHONPATH and in 
some cases, put them first on sys.path.

The only place where confusing results are likely, is the case where 
somebody calls addsitedir() *after* the pkg_resources module is imported, 
thus causing the working_set to be out of sync with sys.path.  Of course, 
this is *already* a possible problem if you munge sys.path after importing 
pkg_resources, which suggests that some additional conflict detection there 
would also be helpful.

(Side rant: this is one of those times where the singleton nature of 
certain Python features is really really annoying.  If only one could 
instantiate interpreters in standard Python!)

Anyway, sys.path manipulation is already fraught with numerous dangers, so 
I won't dwell on that issue for now.  Perhaps in 0.7 or 0.8 I'll revamp the 
WorkingSet class so it can deal with dynamic manipulation of an 
externally-supplied list.


Proposed Installation Strategy
------------------------------

To summarize my proposal for handling installation "conflicts":

* EasyInstall will write special .pth files with a header and trailer 
"import" hack.  It will ignore "import" lines when reading them.

* I'll change the existing site.py hack to accomodate the out-of-order 
insertion, and make EasyInstall update any existing hacked site.py files.

* EasyInstall will stop checking for (and deleting) installation conflicts, 
because there will *no longer be such a thing*.  The options that control 
conflict detection will remain, but will have no function except to issue 
deprecation warnings.

And to summarize the effects of these changes:

* Eggs installed by EasyInstall will have sys.path precedence over 
everything else, including the standard library, script directory, 
etc.  They will, however, have a precedence order amongst themselves that 
reflects the sequence in which the .pth files were loaded.

* Invoking site.addsitedir() after pkg_resources is imported may produce 
slightly weirder results than it already does.  :)

* EasyInstall will be able to upgrade stdlib packages, not just via 
PYTHONPATH installs, but also via site-packages installs.

And the issues that remain open are:

* Should --single-version-externally-managed installations check for 
conflicts?  Of course, users of the option are theoretically more 
experienced users/packagers, and are also accustomed to the distutils' 
default (and thoroughly unsafe) behavior, so perhaps they should be left to 
their own devices.

* Should pkg_resources resort to more extreme measures to track the runtime 
value of sys.path?  We'll perhaps look more at this a bit more closely 
below, in the section on runtime conflict management.

Does anybody see any issues I haven't covered?  Any ideas for improvements?


Runtime Conflicts
=================

The solution to runtime conflicts was a little easier to figure out, 
although now that I've figured out the installation conflicts side, I think 
the runtime side is actually going to be a lot tougher to implement 
correctly, as there are a lot more moving parts.

But the idea is relatively simple to explain: when trying to resolve 
dependencies, handle VersionConflict errors by:

1. Removing the conflicting package from the working set, if and only if 
it's safe to do so.

2. Retry the operation, and repeat.

If a package isn't safe to remove, reraise the conflict error after 
restoring any previously-removed packages, so that the working set is 
rolled back to its original state.

At this level, it's all pretty simple.  The tricky parts are in the details:

1. This isn't really compatible with the existing API, for anything but the 
require() operation.  There's no sane way to change the resolve() API to 
work with this, because resolve() isn't supposed to have side effects, and 
it only returns a list of distributions to *add*, not ones to *remove*.

2. Defining "safe to remove" is complex.  As a first approximation, it's 
safe to remove a distribution from the working set if:

a) it's the only distribution with that file/directory name (Think of 
.egg-info dirs all sitting in site-packages; you can't remove one of those 
babies without removing *all* system-installed eggs in site-packages, since 
they all share a single sys.path entry!)

b) it hasn't had anything imported from it yet

c) no namespace packages it participates in have been imported

    Those latter two conditions are interesting, because they basically 
mean you have to go through every sys.modules entry looking for names 
starting with anything in the egg's top_level list, and then check those 
modules' __file__ and __path__ values to see if anything starts with the 
egg's location.  This is likely to be slow, but that's not really a big 
deal; this is a fallback for resolving a runtime version conflict, after 
all.  But it's somewhat tricky to test well, due to the singletons (e.g. 
sys.modules) that are involved.

3. Working sets don't actually allow you remove anything currently, and I'm 
not sure how to actually implement that feature!

So, after reviewing this, I think I'm going to avoid touching this during 
the remainder of the 0.6 development cycle, because it seems too likely to 
introduce instability.  Also, requirement 2c above isn't really practical 
before 0.7, because of the current eager loading of namespace packages.  It 
would basically rule out conflict resolution for eggs that currently 
participate in namespace packages.

Last, but not least, if I put it off till 0.7, I can go whole hog on 
refactoring the WorkingSet class, to also support better sys.path tracking 
at runtime.  And if some corners of the API have to get tweaked a bit to 
make that work in 0.7, I can probably live with it.

Any other thoughts or suggestions?