[Distutils] patch: solving the two major things that people dislike about setuptools, part 1

[P.J. Eby]

At 03:00 PM 1/22/2009 -0500, Mark Sienkiewicz wrote:
>Phillip J. Eby wrote:
>>At 04:48 PM 1/20/2009 -0700, zooko wrote:
>>>So, this is just to let everyone know that any setuptools variant
>>>(including a future release of PJE's setuptools) that respects the
>>>PYTHONPATH will get at least two users!
>>
>>As I've said before, if somebody creates a patch that addresses the 
>>use cases I pointed out, I'd be happy to accept it.
>
>As I understand it, you require that a package installed by 
>setuptools will be discovered before a package installed by 
>distutils >in the same directory<.
>You do not require that a package installed by setuptools be 
>discovered before a package that is earlier in sys.path.
>
>In that case, the .egg file should go on sys.path immediately before 
>the directory where it is stored.  This ensures that a setuptools 
>.egg file will always be discovered before a distutils-installed 
>package/module in the same directory, but it will not override 
>packages/modules that occur earlier in sys.path.
>
>Do you have any other requirements?
>
>In 
>http://mail.python.org/pipermail/distutils-sig/2008-November/010534.html 
>, I described a change to zooko's patch that implements this behaviour.
>
>In 
>http://mail.python.org/pipermail/distutils-sig/2008-November/010540.html 
>, I address concerns about case-insensitive filesystems.
>So, taken together, does my suggestion meet your criteria for an 
>acceptable patch?
>
>If so,
>
>- I will assemble it into a patch.
>- I will test it on platforms that I have access to, in ways that I 
>know how to use setuptools.  (For the record, that is "python 
>setup.py install --home=/dir"; I never tried the automatic downloads.)
>
>If not,
>
>- what further requirements do you have?

In principle, this sounds pretty good.  The only piece left that I'm 
worried about is that there needs to be a fallback for the .index() 
call in the .pth file.  Otherwise, if for some reason there is *not* 
a match, there will be an error and .pth loading will fail.

On the other hand, maybe it's better for there to be an error in that 
case, I'm just a bit worried about the new and uncontrolled possible 
way for things to break, vs. the known and predictable way they break now.  ;-)

As for the two "big loops" in site-patch.py, the first one is there 
to load the existing site module, without altering sys.path in any 
way.  And it has to use both the path importer cache and 'imp', in 
order to be compatible w/e.g. systems where the stdlib is a zipfile.

The second "big loop" is responsible for moving any paths added by 
.pth files on PYTHONPATH so that they are ahead of the stdlib and 
site-packages.  (Otherwise, if you install something like PIL to 
PYTHONPATH, it can't override a version installed in site-packages.)

It's not clear to me at this moment whether this loop would need to 
change with the patch.  I don't think it will, though.  The 
site-packages eggs would get inserted before site-packages, and would 
be considered "known paths", and thus have their position left 
unchanged.  Whereas PYTHONPATH eggs would be before the stdlib, and 
also left alone.

The last concern I have is whether you can partially upgrade an 
installation, i.e., what happens if you have two PYTHONPATH 
directories, one where this patch has been used, and the other where 
it has not.  The one where it has been used will put its eggs in the 
"right" place, while the other will blindly insert to the 
beginning.  Hm.  Yes, it seems like that would be okay, because the 
"right" place would never be before sys.__egginsert.  Thus, the old 
code could never stick an egg in a place that would mess something 
up.  And both approaches result in PYTHONPATH eggs being before the 
stdlib, and thus left alone by the second loop.

Okay, yes, I think this approach can work... *except* for perhaps the 
case where an old site-patch is in effect, due to being earlier on 
PYTHONPATH, with newer .pth files in a second PYTHONPATH directory.

In that case, the addsitedir() calls in site-patch.py will *not* be normalized.

So, for this to work, site-patch.py should *not* be 
changed.  Instead, the .pth-embedded code needs to do the makepath() 
call before the .index() lookup.  Otherwise, people with multiple 
PYTHONPATH directories would need to specifically update the site.py 
in them, to not have inexplicable breakages.

I'll review an updated patch along these lines.