[Distutils] Easy install, making "Traditional" PYTHONPATH-based Installation work better

[Jim Fulton]

Phillip J. Eby wrote:
> At 10:52 AM 2/5/2006 -0500, Jim Fulton wrote:
> 
>> I'be been learning about setuptools and eggs and am very pleased.
>> There is one area where I'd like to see a small tweak.
>>
>> In:
>>
>>
>> http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations 
>>
>>
>> four options are presented for installing outside of Python installation.
>>
>> Naturally, I prefer the the last option, which the documentation 
>> recommends
>> against. :)  Here's why. An important use case to me is the ability for
>> a developer to work in a development sandbox without affecting anything
>> outside the sandbox.  The solution needs to be cross platform.  The
>> "Administrator Installation" option doesn't work for me because it 
>> requires
>> modifying the Python setup.  Neither the "Mac OS X "User" Installation"
>> nor the 'Creating a "Virtual" Python' options work for me because they
>> are not cross platform.  I also find the "Virtual Python" approach a bit
>> heavy handed.
>>
>> In playing a bit, I find the "Traditional" PYTHONPATH-based Installation
>> works pretty well except for the problem of having to set the path 
>> manually
>> when invoking scripts.  This problem could be addressed by having
>> the generated scripts include the custom path settings.
> 
> 
> Unfortunately, that's probably harder than you think. 

I doubt it. :)

 > The reason I
> advise against the PYTHONPATH-based approach is that it uses a sneaky 
> hack to get site directory processing to work "properly" (i.e., as 
> desired).
> 
> When you use the PYTHONPATH-based approach, it only works if you put the 
> setuptools egg directly on the PYTHONPATH at *Python startup*, because 
> setuptools includes its own version of the 'site' module that bypasses 
> the standard Python 'site' module, so that it can scan .pth files that 
> are *already on* PYTHONPATH.

I realize this.

> This means that no Python code embedded in the scripts is going to 
> affect this. 

Unless it invokes the setuptools site,py code itself.  This is all
about getting the path hacked.

 > If you want to use PYTHONPATH, you have to set PYTHONPATH
> in the environment.  By the time the Python script itself is running, 
> sys.path has already been manipulated such that it's no longer possible 
> to process .pth files in a way that gives them precedence over 
> system-installed packages.

Overriding system-installed packages wasn't a primary goal, although
that would be nice.

> Now, if you don't care about the .pth files and just want to use 
> whatever the script's require() picks up, then it's not a problem.

Actually, this happens to be the case for me.  I plan to always use
-m with easy_install.


> However, since your post talked about .pth and site-dirs, I assumed that 
> means you want to process that stuff.

The only reason that *I* need to do so is to get at the setuptools
installation.  Scripts need to get at setuptools and I don't want the
scripts to hard code the name if the setuptools egg, which has
a version number in it.

Of course, if this was to be a general solution, then I think we'd
need to deal with the .pth files.

> 
>> For now, I can create a wrapper around easy_install that does
>> custom script generation, which could serve as a prototype for the
>> proposed change.
> 
> 
> Try it out, and see if you can get a privately installed package to 
> *override* a system-installed one, without setting PYTHONPATH in the 
> environment prior to Python startup.  I'll be surprised and intrigued if 
> you can find a way.  (*Adding* packages this way is of course easy; it's 
> overriding system-installed packages that's the problem, because you 
> have to get the .pth stuff in *front* of site-packages in sys.path, and 
> that normally doesn't happen.  It only happens with setuptools because 
> of its hacked 'site.py'.)

Well, I could have the generated script code call the hacked
site.py from setuptools, so it should work as long as the system packages
in question haven't been imported yet, which is good enough for me.

> If you don't care about .pth files, then you can simply install the 
> desired eggs or egg-links to the same directory as the scripts,

I don't want to do that, although I can live with it I suppose.

 > and
> PYTHONPATH only needs to include the setuptools egg.  It's either that, 
> or perhaps put some bootstrap code in every script to locate setuptools 
> if it's not already on sys.path.

Again, my plan was to put some bootstrap code in every script that
aranged to have the working library and the setuptools egg in the system
path.

I'll try it and let you know what comes out of it.

It's too bad setuptools isn't in the standard library. If it was, I'd
probably just stick all my eggs and scripts in the same place and be
done ith it. :)  It's still tempting.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org