[Distutils] A prefix option?

[Jim Fulton]

Phillip J. Eby wrote:
> At 06:40 AM 2/10/2006 -0500, Jim Fulton wrote:
> 
>> Phillip J. Eby wrote:
>>
>>> At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote:
>>>
>>>> I really don't think the virtual python approach is viable, at a 
>>>> minimum
>>>> because it doesn't work on windows.  It is also unacceptably heavy IMO.
>>>
>>>
>>> What do you mean by "heavy"?
>>
>>
>> It essentially duplicates the python install via symbolic links.
> 
> 
> That's just a statement of what it does.  I was looking for an 
> explanation of what makes that "heavy".  :)  One assumes that on Unix, 
> directory entries and inodes are relatively cheap.

"heavy" is qualitative.  I was explaining what it did that made it seem
heavy to me.  Perhaps I should have said "heavy handed".  So, to be more
specific, in addition to the fatal flaw of not working on windows:

- It copies, not links, the python executable. On my system, that's
   a 3 meg file.  Is that heavy? Some would say yes.  With disks
   measured in hundreds of gigabytes these days, it doesn't get me
   very excited.  I do think it is pretty heavy handed.

- While it links individual files it copies the directory structure,
   including include and site-packages.  This means that new packages
   installed into the base python won't be reflected in the virtual python.

Aside from the lack of windows suport, I might be willing to live with this
approach if we couldn't do better, but I think we can find better solutions.

> 
>>> PYTHONPATH-based installs allow all of this, as long as you add the 
>>> setuptools .egg to PYTHONPATH.
>>
>>
>> Agreed. I think PYTHONPATH-based installations are close to what some 
>> of us
>> need.  I think that having to set the path is a hassle that is possible
>> to overcome and worth doing.  As I've suggested before, since easy
>> install generates startup scripts, it should be possible to for it
>> to generate startup scripts that take care of setting the path.
>>
>> I think it should be possible to have Python startup scripts invoke
>> setuptools' site.py to add necessary .pth files to the path.
> 
> 
> The regular site.py suffices for that.  What it doesn't suffice for is 
> getting those directories on *ahead* of the system-provided libraries.  
> If you don't care about that, then adding a series of statements like 
> this to the script would suffice:
> 
>     __import__('site').addsitedir("some target dir")
> 
> Where the series of directories represents your "instance", in the order 
> of highest to lowest precedence.

Are you worried about system modules that might already have been imported?
I presume that site.py could put the local directories in front of system
directories in sys.path.

> Of course, this is unnecessary *except* for obtaining the location of 
> setuptools itself, unless the goal is to have plugins installed using 
> .pth files, and to manage that setup externally to the actual programs.  
 >
 > (By which I mean, the programs themselves have no notion of a "plugin
 > directory", they simply expect everything to be already set up on
 > sys.path when they run.)

I'm not sure.  I'm about 80% sure that I don't care about .pth files
except setuptool's, but others may care. <shrug>

...

> Well, you haven't really explained your use case enough for me to 
> understand even what these programs are or what people are doing with 
> them.  You just listed some requirements.

Not for lack of trying.


>> My goal is to have self contained working directories.  In addition,
>> I want to have chained self-contained working directories, which is
>> why the excellent "put scripts and eggs in the same place" solution
>> doesn't work for me (more below).
> 
> 
> You could generate a sitecustomize.py in a scripts+eggs directory, and 
> have it do:
> 
>     import site
>     site.addsitedir("this directory")
>     site.addsitedir("another directory")
> 
> etc.  This would basically chain the directories in a way that supports 
> .pth files.  Then just easy_install scripts and eggs to the single 
> target directory, including the target directory in --site-dirs so 
> easy_install knows it can use .pth files.
> 
> Presto - you're done.  Running a script would now suffice to set its 
> environment.  Note that anything it references via .pth files will be 
> *subordinate* to the same things in site-packages, but since you're 
> presumably controlling what Python install is used for this, and in any 
> case the scripts have require() used to bump up anything the script 
> directly requires, this shouldn't be a big issue.
 >
> Note that in this setup, sitecustomize.py is the only configuration file 
> you need, and you don't need any upgrades to setuptools as it exists 
> today to make this work.  Just slap sitecustomize in the same directory 
> with the scripts and eggs, and all is well.

Why must eggs be in the same directory?  Since sitecustomize adds additional
directories, couldn't the eggs be there?

 > You can, if you wish,
> modify it to do other sys.path munging according to your needs.

Yup.


> Unfortunately, due to differences in path processing on Windows and 
> 'nix, the above procedure appears to only work correctly on Windows.  On 
> 'nix, the current dir and/or script dir aren't on sys.path soon enough.  
> *sigh*.

Dang.  You had be going there. :)

> Okay, I give up.  PYTHONPATH manipulation is the only thing that works.  
> After further experimenting, it seems the only consistent way to get 
> sitecustomize.py to work, on all versions and platforms, is to have it 
> on PYTHONPATH when Python starts execution.  In which case, you might as 
> well use the site.py hack and control everything with .pth files.
> 
> Now you've got me wondering if maybe I should always have easy_install 
> create script wrappers that set a fixed PYTHONPATH.  The main problem I 
> see with it is what happens when somebody deliberately sets a different 
> PYTHONPATH.  Do you append to it?  Prepend to it?  Ignore it? Should 
> PYTHONPATH munging be optional?  Mandatory?  Hrm.
> 
> Ignoring PYTHONPATH when running a script has security advantages, to be 
> certain.  Putting the fixed paths in front of the user-supplied ones is 
> almost as good.
> 
> This needs a lot more thought, though. 

Yup.  Thought is good.

 > Unfortunately, it's making me
> aware of some deficiencies in the existing script system, which doesn't 
> allow for e.g. running scripts with -O at the moment, let alone any 
> other options.  These are allowed in "traditional" distutils scripts, 
> but not entry-point based scripts, which have no place to specify these 
> things at the moment.  I'm probably going to have to add a way to 
> control these and other script-generation options at some point.

Yup.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org