[Patches] [ python-Patches-1174614 ] site enhancements

Fri Jun 3 02:44:43 CEST 2005

Patches item #1174614, was opened at 2005-04-01 00:24
Message generated for change (Comment added) made by etrepum
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1174614&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Library (Lib)
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Submitted By: Bob Ippolito (etrepum)
Assigned to: Nobody/Anonymous (nobody)
Summary: site enhancements

Initial Comment:
The current site.py has three major deficiencies:

(1) All site dirs must exist on the filesystem:  Since PEP 302 (New 
Import Hooks) was adopted, this is not necessarily true.  
sys.meta_path and sys.path_hooks can have valid uses for non-
existent paths.  Even the standard zipimport hook supports in-zip-
file paths (i.e. foo.zip/bar).

(2) The directories added to sys.path by .pth files are not scanned 
for further .pth files.  If they were, you could make life much easier 
on developers and users of multi-user systems.  For example, it 
would be possible for an administrator to drop in a .pth file into the 
system-wide site-packages to allow users to have their own local 
site-packages folder.  Currently, you could try this, but it wouldn't 
work because many packages such as PIL, Numeric, and PyObjC 
take advantage of .pth files during their installation.

(3) To support the above use case, .pth files should be allowed to 
use os.path.expanduser(), so you can toss a tilde in front and do the 
right thing.  Currently, the only way to support (2) is to use an ugly 
"import" pth hook.

Attached is a patch to CVS HEAD that:
(1) Removes the os.path.exists() / os.path.isdir() restrictions
(2) Changes the .pth reader to use addsitedir() (top down) rather than 
sys.path.append()
(3) makepath() uses os.path.expanduser() after the os.path.join().

----------------------------------------------------------------------

>Comment By: Bob Ippolito (etrepum)
Date: 2005-06-02 20:44

Message:
Logged In: YES 
user_id=139309

Did you read my whole comment?  I did come up with an example that is 
not Mac OS X based: CGIs on a web server.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2005-06-02 20:43

Message:
Logged In: YES 
user_id=6380

Sorry, I'm still -1, but I don't have the time to discuss it
further. All your examples seem to be Mac OS X based;
perhaps you can come up with a Mac OS X specific solution?

----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2005-06-02 20:41

Message:
Logged In: YES 
user_id=139309

Well in practice, .pth files aren't used a lot (most people will see maybe 
wxPython, PIL, and Numeric), so those three os.listdir() probably aren't 
going to be significant.  If users are creating pth files themselves (which 
they probably should be doing, during development), then they'd likely 
take advantage of the new features.

As far as power for site admins goes.. why should they have power to 
install Python at all?  Lots of things an admin could do can cause 
surprises.

PYTHONPATH is only useful when dealing with the command prompt 
directly, which is probably not the case for users of Mac OS X and 
Windows.  It shouldn't have to be, anyway.  Mac OS X doesn't even 
ship with a GUI interface to edit login environment variables, and the 
normal rc scripts aren't run in the context of LaunchServices.  Also, 
consider the situation where a user of some shared web server is writing 
CGI scripts that depend on Numeric installed in their home directory.  
They'd need to know to add the following to the top of *EVERY* cgi 
script:

import site
import os
site.addsitedir(os.path.expanduser("~/lib/python" + sys.version[:3] + "/
site-packages"))

If the admin had configured a directory to put things by installing a .pth 
file that pointed there, they simply would have to tell users what flags to 
pass to distutils.

For Mac OS X, there is a workaround, in that ~/Library/Python/2.X/site-
packages is a second site dir, but this doesn't need to be a workaround 
and should be available elsewhere.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2005-06-02 20:28

Message:
Logged In: YES 
user_id=6380

It still looks like a potentially considerable slowdown
(since every directory added by a .pth file is scanned
recursively), and I'm doubting the usefulness.

Why should a site admin have the power to make Python search
in an arbitrary place in my home directory? That could cause
surprises too!

When you are regularly switching between multiple Python
interpreters, wouldn't it make more sense to have a few
aliases (or shell scripts, .bat files, etc.) that set
PYTHONPATH without exporting it? Methinks that many people
have dealt with this issue *without* feeling the need to
patch site.py.

(BTW I'm okay with the other changes, from your description.)

----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2005-06-02 20:21

Message:
Logged In: YES 
user_id=139309

If you read closer, it only scans directories added by .pth files, not all 
directories on sys.path.  This is to facilitate better encapsulation, so an 
admin user can create a .pth file that says "~/lib/python2.4/site-
packages" and any user that starts Python will have their own site-
packages dir, where .pth files work, without resorting to PYTHONPATH.  
This way you can use distutils to install packages into your homedir, 
and they will actually work as expected.

PYTHONPATH is utterly disastrous if you regularly deal with several 
different Python interpreters.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2005-06-02 20:14

Message:
Logged In: YES 
user_id=6380

Hm. Looking for .pth files only in site-packages was
intentional; it will slow down startup considerably if you
have to do an os.listdir() of every directory on sys.path
(even worse if some are on NFS!). I'd recommend against that
part.

----------------------------------------------------------------------

Comment By: Just van Rossum (jvr)
Date: 2005-04-07 04:33

Message:
Logged In: YES 
user_id=92689

Patch looks good, the new functionality is great to have, recommend apply.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1174614&group_id=5470