[Python-Dev] PEP 370, open questions

[glyph at divmod.com]

On 07:55 am, lists at cheimes.de wrote:
>A CC of the mail goes to the authors of setuptools, virtual python,
>working env and virtual env. What's your opinion on the PEP? Do you 
>have
>some input and advice for me?

I wrote a similar tool called Combinator 
(http://divmod.org/trac/wiki/DivmodCombinator) which Divmod uses quite 
intensively.  I think a significant portion of the Twisted community 
also uses it for development.  Luckily you don't need to CC me specially 
as I'm here already ;-).  One of the features it provides is a user 
site-packages directory much in the same style that the PEP proposes, so 
I'll comment on my experiences maintaining it.
>The PEP 370 (http://www.python.org/dev/peps/pep-0370) "per user site
>packages directory" has several open questions:
>
>* Are the directories for Windows, Mac and Unix fine?
>* Mac: Should framework and non-framework builds of Python use
>  the same directories?

One of the problems we've encountered over and over again with 
Combinator is that MacOS has a dual personality.  Its personality is not 
just an issue of framework vs. non-framework build, but a fundamental 
question of "is this platform a UNIX or is it not a UNIX".

An example: Let's say you have a GNU autotools project in C, which we'll 
call "Xxx", and a set of Python bindings, PyXxx.  Combinator deals with 
this by using ~/.local, and providing scripts to set up PATH and 
LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, respectively. 
I'm not suggesting that Python come with such a tool (although it might 
be handy, at least until this convention catches on with distributors), 
but it should at least make it clear how one would achieve the following 
straightforwardly:

  cd xxx-0.2.4
  ./configure --prefix ~/.local
  make
  make install
  cd ../pyxxx-0.0.1
  python setup.py install --prefix ~/.local

Using Combinator, the user is now fully set up to "import xxx".  But 
only if they're using Linux, because I made the same mistake (which we 
will probably be correcting at some point soon) of using Python's 
*existing* per-user site-packages directory of ~/Library/Python on the 
mac, and not adding ~/.local.

On the Mac, our user now has a problem: given that ~/Library/Python/ 
doesn't follow the /usr or /usr/local style filesystem layout, 
autotools-based projects will not build their libraries in the right 
places using a single command-line option.  (I guess maybe you could do 
something with --bindir and --sbindir and --exec-prefix or whatever, I 
don't know.  I tend to get bored and wander off when I have to pass more 
than 3 options to a configure script.)

A user familiar with the Mac alone might know what to do at this point 
(to be honest, I don't!), but I do know that people familiar with both 
platforms are confused by this apparently gratuitous inconsistency. 
They follow familiar procedure on a Linux box, it works great, then they 
do the same thing on a Mac (with the same shell, an apparently similar 
version of Python) and it doesn't.  Keep in mind that "./configure 
--prefix" is familiar procedure from a lot more places than Combinator. 
For example, on shared hosting where I didn't have root, I've used this 
same trick without Combinator, building Python itself with --prefix 
~/.local and editing .bashrc to modify the appropriate env vars.

My suggestion would be to have *both* ~/.local *and* ~/Library/Python be 
honored on the Mac, because there really isn't much harm in doing so. 
Perhaps it would make sense for non-framework builds to not honor 
~/Library/Python, but I am pretty certain, based on experience fielding 
requests from confused users, that framework builds should still honor 
~/.local/lib/python.../site-packages.

Windows has this problem less because everything has to be done so 
completely differently.

In any event, the really important thing from my perspective is that the 
PEP should explain how this very common use-case for per-user 
installation of libraries can be accomplished on each of the "big three" 
platforms.  This explanation should be put somewhere that users can 
easily find it when they are building libraries.

I don't know what the "right way" to do this on Windows is though, so I 
can't offer much help there.  Something to do with MinGW and intense 
suffering, I would guess.
>* The patch also adds a usecustomize hook to site. Is it useful and
>  should it stay?

Should this be "usercustomize"?  I thought it was a typo but I see the 
same typo in the PEP.  I have often wished for something like this for 
debugging, and it might have other uses, but there should be a caution 
against abuse :).