[Python-Dev] PEP 370, open questions

[glyph at divmod.com]

On 12:02 pm, ronaldoussoren at mac.com wrote:
>On 17 Jan, 2008, at 9:40, glyph at divmod.com wrote:
>>On 07:55 am, lists at cheimes.de wrote:

>The framework build of Python definitly targets the upper layer of the 
>OSX stack, not just the Unix core. This sometimes causes confusion, 
>but mostly from developers wandering over from a Linux system that 
>complain that OSX isn't Linux.

The framework build targets both layers, as I understand it - and it 
sounds like you're saying it too, since it's not "just" the UNIX core. 
Solaris isn't Linux either, by the way.  These conventions hold across 
far more than one operating system :).
>Note that even Darwin is not Linux, there are several differences that 
>cause problems for naive porters. Two of those: Darwin uses different 
>binary formats for shared libraries and plugins;  the darwin linker 
>hardcodes the full path to shared libraries into executables (without 
>a runtime search path).

Distutils should take care of this distinction in Python's case, no? 
Xxx's autotools generate a shared library, PyXxx's setup.py generates a 
plugin (or "dylib", is that still what they're called these days?).
>>An example: Let's say you have a GNU autotools project in C, which 
>>we'll
>>call "Xxx", and a set of Python bindings, PyXxx.  Combinator deals 
>>with
>>this by using ~/.local, and providing scripts to set up PATH and
>>LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib, 
>>respectively.
>
>~/Library/Combinator would be a better installation root on OSX, that 
>location fits better into guidelines from Apple and also avoids 
>completely hiding the Combinator data from the user.

I disagree, but Combinator's a complex beast and has a lot of other 
moving parts which need to live in specific places.  Its main purpose is 
to manage your import path to easily switch between different 
development branches of multiple projects, and so most of its data is 
already in locations that the user has specified.

A key thing about ~/.local in this case is that it isn't specific to 
Combinator.  It's any per-user installed dependency libraries for 
development purposes, not necessarily on Combinator-managed projects, 
and not even necessarily Python projects.
>This is probably off-topic for python-dev, but how is combinator 
>different from zc.buildout and virtualenv?

We are definitely veering in that direction, but it probably bears a 
brief description just so folks here can see how it does and doesn't 
apply to the PEP.  zc.buildout and virtualenv are primarily 
heterogeneous deployment tools, with development being just a different 
type of deployment.  They're ways of installing Python packages into an 
isolated, configured environment.

Combinator is primarily a development tool.  Although I've used it as a 
deployment tool (as you can use zc.buildout as a development tool) 
that's not really its focus.  Combinator has no "installation" step for 
most projects.  ~/.local is a special case, reserved for common 
unchanging dependencies that require building; most code managed by 
Combinator lives in ~/Projects/YourProject/trunk or 
~/Projects/YourProject/branches.  (I *used* to be a Mac guy.  Can you 
tell? :-))

The idea with zc.buildout is you are installing application A which 
requires library X version Q, and installing application B which 
requires library X version M; you want to keep those separated.  The 
idea with combinator is that you are *developing* application A, and you 
want to make sure that it continues working with both version Q and M, 
so you can easily do

    chbranch X releases/Q  # the most common combinator command
    trial a.test
    chbranch X releases/M
    trial a.test

It also has specific integration with subversion for creating and 
merging branches.  chbranch will try to check out releases/Q if there 
isn't already such a directory present, based on svn metadata.  When you 
create a branch to start working on a feature, your environment is 
automatically reconfigured to import code from that branch.  When you 
merge a branch to trunk, it is adjusted to load code from the merged 
trunk again.  Hopefully some day soon it will also have integration with 
bazaar too.
>Why? Just because users can't remember on which platform they are 
>developing ;-)? That said, there's a libpython.a file in the framework 
>build for basicly that reason: enough users complained about there not 
>being a python library they could link to that it was easier to add a 
>symlink then trying to educate everyone.

The system installation of Python on the mac has a nod in this direction 
as well.  /usr/bin/python is also present, as is /usr/lib/pythonX.Yÿ0Cas 
symlinks between the two locations.
>You could teach Combinator about running configure scripts ;-).

Better yet, perhaps somebody should teach configure about MacOS, and 
about per-user installation ;).  But the real question here is not what 
Combinator should do, but what Python should do.
>This is not a gratuitous inconsistency.  The Mac developers (mostly 
>Jack Jansen, but I was involved as well) decided that following the 
>platform conventions is a good thing.

I agree.  I'm not saying Python shouldn't follow the platform 
conventions of NeXTSTEP^WOpenStep^WYellowBox^WCocoa - is it still called 
Cococa? - but to follow both those conventions and the conventions of 
its other pedigree, at least until autotools knows how to properly 
populate ~/Library/Frameworks.
>As long as the default installation location is still ~/Library/Python/ 
>X.Y for framework builds I wouldn't mind too much.

Is there a default "installation" location?  When we started this part 
of the discussion, it was just which paths should be on sys.path.

I agree, though.  I would like to propose that if the user does not have 
write access to the system installation location, distutils should 
select ~/.local or ~/Library (or %APPDATA%) as appropriate by default. 
The only problem with this proposal is that distutils' --prefix ~/.local 
will pick up on the autotools-friendly ~/.local/include and ~/.local/lib 
for building C extensions, whereas it's not clear where to look for 
where autotools might have put those things in ~/Library.  That is 
arguably a separate issue though, and I think we should close one can of 
worms before opening another, if possible.
>I don't know if the PEP should mention this, but I do agree that this 
>should be in the Python documentation.

The PEP interacts with it, so it should make reference to it, but the 
explanation needs to go somewhere users will see it.