[Pythonmac-SIG] Python 2.3.4 Installation

[Bob Ippolito]

On Nov 28, 2004, at 10:53 AM, tmk wrote:

> First I must confess that I don't really understand the details of the  
> problem of building extensions with multiple versions of Python  
> installed on Panther...

Very few people do :)

> Out of curiosity I just checked and the lib/python2.3/config/Makefile  
> file from my compiled from source python 2.3.4 don't appear to be  
> patched with the patch below.

The correct linker settings are not adopted by any mainline version of  
Python 2.3.  I had thought they were in 2.3.4, but they're not.  Only  
Python 2.4 builds correctly by default.  Also, Apple's build of Python  
2.3.3 in the WWDC 2004 preview  
<http://www.opensource.apple.com/darwinsource/WWDC2004/python-11/>  
adopts a similar method to this patch, but slightly modified so that it  
does not depend on MACOSX_DEPLOYMENT_TARGET being set:

(this is what a good  
/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/ 
config/Makefile should look like, starting on or around line 98)

LDSHARED=   env MACOSX_DEPLOYMENT_TARGET=10.3 $(CC) $(LDFLAGS) -bundle  
-undefined dynamic_lookup
BLDSHARED=  env MACOSX_DEPLOYMENT_TARGET=10.3 $(CC) $(LDFLAGS) -bundle  
-undefined dynamic_lookup

> Am I correct in assuming that this is ok and that the patch below is  
> only meant for the Apple-installed python 2.3?

Unfortunately not, it recently came to my attention that it is VERY  
IMPORTANT TO PATCH THIS MAKEFILE.  Here is some more information  
(listing what will happen with various permutations of common Python  
installations).  I hope to never have to repeat this again by email :)

Versions of Mac OS X prior to 10.3: This patch isn't at all compatible  
because dyld did not support this feature until Darwin 7.  That sucks,  
but there's little that can be done about that!

= Scenario 1 =

[a] Python 2.3.0 in /System/Library, unpatched
[b] Python 2.3.x in /Library, unpatched

Building extensions with [a] no longer works, because -framework Python  
locates [b].  Therefore, it actually ends up building extensions only  
compatible with [b].
Building extensions with [b] correctly creates extensions, but are only  
compatible with [b].
Loading correctly built extensions built for [a] from [b] or vice versa  
will crash the interpreter.

= Scenario 2 =

[a] Python 2.3.0 in /System/Library, patched
[b] Python 2.3.x in /Library, unpatched

Building extensions with [a] will build extensions compatible with [b]  
(or any other Python 2.3.x, even non-framework builds).
Building extensions with [b] correctly creates extensions, but are only  
compatible with [b].
Loading correctly built extensions built for [b] from [a] will crash  
the interpreter.

= Scenario 3 =

[a] Python 2.3.0 in /System/Library, unpatched
[b] Python 2.3.x in /Library, patched

Building extensions with [a] no longer works, because -framework Python  
locates [b].  Therefore, it actually ends up building extensions only  
compatible with [b].
Building extensions with [b] will build extensions compatible with [a]  
(or any other Python 2.3.x, even non-framework builds).
Loading correctly built extensions built for [a] from [b] will crash  
the interpreter.

= Scenario 4 =

[a] Python 2.3.0 in /System/Library, patched
[b] Python 2.3.x in /Library, patched

Building extensions with [a] will build extensions compatible with [b]  
(or any other Python 2.3.x, even non-framework builds) and vice versa.

= Scenario 5 =

[a] Python 2.3.0 in /System/Library, unpatched
[b] Python 2.3.x in /Library, unpatched
[c] Python 2.4.x in /Library <no patch necessary>

Building extensions with [a] no longer works, because -framework Python  
locates [c].  These extensions are not compatible with ANY Python  
(built with 2.3 headers, linked with 2.4).
Building extensions with [b] no longer works, because -framework Python  
locates [c].  These extensions are not compatible with ANY Python  
(built with 2.3 headers, linked with 2.4).
Loading correctly built extensions built for [a] from [b] or vice versa  
will crash the interpreter.

= Scenario 6 =

[a] Python 2.3.0 in /System/Library, patched
[b] Python 2.3.x in /Library, unpatched
[c] Python 2.4.x in /Library <no patch necessary>

Building extensions with [b] no longer works, because -framework Python  
locates [c].  These extensions are not compatible with ANY Python  
(built with 2.3 headers, linked with 2.4).
Loading correctly built extensions built for [b] from [a] will crash  
the interpreter.

= Scenario 7 =

[a] Python 2.3.0 in /System/Library, unpatched
[b] Python 2.3.x in /Library, patched
[c] Python 2.4.x in /Library <no patch necessary>

Building extensions with [a] no longer works, because -framework Python  
locates [c].  These extensions are not compatible with ANY Python  
(built with 2.3 headers, linked with 2.4).
Loading correctly built extensions built for [a] from [b] will crash  
the interpreter.

= Scenario 8 =

[a] Python 2.3.0 in /System/Library, patched
[b] Python 2.3.x in /Library, patched
[c] Python 2.4.x in /Library <no patch necessary>

Building extensions with [b] will build extensions compatible with [a]  
(or any other Python 2.3.x, even non-framework builds) and vice versa.

...

So obviously, it's very important to patch your Python!  The sooner you  
patch, the less extensions you will build that are tied to a particular  
installation location.  Note that nearly all precompiled extensions you  
will find for Python 2.3.0 are generally of the unpatched variety, so  
they aren't compatible with other installations of Python 2.3.x.

Considering [c] can screw up the whole mix, even though it doesn't have  
it's own problems... it turns out that there is a fundamental flaw in  
the way gcc's compiler and linker finds headers and dylibs from  
frameworks!  It can't take version numbers into account!  The runtime  
does just fine, but link time does not.. This is pretty bad!  It's not  
solvable without patching gcc either, because from the source files you  
say <FrameworkName/HeaderName.h> but HeaderName.h doesn't live in a  
directory named FrameworkName, it lives in a directory named Headers,  
so it's not possible to specify an -I that works correctly if you use  
this feature.

There are a few workarounds to versioned framework issues, and they all  
suck.

[1] Keep versioned frameworks in separate places and use -F  
appropriately.
[2] Use -I with only "inside-framework" header names (i.e. <Python.h>  
instead of <Python/Python.h>) and link directly to the framework's  
dylib as if it were an .o file (i.e.  
/Library/Frameworks/Python.framework/Versions/2.3/Python instead of  
-framework Python)
[3] Name your frameworks with the version number in them instead of  
using the versioning capability of frameworks (i.e. Python23.framework)

Note that Python 2.4 uses a hybrid of [1] and [2].  When building the  
Python framework, it uses -F into the build directory, so it always  
links the interpreter correctly.  All sources expect the headers to be  
available via "Python.h" and it uses an appropriate -I to make this  
work.  When building extensions, it uses -undefined dynamic_lookup, so  
it doesn't link directly to any Python.

> Also in a separate but related e-mail you also suggested that one  
> would "also need to set the "MACOSX_DEPLOYMENT_TARGET" environment  
> variable to "10.3" when using this patch."
>
> Could you please elaborate on how/where one should set that  
> environment variable (I set it in ~/.MacOSX/environment.plist).

Setting it there is fine.  You could also, instead, adopt a better  
patch like Apple has as stated above.

-bob