[Pythonmac-SIG] Why do I need PantherPythonFix?

[Bob Ippolito]

On Feb 21, 2005, at 2:53 PM, Roger Binns wrote:

>> Did you read and understand the versioned frameworks blog entry I 
>> linked to?  What you're saying has basically no relevance to what I 
>> said.
>
> I read it all.  I see the layout.  The section about Mach-O, MH_DYLIB
> and dyld tie directly in to what I was asking about ELF, 
> LD_LIBRARY_PATH
> and rpath which you said wasn't relevant.

You quoted something about link-time linker support for frameworks and 
then talk about runtime linker support..

>> Under certain circumstances, prior to Python 2.3.5/2.4.1, then 
>> extensions built on 2.3 would magically work for neither Python 2.3 
>> or 2.4 because they were linked against the 2.4 dylib with 2.3 
>> headers, so you'd get an API mismatch (which is a warning not a hard 
>> error, but still..).
>
> Ok, now we are getting closer.  How would one make a failed extension
> like this?  On the other platforms extensions end up below the relevant
> site-packages directory and distutils ensures the correct header
> and library directories are supplied.

If you have Python 2.X and Python 2.Y both installed, as frameworks, in 
the same location, then "-framework Python" will pick the version 
marked as Current by way of symlink, which is the one that is installed 
last.  The headers are picked up by way of -I, which points to the 
correct headers.  If there is a mismatch, then you will have the API of 
Python 2.X due to correct headers but you will be using the symbols 
from Python 2.Y because GCC will have linked to the wrong Python.

Extensions linked in this way are stillborn, if loaded into a Python 
2.X process then they will end up loading two Pythons in-process and 
will get "Fatal Python error: Interpreter not initialized (version 
mismatch?)".  If loaded into a Python 2.Y process (which would take 
some serious effort, since they will be installed to the 2.X 
site-packages) then they will get "warning: Python C API version 
mismatch for module".

The fix was to not use "-framework Python" and instead point directly 
to the correct dylib.  These fixes are fixing clearly broken behavior, 
not adding features or functionality.

> On Windows the Python interpretter shared library has a version number
> as part of the filename - eg python23.dll.  Consequently, once built
> you cannot force an extension to use any other version of Python.

This is true on Mac OS X if NOT using "-undefined dynamic_lookup" (10.2 
compatible).

> On Linux(ELF) the extensions aren't linked to Python in any way.  The
> symbols are unresolved in the extension.  When the extension is loaded 
> into a process, they are resolved at that point against the symbols
> that exist in the running process.  Consequently you could try to
> build an extension against 2.3 and load it into a process with a
> 2.4 interpretter.  But this never happens in practise because the
> extensions are stored in a Python version specific site-packages
> directory.

This is true on Mac OS X if using "-undefined dynamic_lookup" (not 10.2 
compatible).

>> When Python is built to target Mac OS X 10.3 or later using the 
>> "-undefined dynamic_lookup" linker flag, when using PantherPythonFix 
>> on 2.3.0, Python 2.3.5, or Python 2.4.x, then extensions are not even 
>> linked to the Python interpreter so this problem goes away and the 
>> invariant is preserved.
>
> Ok, so that gets the same behaviour as ELF.  And presumably if
> someone tried hard enough they could end up with an extension
> for 2.3.x running in a process with 2.4.x Python.

You don't have to try *that* hard, but yes.

> On other platforms the extensions always end up in a site-packages
> directory below the relevant Python version.  If you have 3 different
> Pythons installed, you build your extensions 3 times.
>
> Is there some goal to avoid having to do that on Mac?

No.

>> I understand your frustration in trying to understand the issues at 
>> hand here, but please know that they are tricky but solved.  This 
>> topic comes up a lot and I'm pretty tired of talking about it.
>
> I guess the issue is that things are done differently, there appear to 
> be some breakages, and then some unstated goals, and finally lots of 
> information about fixes for all this, mixed in with various version 
> numbers.  The initial thing that came to my mind was "then why do
> things differently".  The unstated goals is probably the hardest bit 
> to understand.

Things aren't done differently..

> Is there some attempt to make extensions work with multiple versions,
> or did that happen anyway and was a bug?  Is there an attempt to make
> extensions compiled on one machine work with the same version of Python
> but installed in a different location on another machine?

There is no attempt to make extensions work on multiple versions of 
Python.  Extensions built without the fix may have had a mismatch 
between Pythons but they didn't actually work in either Python.  Yes it 
was clearly a bug.

Having extensions that can migrate between Python interpreters 
installed in different locations (such as inside of an application 
bundle) is definitely a goal, and was achieved with "-undefined 
dynamic_lookup" -- but is incompatible with Mac OS X 10.2 and earlier 
-- so both behaviors have to be supported.

py2app of course prefers relocatable extensions, but it can solve the 
issue the hard way by rewriting the extension's Mach-O load commands to 
relocate it at build time, but there are some edge cases where it won't 
work (if there isn't enough room in the header to accommodate the new 
load commands, which has a very, very, very low chance of occurring -- 
I have never seen it nor heard any reports of it).

> Jack's list from Friday is a good start since it doesn't require
> understanding.  Being in the form of "if your goal is X, then
> install Y" helps a lot.  It will also need some idea of what
> environment variables (eg PATH) should be set.

What's hard to understand about "Install this fix if you use Python at 
all on Mac OS X 10.3"?  If you don't do it, you will probably run 
across a problem eventually.

I'd like to add the following goal to the list:
If your goal is to use any version of Python at all on Mac OS X 10.3, 
then install PantherPythonFix as soon as possible.

Yes, a tutorial about how to manipulate your PATH and environment plist 
is certainly useful information.

>>  This can include the Python interpreter, if you are using
>>  a Python interpreter shipped with Mac OS X.  Thus your application  
>> may be "tightly bound" to a particular major version of Mac OS X if 
>> you  are using the vendor Python.
>
> What is a major version in this context?  (ie is it tightly bound to
> version 10 or 10.3.x?) If the former I don't understand why it is even 
> mentioned.  If the latter then it appears to be another Mac-ism.  The 
> Windows Python works on Win95 onwards.  The Linux one will work on 
> that version of Linux and most other ones released after that version.

Like Python, a major version of OS X is the combination of the first 
*two* numbers in the version.  So yes, Mac OS X 10.3 is a major 
version.

You didn't really include enough context here.  What py2app says is 
that it will never, under any circumstance, include ANY vendor files.  
In other words, it will not (probably illegally) redistribute any files 
that Apple dropped on your computer if they are in their original 
locations.

Therefore, if you build an application with a Python interpreter that 
was distributed by Apple, then your application *will not embed 
Python*.  So, it will *only* work on machines that have the *same* 
version of Python installed to the *same location*.  Therefore, if Mac 
OS X 10.4 does not include Python 2.3, then applications built using 
the standard Mac OS X 10.3 interpreter will no longer work.  If it does 
include Python 2.3, and it is installed in the same location, then it 
is going to work.

This is *not* a Mac-ism.

-bob