[Pythonmac-SIG] Why do I need PantherPythonFix?
bob at redivi.com
Mon Feb 21 21:40:39 CET 2005
On Feb 21, 2005, at 2:53 PM, Roger Binns wrote:
>> Did you read and understand the versioned frameworks blog entry I
>> linked to? What you're saying has basically no relevance to what I
> I read it all. I see the layout. The section about Mach-O, MH_DYLIB
> and dyld tie directly in to what I was asking about ELF,
> and rpath which you said wasn't relevant.
You quoted something about link-time linker support for frameworks and
then talk about runtime linker support..
>> Under certain circumstances, prior to Python 2.3.5/2.4.1, then
>> extensions built on 2.3 would magically work for neither Python 2.3
>> or 2.4 because they were linked against the 2.4 dylib with 2.3
>> headers, so you'd get an API mismatch (which is a warning not a hard
>> error, but still..).
> Ok, now we are getting closer. How would one make a failed extension
> like this? On the other platforms extensions end up below the relevant
> site-packages directory and distutils ensures the correct header
> and library directories are supplied.
If you have Python 2.X and Python 2.Y both installed, as frameworks, in
the same location, then "-framework Python" will pick the version
marked as Current by way of symlink, which is the one that is installed
last. The headers are picked up by way of -I, which points to the
correct headers. If there is a mismatch, then you will have the API of
Python 2.X due to correct headers but you will be using the symbols
from Python 2.Y because GCC will have linked to the wrong Python.
Extensions linked in this way are stillborn, if loaded into a Python
2.X process then they will end up loading two Pythons in-process and
will get "Fatal Python error: Interpreter not initialized (version
mismatch?)". If loaded into a Python 2.Y process (which would take
some serious effort, since they will be installed to the 2.X
site-packages) then they will get "warning: Python C API version
mismatch for module".
The fix was to not use "-framework Python" and instead point directly
to the correct dylib. These fixes are fixing clearly broken behavior,
not adding features or functionality.
> On Windows the Python interpretter shared library has a version number
> as part of the filename - eg python23.dll. Consequently, once built
> you cannot force an extension to use any other version of Python.
This is true on Mac OS X if NOT using "-undefined dynamic_lookup" (10.2
> On Linux(ELF) the extensions aren't linked to Python in any way. The
> symbols are unresolved in the extension. When the extension is loaded
> into a process, they are resolved at that point against the symbols
> that exist in the running process. Consequently you could try to
> build an extension against 2.3 and load it into a process with a
> 2.4 interpretter. But this never happens in practise because the
> extensions are stored in a Python version specific site-packages
This is true on Mac OS X if using "-undefined dynamic_lookup" (not 10.2
>> When Python is built to target Mac OS X 10.3 or later using the
>> "-undefined dynamic_lookup" linker flag, when using PantherPythonFix
>> on 2.3.0, Python 2.3.5, or Python 2.4.x, then extensions are not even
>> linked to the Python interpreter so this problem goes away and the
>> invariant is preserved.
> Ok, so that gets the same behaviour as ELF. And presumably if
> someone tried hard enough they could end up with an extension
> for 2.3.x running in a process with 2.4.x Python.
You don't have to try *that* hard, but yes.
> On other platforms the extensions always end up in a site-packages
> directory below the relevant Python version. If you have 3 different
> Pythons installed, you build your extensions 3 times.
> Is there some goal to avoid having to do that on Mac?
>> I understand your frustration in trying to understand the issues at
>> hand here, but please know that they are tricky but solved. This
>> topic comes up a lot and I'm pretty tired of talking about it.
> I guess the issue is that things are done differently, there appear to
> be some breakages, and then some unstated goals, and finally lots of
> information about fixes for all this, mixed in with various version
> numbers. The initial thing that came to my mind was "then why do
> things differently". The unstated goals is probably the hardest bit
> to understand.
Things aren't done differently..
> Is there some attempt to make extensions work with multiple versions,
> or did that happen anyway and was a bug? Is there an attempt to make
> extensions compiled on one machine work with the same version of Python
> but installed in a different location on another machine?
There is no attempt to make extensions work on multiple versions of
Python. Extensions built without the fix may have had a mismatch
between Pythons but they didn't actually work in either Python. Yes it
was clearly a bug.
Having extensions that can migrate between Python interpreters
installed in different locations (such as inside of an application
bundle) is definitely a goal, and was achieved with "-undefined
dynamic_lookup" -- but is incompatible with Mac OS X 10.2 and earlier
-- so both behaviors have to be supported.
py2app of course prefers relocatable extensions, but it can solve the
issue the hard way by rewriting the extension's Mach-O load commands to
relocate it at build time, but there are some edge cases where it won't
work (if there isn't enough room in the header to accommodate the new
load commands, which has a very, very, very low chance of occurring --
I have never seen it nor heard any reports of it).
> Jack's list from Friday is a good start since it doesn't require
> understanding. Being in the form of "if your goal is X, then
> install Y" helps a lot. It will also need some idea of what
> environment variables (eg PATH) should be set.
What's hard to understand about "Install this fix if you use Python at
all on Mac OS X 10.3"? If you don't do it, you will probably run
across a problem eventually.
I'd like to add the following goal to the list:
If your goal is to use any version of Python at all on Mac OS X 10.3,
then install PantherPythonFix as soon as possible.
Yes, a tutorial about how to manipulate your PATH and environment plist
is certainly useful information.
>> This can include the Python interpreter, if you are using
>> a Python interpreter shipped with Mac OS X. Thus your application
>> may be "tightly bound" to a particular major version of Mac OS X if
>> you are using the vendor Python.
> What is a major version in this context? (ie is it tightly bound to
> version 10 or 10.3.x?) If the former I don't understand why it is even
> mentioned. If the latter then it appears to be another Mac-ism. The
> Windows Python works on Win95 onwards. The Linux one will work on
> that version of Linux and most other ones released after that version.
Like Python, a major version of OS X is the combination of the first
*two* numbers in the version. So yes, Mac OS X 10.3 is a major
You didn't really include enough context here. What py2app says is
that it will never, under any circumstance, include ANY vendor files.
In other words, it will not (probably illegally) redistribute any files
that Apple dropped on your computer if they are in their original
Therefore, if you build an application with a Python interpreter that
was distributed by Apple, then your application *will not embed
Python*. So, it will *only* work on machines that have the *same*
version of Python installed to the *same location*. Therefore, if Mac
OS X 10.4 does not include Python 2.3, then applications built using
the standard Mac OS X 10.3 interpreter will no longer work. If it does
include Python 2.3, and it is installed in the same location, then it
is going to work.
This is *not* a Mac-ism.
More information about the Pythonmac-SIG