[Distutils] Trouble building visual python extension on MacOSX 10.8... with python3.2

[Ned Deily]

In article <B29EB804-3977-401C-97F5-62D39AE869D1 at spvi.com>,
 Steve Spicklemire <steve at spvi.com> wrote:
> I'm hoping someone on the distutils list may have some hints about 
> discovering the incantation necessary to produce a working boost-linked 
> python extension for ptyhon.org Python-3.2.3 on MacOS 10.8.

Sorry, I don't have any experience with boost but I can give you a few 
general hints.
 
> Here's my situation. I'm attempting to help out with a wxPython port of 
> visual python, or vpython <http://www.vpython.org> and particularly at 
> <https://github.com/BruceSherwood/vpython-wx>. I cooked up a setup.py script 
> that works fine (i.e., builds the extension) on python2.7 for both MacOSX 
> 10.6, 10.7 and 10.8 (using the python.org 32/64 binary for python-2.7.3). I'm 
> attempting to use the same, or similar script to build the same extension 
> using python3.2, but while it works OK on MacOSX 10.6, it fails to produce a 
> working extension on MacOS 10.8 using the same MacOSX 10.6 SDK. 
> 
> Here's the setup.py: <https://dl.dropbox.com/u/20562746/dist-build/setup.py>
> 
> At first I thought it was different versions of Xcode, but no, that can't be 
> on the 10.8 system I have Xcode 3.2.5 installed and defined the link:
> 
> /Developer -> /Developer-3.2.5
> 
> With the link I have the *exact* *same* files in 
> /Developer/SDKs/MacOSX10.6.sdk on both systems. 
> 
> The setup.py script produces exactly the same output on both systems:
> 
> On 10.6 -> <https://dl.dropbox.com/u/20562746/dist-build/build_on_10.6.txt>
> 
> On 10.8 -> <https://dl.dropbox.com/u/20562746/dist-build/build_on_10.8.txt>
> 
> Except for one error message in the 10.6 build:
> 
> ld: warning: in 
> /Developer/SDKs/MacOSX10.6.sdk/Library/Frameworks//Python.framework/Python, 
> missing required architecture x86_64 in file
> 
> (but I'm not smart enough to tell how that matters.. I'm not using that 
> Python Framework!)

When you specify an SDK to the compiler front-end via sysroot, searches 
for headers and libraries files may be automatically prefixed with the 
SDK.  So, for example, specifying -isysroot 
/Developer/SDKs/MacOSX10.6.sdk along with -framework Python will search 
for the framework in:

        /Developer/SDKs/MacOSX10.6.sdk/Library/Frameworks
then /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks

So you should make sure that, in all of the SDKs that you use, 
/Developer/SDks/.../Library/Frameworks points to the right thing.  You 
may want it to be symlinked to your /Library/Frameworks directory if 
that's where you have python3.2 installed.  I believe that was the 
default case with Xcode 3, although with the last release of Xcode 3 
(3.2.6) I recall that there was a problem with the symlinks that it 
generated; not sure about 3.2.5.  With the latest versions of Xcode 4, I 
believe it is the case that no symlinks are created automatically.  So 
you either have to create them yourself or install copies of the 
frameworks you want directly into the SDKs Library directory.  (Note, 
don't alter the SDK System/Library directory there!)

Also, be aware that with current Xcode 4's, the SDKs are no longer 
installed into /Developer, in fact, nothing is.   They are included 
within Xcode.app itself.  You can use xcodebuild with either Xcode 3 or 
4 to find the SDK location:
  xcodebuild -version -sdk macosx Path
  xcodebuild -version -sdk macosx10.6 Path

Another thing - and this may be significant.  When using -framework 
Python, I expect that build tool chain will follow the Versions/Current 
link in the framework to find the appropriate version.  As it stands, 
Python 3 installs normally do not change that Current link.  So it may 
be possible that Python 2 include files and libraries are being used 
wherever you have -framework Python.  So you should ensure that, during 
the build, the Library/Frameworks/Python.frameworks linked to from the 
SDK has Current pointing to 3.2.  Or change the build to avoid 
-framework Python and link to the desired Framework -I and -L 
directories directly.
 
> Finally.. I thought I'd try to inspect the produced .so file. With otool I 
> figured out how to dump the symbol table (-vI) and found that while they have 
> exactly the same size, they have very different symbol tables:
> 
> 10.6 -> <https://dl.dropbox.com/u/20562746/dist-build/t6.txt>
> 
> 10.8 -> <https://dl.dropbox.com/u/20562746/dist-build/t8.txt>

Have you checked the dynamic link dependencies of the built .so files 
with otool -L to make sure any dynamic links are to the paths of the 
frameworks and libraries you expect?

In general, when building a complex project intended to run on multiple 
releases and with so many pieces like this outside of Xcode, my 
experience is that the safest approach is to be conservative and build 
on the oldest supported system.  Yes, it is possible to install older 
Xcode releases on newer systems and build there but it is also really 
easy for a slip-up to occur and not be noticed until it's too late.  For 
example, up until recently in the builds of Python itself, there was a 
long-standing undetected bug along these lines where the system version 
of the sqlite3 headers was mistakenly being picked up rather than the 
SDK version.

I should also mention that there are some unfortunate differences in OS 
X specific Distutils behavior between Python 3.2.x and 2.7.x , 
particularly in regard to dealing with extension module building on 
newer OS X systems in the absence of SDKs.  I *believe* those 
differences should be all sorted out in the upcoming 3.2.4 and 2.7.4 
maintenance releases, which should be out in the near future.  From a 
very quick glance at what you are doing here, I can't think of anything 
that would affect it, though.

Good luck!

-- 
 Ned Deily,
 nad at acm.org