[Pythonmac-SIG] [PATCH]: Python should use dlopen() on 10.3+

[Zachary Pincus]

Hello folks,

Here is a patch to make Python on OS X 10.3 and above use dlopen()  
(via dynload_shlib.c) to load extension modules, and to make the dl  
module avaliable. It ought to be incorporated into the SVN for python  
2.5, if not for further development on 2.4.

If this is an inappropriate channel for this patch, please let me  
know to where I should direct it. Bob Ippolito initially suggested  
this patch be sent hither.

Zach Pincus
Program in Biomedical Informatics and Department of Biochemistry
Stanford University School of Medicine


RATIONALE
---------
On most other unix-like operating systems, Python uses the dlopen()  
call to load extension modules. In addition, the way that these  
modules are opened can be modified via 'sys.setdlopenflags(...)'.

Modifications of how extensions are loaded are useful for several  
reasons (enough so that the standard Python docs (e.g. http:// 
docs.python.org/lib/module-sys.html ). In particular, if multiple  
modules need to share symbols, a call to sys.setdlopenflags is  
necessary.

Symbol sharing is especially important for interoperability of  
modules which wrap C++ classes, because GCC creates classes that  
resolve their run-time type identification by symbol identity. Thus,  
symbols must be shared globally for many C++ features to work  
properly for objects passed between modules.

On OS X / Darwin, Python uses some NeXT-derived APIs to load modules.  
Though these APIs provide analogues to the dlopenflags used to  
control how dlopen() loads modules, this interface is *not* exposed  
to the Python interpreter. Worse, sys.setdlopenflags remains  
available on Darwin, though calls to it are never heeded.

Fortunately, on OS X 10.3 and above, Apple has included dlopen as a  
standard function. In 10.3, this call is provided by a compatibility  
API; in 10.4, the dlopen() call is written to interface directly with  
the library loading mechanism and is now the recommended method for  
libraries to be opened for non Carbon/Cocoa tools.

IMPLEMENTATION
--------------
This (trivial) patch instructs the Python build process to use  
dynload_shlib.c (which uses dlopen) instead of dynload_next.c (which  
uses the NeXT-derived APIs). It also allows for the dl module to be  
built in order to provide access to the proper values for the various  
dlopen flags.

TESTING
-------
This patch can be configured and built into executables that build  
and test correctly on 10.3 and 10.4. The patch is nominally targeted  
at the SVN head as of today, but can be applied (and I have done so  
with success) to Python 2.4.2.

Because Python 2.5 and 2.4 do not currently compile properly on OS X  
10.2, I have not built or tested this patch on that OS version.  
However, the configure and compile process does select the  
appropriate dynload_next.c file to use, and compiles that correctly  
before breaking elsewhere. Thus, if the other errors are fixed for  
10.2, these patches will work fine. (This is because they only change  
Python's behavior for 10.3 and up.)

PATCHES
-------
Here are three sets of patches, as unified diffs.

FIRST: Patch the configure.in file to use dynload_shlib.c when it  
can. Also regenerate the configure script.

Index: configure.in
===================================================================
--- configure.in        (revision 43150)
+++ configure.in        (working copy)
@@ -2105,7 +2105,8 @@
         ;;
         BeOS*) DYNLOADFILE="dynload_beos.o";;
         hp*|HP*) DYNLOADFILE="dynload_hpux.o";;
-       Darwin/*) DYNLOADFILE="dynload_next.o";;
+       # Use dynload_next.c only on 10.2 and below, which don't have  
native dlopen()
+       Darwin/@<:@0123456@:>@.*) DYNLOADFILE="dynload_next.o";;
         atheos*) DYNLOADFILE="dynload_atheos.o";;
         *)
         # use dynload_shlib.c and dlopen() if we have it; otherwise  
stub
Index: configure
===================================================================
--- configure   (revision 43150)
+++ configure   (working copy)
@@ -1,5 +1,5 @@
#! /bin/sh
-# From configure.in Revision: 42437 .
+# From configure.in Revision: 42563 .
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.59 for python 2.5.
#
@@ -13974,7 +13974,8 @@
         ;;
         BeOS*) DYNLOADFILE="dynload_beos.o";;
         hp*|HP*) DYNLOADFILE="dynload_hpux.o";;
-       Darwin/*) DYNLOADFILE="dynload_next.o";;
+       # Use dynload_next.c only on 10.2 and below, which don't have  
native dlopen()
+       Darwin/[0123456].*) DYNLOADFILE="dynload_next.o";;
         atheos*) DYNLOADFILE="dynload_atheos.o";;
         *)
         # use dynload_shlib.c and dlopen() if we have it; otherwise  
stub



SECOND: Allow the dl module to be built and tested


Index: setup.py
===================================================================
--- setup.py    (revision 43150)
+++ setup.py    (working copy)
@@ -883,7 +883,7 @@
          if sys.maxint == 0x7fffffff:
              # This requires sizeof(int) == sizeof(long) == sizeof 
(char*)
              dl_inc = find_file('dlfcn.h', [], inc_dirs)
-            if (dl_inc is not None) and (platform not in ['atheos',  
'darwin']):
+            if (dl_inc is not None) and (platform not in ['atheos']):
                  exts.append( Extension('dl', ['dlmodule.c']) )
          # Thomas Heller's _ctypes module
Index: Lib/test/test_dl.py
===================================================================
--- Lib/test/test_dl.py (revision 43150)
+++ Lib/test/test_dl.py (working copy)
@@ -10,6 +10,7 @@
      ('/usr/lib/libc.so', 'getpid'),
      ('/lib/libc.so.6', 'getpid'),
      ('/usr/bin/cygwin1.dll', 'getpid'),
+    ('/usr/lib/libc.dylib', 'getpid'),
      ]
for s, func in sharedlibs:



THIRD: (optional) Tell Python that the dl test is not expected to be  
skipped anymore. This is optional because if Python is ever built on  
10.2, the test script will expect dl to work, when it only works on  
10.3 and above. However, if Python on 10.2 is officially not  
supported, then this change should be made to properly test the dl  
functionality.


Index: Lib/test/regrtest.py
===================================================================
--- Lib/test/regrtest.py        (revision 43150)
+++ Lib/test/regrtest.py        (working copy)
@@ -930,7 +930,6 @@
          test_cd
          test_cl
          test_curses
-        test_dl
          test_gdbm
          test_gl
          test_imgfile