[Numpy-discussion] NumPy, shared libraries and ctypes

[Albert Strasheim]

Hello all

With the nice ctypes integration in NumPy, and with Python 2.5 which will
include ctypes around the corner, a remote possibility exists that within
the next year or two, I might not be the only person that wants to use NumPy
with ctypes.

This is probably going to mean that this someone is going to want to build a
shared library for use with ctypes. This is all well and good if you're
using a build tool that knows about shared libraries, but in case this
person is stuck with distutils, here is what we might want to do.

Following this thread from SciPy-dev:

http://projects.scipy.org/pipermail/scipy-dev/2006-April/005708.html

I came up with the following plan.

As it happens, pretending your shared library is a Python extension mostly
works. In your setup.py you can do something like this:

config = Configuration(package_name,parent_package,top_path)
config.add_extension('libsvm_',
                     define_macros=[('LIBSVM_EXPORTS', None),
                                    ('LIBSVM_DLL', None)],
                     sources=[join('libsvm-2.82', 'svm.cpp')],
                     depends=[join('libsvm-2.82', 'svm.h')])

First caveat: on Windows, distutils forces the linker to look for an
exported symbol called init<yourextensionname>. In your code you'll have to
add an empty function like this:

void initlibsvm_() {}

This gets us a compiled Python extension, which also happens to be a shared
library on every platform I know of, which is Linux and Windows.
Counter-examples anyone?.

Next caveat: on Windows, shared libraries aka DLLs, typically have a .dll
extension. However, Python extensions have a .pyd extension.

We have a utility function in NumPy called ctypes_load_library which handles
finding and loading of shared libraries with ctypes. Currently, shared
library extensions (.dll, .so, .dylib) are hardcoded in this function.

I propose we modify this function to look something like this:

def ctypes_load_library(libname, loader_path, distutils_hack=False):
    ...

If distutils_hack is True, instead of the default mechanism (which is
currently hardcoded extensions), ctypes_load_library should do:

import distutils.config
so_ext = distutils.sysconfig.get_config_var('SO')

to figure out the extension it should use to load shared libraries. This
should make it reasonably easy for people to build shared libraries with
distutils and use them with NumPy and ctypes.

Comments appreciated. Someone checking something along these lines into SVN
appreciated more. A solution that doesn't make me want to cry appreciated
most.

Thanks for reading.

Regards,

Albert

P.S. As it happens, the OOF2 guys have already created a SharedLibrary
builder for distutils, but integrating this into numpy.distutils is probably
non-trivial.

http://www.ctcms.nist.gov/oof/oof2.html