On 8/9/06, Albert Strasheim <fullung@gmail.com> wrote:

Next caveat: on Windows, shared libraries aka DLLs, typically have a .dll extension. However, Python extensions have a .pyd extension.

We have a utility function in NumPy called ctypes_load_library which handles finding and loading of shared libraries with ctypes. Currently, shared library extensions (.dll, .so, .dylib) are hardcoded in this function.

I propose we modify this function to look something like this:

def ctypes_load_library(libname, loader_path, distutils_hack=False): ...

If distutils_hack is True, instead of the default mechanism (which is currently hardcoded extensions), ctypes_load_library should do:

import distutils.config so_ext = distutils.sysconfig.get_config_var('SO')

to figure out the extension it should use to load shared libraries. This should make it reasonably easy for people to build shared libraries with distutils and use them with NumPy and ctypes.

Wouldn't it make more sense to just rename the .pyd generated by distutils to .dll or .so? Especially since the .pyd generated by distutils won't actually be a python extension module. This renaming could be automated by a simple python script that wraps distutils. The addition of the init{modulename} function could also be done by that script. --bb

Dear Albert, I have started to use numpy and ctypes together and I've been quite pleased. Thanks for your efforts and writings on the wiki. On the topic of ctypes but not directly following from your email: I noticed immediately that the .ctypes attribute of an array is going to be a de-facto array interface, and wondered whether it would actually be better to write some code that takes the __array_struct__ interface and exposes that as an object with ctypes-providing attributes. This way, it could be used by all software exposing the __array_struct__ interface. Still, even with today's implementation, this could be acheived with numpy.asarray( my_array_struct_object ).ctypes. Back to your email: I don't understand why you're trying to build a shared library with distutils. What's wrong with a plain old c-compiler and linker (and mt.exe if you're using MS VC 8)? You can build shared libraries this way with Makefiles, scons, Visual Studio, and about a billion other solutions that have evolved since early C days. I can understand the desire of getting "python setup.py install" to work, but I suspect spawning an appropriate subprocess to do the compilation would be easier and more robust than attempting to get distutils to do something it's not designed for. (Then again, to see what numpy distutils can do, well, let's just say I'm amazed.) Along these lines, I noticed that ctypes-itself seems to have put some hooks into setup.py to perform at least part of the configure/make dance on linux, although I haven't investigated any further yet. Perhaps that's a better way to go than bending distutils to your will? Finally, the ctypes_load_library() function was broken for me and so I just ended up using the appropriate ctypes calls directly. (I should report this bug, I know, and I haven't yet... Bad Andrew.) But the bigger issue for me is that this is a ctypes-level convenience function, and I can't see why it should be in numpy. Is there any reason it should go in numpy and not into ctypes itself where it would surely receive more review and widespread use if it's useful? Albert Strasheim wrote:

Hello all

With the nice ctypes integration in NumPy, and with Python 2.5 which will include ctypes around the corner, a remote possibility exists that within the next year or two, I might not be the only person that wants to use NumPy with ctypes.

This is probably going to mean that this someone is going to want to build a shared library for use with ctypes. This is all well and good if you're using a build tool that knows about shared libraries, but in case this person is stuck with distutils, here is what we might want to do.

Following this thread from SciPy-dev:

http://projects.scipy.org/pipermail/scipy-dev/2006-April/005708.html

I came up with the following plan.

As it happens, pretending your shared library is a Python extension mostly works. In your setup.py you can do something like this:

config = Configuration(package_name,parent_package,top_path) config.add_extension('libsvm_', define_macros=[('LIBSVM_EXPORTS', None), ('LIBSVM_DLL', None)], sources=[join('libsvm-2.82', 'svm.cpp')], depends=[join('libsvm-2.82', 'svm.h')])

First caveat: on Windows, distutils forces the linker to look for an exported symbol called init<yourextensionname>. In your code you'll have to add an empty function like this:

void initlibsvm_() {}

This gets us a compiled Python extension, which also happens to be a shared library on every platform I know of, which is Linux and Windows. Counter-examples anyone?.

Next caveat: on Windows, shared libraries aka DLLs, typically have a .dll extension. However, Python extensions have a .pyd extension.

We have a utility function in NumPy called ctypes_load_library which handles finding and loading of shared libraries with ctypes. Currently, shared library extensions (.dll, .so, .dylib) are hardcoded in this function.

I propose we modify this function to look something like this:

def ctypes_load_library(libname, loader_path, distutils_hack=False): ...

If distutils_hack is True, instead of the default mechanism (which is currently hardcoded extensions), ctypes_load_library should do:

import distutils.config so_ext = distutils.sysconfig.get_config_var('SO')

to figure out the extension it should use to load shared libraries. This should make it reasonably easy for people to build shared libraries with distutils and use them with NumPy and ctypes.

Comments appreciated. Someone checking something along these lines into SVN appreciated more. A solution that doesn't make me want to cry appreciated most.

Thanks for reading.

Regards,

Albert

P.S. As it happens, the OOF2 guys have already created a SharedLibrary builder for distutils, but integrating this into numpy.distutils is probably non-trivial.

http://www.ctcms.nist.gov/oof/oof2.html

------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Albert Strasheim wrote:

Comments appreciated. Someone checking something along these lines into SVN appreciated more. A solution that doesn't make me want to cry appreciated most.

P.S. As it happens, the OOF2 guys have already created a SharedLibrary builder for distutils, but integrating this into numpy.distutils is probably non-trivial.

http://www.ctcms.nist.gov/oof/oof2.html

I recommend using OOF2's stuff, not the .pyd hack. The latter makes *me* want to cry. If you come up with a patch, post it to the numpy Trac, and I'll check it in. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco