Options for wrapping C and C++ code for use with Numeric
Hi all, I suddenly seem to have the need for working with a bunch of different existing C and C++ code, so I'm looking for a way to make it easier. I love NumPy, so I really want to use NumPy arrays My needs fall into three categories: 1) Writing custom extension code from scratch: In the past, I've done this by just using the NumPy API, but it seems that I shouldn't have to do all that book keeping myself. 2) Wrapping existing libraries: At the moment, I'd like to wrap Richard Shewchuk's triangle: http://www.cs.cmu.edu/~quake/triangle.html Which is straight C, as well as the Proj4 map projections library, also C, and there may be others. 3) Wrapping in house code, C & C++, that is under development, but I want to be able to use it from Python and C++, and also perhaps to write tests for it in Python. The options I'm looking at are: Pyrex: This seems like perhaps the easiest way to write extension code, but it doesn't do any automatic wrapping. Boost::Python: This looks like a very nice way to write extension modules, and it even appears to already include support for NumPy arrays (which ones? is that support being maintained, and will it support SciPy.base?) SWIG: The has the major advantage of automatically wrapping code. I see this as a particular strength for wrapping our in house code that is under development. In theory, once I've written a bunch of type maps, i can just re-run it whenever the code base changes. For each of these, I'd love to hear what people's experiences have been, and it would be great if anyone can point me to samples that are small but non-trivial. Other options I should consider would be great too. One more question: Should I use NumPy arrays to pass data back and forth between Python and C, or should I use the new array object/protocol instead? If so, has anyone developed examples, SWIG type maps, etc for this? Thanks for any feedback, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi, I found SIWG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs. Regards Bruce On 11/3/05, Chris Barker <Chris.Barker@noaa.gov> wrote:
Hi all,
I suddenly seem to have the need for working with a bunch of different existing C and C++ code, so I'm looking for a way to make it easier. I love NumPy, so I really want to use NumPy arrays My needs fall into three categories:
1) Writing custom extension code from scratch:
In the past, I've done this by just using the NumPy API, but it seems that I shouldn't have to do all that book keeping myself.
2) Wrapping existing libraries:
At the moment, I'd like to wrap Richard Shewchuk's triangle:
http://www.cs.cmu.edu/~quake/triangle.html
Which is straight C, as well as the Proj4 map projections library, also C, and there may be others.
3) Wrapping in house code, C & C++, that is under development, but I want to be able to use it from Python and C++, and also perhaps to write tests for it in Python.
The options I'm looking at are:
Pyrex:
This seems like perhaps the easiest way to write extension code, but it doesn't do any automatic wrapping.
Boost::Python:
This looks like a very nice way to write extension modules, and it even appears to already include support for NumPy arrays (which ones? is that support being maintained, and will it support SciPy.base?)
SWIG:
The has the major advantage of automatically wrapping code. I see this as a particular strength for wrapping our in house code that is under development. In theory, once I've written a bunch of type maps, i can just re-run it whenever the code base changes.
For each of these, I'd love to hear what people's experiences have been, and it would be great if anyone can point me to samples that are small but non-trivial. Other options I should consider would be great too.
One more question: Should I use NumPy arrays to pass data back and forth between Python and C, or should I use the new array object/protocol instead? If so, has anyone developed examples, SWIG type maps, etc for this?
Thanks for any feedback,
-Chris
-- Christopher Barker, Ph.D. Oceanographer
NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Bruce Southey wrote:
Hi, I found SWIG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs.
Yes, but can't I deal with them in writing typemaps, and then let SWIG do the rest of the work? I think I've seen some examples like this somewhere, but it's been a while and I need to go digging more. Anyone got one? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi Chris On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote:
Bruce Southey wrote:
Hi, I found SWIG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs.
Yes, but can't I deal with them in writing typemaps, and then let SWIG do the rest of the work? I think I've seen some examples like this somewhere, but it's been a while and I need to go digging more.
I use SWIG and Blitz++ this way, and it works well. I modified Fernando's typemap to work with templates. See attached (does this need to be modified for Numeric3?). It is easy to create a Blitz matrix from a Numeric Array without copying data. Unfortunately, Blitz jealously guards its data (restricted pointers), so that it is not so easy to do the conversion in the other direction. If anyone knows an answer to this problem, I'd be glad to hear it. Stéfan
Stefan van der Walt wrote:
Hi Chris
On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote:
Bruce Southey wrote:
Hi, I found SWIG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs.
Yes, but can't I deal with them in writing typemaps, and then let SWIG do the rest of the work? I think I've seen some examples like this somewhere, but it's been a while and I need to go digging more.
I use SWIG and Blitz++ this way, and it works well. I modified Fernando's typemap to work with templates. See attached (does this need to be modified for Numeric3?).
Only a little bit. I'll mark the changes
Stéfan
------------------------------------------------------------------------
// -*- C++ -*-
%{ #include <blitz/array.h> #include <blitz/tinyvec.h> #include <Numeric/arrayobject.h>
#include "scipy/arrayobject.h"
%}
namespace blitz {
template <class T, int N> class Array {
%typemap(in) Array<T,N> & (Array<T,N> M) { int T_size = sizeof(T);
blitz::TinyVector<int,N> shape(0); blitz::TinyVector<int,N> strides(0);
int *arr_dimensions = ((PyArrayObject*)$input)->dimensions; int *arr_strides = ((PyArrayObject*)$input)->strides;
for (int i = 0; i < N; i++) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; }
M.reference(blitz::Array<T,N>((T*) ((PyArrayObject*)$input)->data, shape, strides, blitz::neverDeleteData));
$1 = &M; }
}; }
%template(matrix1d) blitz::Array<double, 1>; %template(matrix2d) blitz::Array<double, 2>; %template(matrix3d) blitz::Array<double, 3>;
%template(matrix1f) blitz::Array<float, 1>; %template(matrix2f) blitz::Array<float, 2>; %template(matrix3f) blitz::Array<float, 3>;
On 32-bit platforms, this code would work fine. On 64-bit platforms, you would need to change at least the arr_dimensions and arr_strides arrays to be intp (typedef to an integer the size of a pointer on the platform). -Travis
Chris, I have written an wrapper to some COIN-OR libraries for Python using Swig. In this process I have developed some simple typemaps for numarrays array (floating point arrays). It worked flawlessly. I haven't changed the code for a while, but you may want to take a look at it, specially the numarray.i file where I define the numarray typemaps: http://www.ime.usp.br/~pjssilva/software.html Good luck, Paulo
I'm planning to update my boost numeric wrappers (http://www.eos.ubc.ca/research/clouds/num_util.html) to scipy_core as soon as things settle down. Here's an example of some fortran program wrapped by boost that shows a couple of nice features: i) mirroring of python types in C++, with indexing, etc., ii) transparent memory management (all increfs and decrefs are handled by boost shared pointers) and iii) docstrings. I'd be happy to contribute to a page with some SWIG and boost examples for similar code fragments. #define PY_ARRAY_UNIQUE_SYMBOL PyArrayHandle #include <num_util.h> #include <iostream> #include "boost/scoped_array.hpp" namespace { const char* rcsid = ("@(#) $Id: thermwrap.C,v 1.2 2005/09/05 22:47:57 phil Exp $:"); }; using namespace std; namespace py = boost::python; namespace nbpl = num_util; typedef int wave_unit; extern "C" { void ptqv_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thsp, float& thesp); void ptpsp_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thpt,float& thes, float& thsp, float& thesp); void mccla_(const char* atmos, float* z, float* p, float* t,float* rvden, float* o3den, float* den, int& strnlength); void thinv_(float& p,float& t,float& thsp); void theinv_(float& p,float& t, float& qsp, float& thsp, float& thesp); } py::dict ptqv(float p, float t, float qv) { float psp,tsp,qsp,thsp,thesp; ptqv_(p,t,qv,psp,tsp,qsp,thsp,thesp); py::dict result; result["psp"]=psp; result["tsp"]=tsp; result["qsp"]=qsp; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict ptpsp(float p, float t, float psp) { float qv,tsp,qsp,thpt,thes,thsp,thesp; ptpsp_(p,t,qv,psp,tsp,qsp,thpt,thes,thsp,thesp); py::dict result; result["qv"]=qv; result["tsp"]=tsp; result["qsp"]=qsp; result["thpt"]=thpt; result["thes"]=thes; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict thinv(float p, float thsp) { float t; thinv_(p,t,thsp); py::dict result; result["t"]=t; return result; } py::dict theinv(float p, float t, float the) { //t input is first guess float qv,thsp; theinv_(p,t,qv,thsp,the); py::dict result; result["t"]=t; result["qsp"]=qv; result["thsp"]=thsp; return result; } py::numeric::array mcclatchey(string atmos) { int strlength=atmos.length(); int length=33; boost::scoped_array<float> z(new float[length]); boost::scoped_array<float> p(new float[length]); boost::scoped_array<float> t(new float[length]); boost::scoped_array<float> rvden(new float[length]); boost::scoped_array<float> o3den(new float[length]); boost::scoped_array<float> den(new float[length]); mccla_(atmos.c_str(), z.get(), p.get(), t.get(), rvden.get(), o3den.get(), den.get(),strlength ); std::vector<int> dims; dims.push_back(6); dims.push_back(33); py::numeric::array standsound(nbpl::makeNum(dims, PyArray_DOUBLE)); double* sndPtr=(double*) nbpl::data(standsound); int index; for(int i=0;i<33;++i){ index=i; sndPtr[index]=z[i]; index=33 + i; sndPtr[index]=p[i]; index=2*33 + i; sndPtr[index]=t[i]; index=3*33 + i; sndPtr[index]=rvden[i]; index=4*33 + i; sndPtr[index]=o3den[i]; index=5*33 + i; sndPtr[index]=den[i]; } return standsound; } BOOST_PYTHON_MODULE(thermwrap) { using namespace boost::python; import_array(); numeric::array::set_module_and_type("Numeric", "ArrayType"); scope().attr("RCSID") = rcsid; scope().attr("__doc__") = "wrappers for BettsThermo.f routines: ptqv, ptpsp"; string docstring("input: pressure (hPa), temp (K), qv (kg/kg)\n"); docstring += "output psp (hPa), tsp (K), qsp (kg/kg), thsp (K), thesp (K)"; def("ptqv", ptqv,docstring.c_str()); docstring="input: pressure (hPa), temp (K), psp (hPa)\n"; docstring += "output: qv (kg/kg), tsp (K), qsp (kg/kg), thpt (theta(p,t)), thes (K), \n"; docstring +="thsp (theta(psl,tsl), thesp equiv. potential temp.(theta-e at sl)"; def("ptpsp",ptpsp,docstring.c_str()); def("thinv",thinv,"thinv"); def("theinv",theinv,"theinv"); def("mcclatchey",mcclatchey,"mcclatchey"); }
participants (6)
-
Bruce Southey -
Chris Barker -
Paulo J. S. Silva -
Philip Austin -
Stefan van der Walt -
Travis Oliphant