Hi,
I want to wrap some code I've done in the past with a custom array and pass numpy arrays to it. So I need to transform numpy arrays to my arrays at the construction of an instance of my class, as well as each call to a method (pass by value). Then, some method return by value an array I have to transform back in numpy arrays. I imagine that such wrappers must be done in the .i file... I'm still new to SWIG, even though I understand the easy wrapping, such wrapping is a bit difficult for me at the moment. I do not understand for instance the numpy.i wrappers, what they do, ... Is there a simpler doc somewhere to explain this ? Or a simple SWIG tutorial and a Numpy interface tutorial - Travis' book ? - ?
Matthieu
Matthieu Brucher wrote:
Hi,
I want to wrap some code I've done in the past with a custom array and pass numpy arrays to it. So I need to transform numpy arrays to my arrays at the construction of an instance of my class, as well as each call to a method (pass by value). Then, some method return by value an array I have to transform back in numpy arrays. I imagine that such wrappers must be done in the .i file... I'm still new to SWIG, even though I understand the easy wrapping, such wrapping is a bit difficult for me at the moment. I do not understand for instance the numpy.i wrappers, what they do, ... Is there a simpler doc somewhere to explain this ? Or a simple SWIG tutorial and a Numpy interface tutorial - Travis' book ? - ?
Matthieu
Hello Matthieu :)
You should give ctypes a try, I find it much better than swig most of the time for wrapping. You can find some doc here:
http://www.scipy.org/Cookbook/Ctypes2
Basically, once you get your dll/so with a function foo(double *a, int n), you can call it directly in numpy by passing directly a numpy array. The catch is that the layout of the array should be exactly the same than your function expects (most of the time a contiguous array), but there are numpy functions which enforce that. As long as you are not calling thousand of function with small arrays, the wrapping is pretty efficient in my experience.
cheers,
David
You should give ctypes a try, I find it much better than swig mostof the time for wrapping. You can find some doc here:
http://www.scipy.org/Cookbook/Ctypes2 Basically, once you get your dll/so with a function foo(double *a,int n), you can call it directly in numpy by passing directly a numpy array. The catch is that the layout of the array should be exactly the same than your function expects (most of the time a contiguous array), but there are numpy functions which enforce that. As long as you are not calling thousand of function with small arrays, the wrapping is pretty efficient in my experience.
cheers, David
Thanks for the fast answer ;) I was wondering if ctypes would fit better, but in this case, I have to make the wrapper myself, I suppose ? Here is the basic prototype of what I have - it's a cost function I want to optimize with optimizers I proposed on scipy recently - :
template<class MatrixType> struct CostFunction { CostFunction(const MatrixType& points, ...) { }
const MatrixType gradient(const Matrixtype& parameters) { // ... return aNewGradient; } };
So I have to create a function that takes a numpy array and that returns and instance of my cost function, this is pretty simple to do, the matrix can be constructed from its two dimensions and a pointer, but for the gradient method, I have trouble seeing how to do wrap it :(
Matthieu
Matthieu Brucher wrote:
You should give ctypes a try, I find it much better than swig most of the time for wrapping. You can find some doc here: http://www.scipy.org/Cookbook/Ctypes2 Basically, once you get your dll/so with a function foo(double *a, int n), you can call it directly in numpy by passing directly a numpy array. The catch is that the layout of the array should be exactly the same than your function expects (most of the time a contiguous array), but there are numpy functions which enforce that. As long as you are not calling thousand of function with small arrays, the wrapping is pretty efficient in my experience. cheers, DavidThanks for the fast answer ;) I was wondering if ctypes would fit better, but in this case, I have to make the wrapper myself, I suppose ? Here is the basic prototype of what I have - it's a cost function I want to optimize with optimizers I proposed on scipy recently - :
template<class MatrixType> struct CostFunction { CostFunction(const MatrixType& points, ...) { }
const MatrixType gradient(const Matrixtype& parameters) { // ... return aNewGradient; } };
So I have to create a function that takes a numpy array and that returns and instance of my cost function, this is pretty simple to do, the matrix can be constructed from its two dimensions and a pointer, but for the gradient method, I have trouble seeing how to do wrap it :(
I forgot you are using C++. Using C++ with ctypes makes the wrapping more difficult, I think, specially since you are using templates: you would have to write a C wrapper around your classes, and then using ctypes on the wrapper (there may be better ways, but I don't see an obvious one). Basically, using ctypes is like using code from a dynamically loaded library (ala VST :) ). But writing some C code kills one of the strong point of ctypes (doing all the wrapping in python).
I have attached a simplistic example which show how to wrap some functions.
David
I forgot you are using C++.
Yes, all my code is in C++, and in fact the code I use in Python should be object-oriented as well, that's why it is not that easy to define how I'm going to do this...
Using C++ with ctypes makes the wrapping
more difficult, I think, specially since you are using templates: you would have to write a C wrapper around your classes, and then using ctypes on the wrapper (there may be better ways, but I don't see an obvious one). Basically, using ctypes is like using code from a dynamically loaded library (ala VST :) ).
That works great for C then, not that well for C++...
But writing some C code kills
one of the strong point of ctypes (doing all the wrapping in python).
Yes, but even with SWIG, I think I'll have to write wrapper in C or in SWIG to transform one array in my matrix and vice versa. Well, that should be a great example for my book ;)
I have attached a simplistic example which show how to wrap some functions.
Many thanks, it is simple, I only have to provide the wrappers :| and I have to do something with the class instance :| I'll think a little bit about how to do this properly and efficiently
Matthieu
On 4/19/07, Matthieu Brucher matthieu.brucher@gmail.com wrote:
I forgot you are using C++.
Yes, all my code is in C++, and in fact the code I use in Python should be object-oriented as well, that's why it is not that easy to define how I'm going to do this...
Doing the wrapping in an object oriented way is difficult, and maybe not that useful. This does not prevent the API exposed to python to be OO, of course.
Using C++ with ctypes makes the wrapping more difficult, I think, specially since you are using templates: you would have to write a C wrapper around your classes, and then using ctypes on the wrapper (there may be better ways, but I don't see an obvious one). Basically, using ctypes is like using code from a dynamically loaded library (ala VST :) ).
That works great for C then, not that well for C++...
Well, this is an inherent problem of C++ when you try to use it from other languages, but let's not start this (again :) ),
Yes, but even with SWIG, I think I'll have to write wrapper in C or in SWIG to transform one array in my matrix and vice versa. Well, that should be a great example for my book ;)
I have attached a simplistic example which show how to wrap some
functions.
Many thanks, it is simple, I only have to provide the wrappers :| and I have to do something with the class instance :| I'll think a little bit about how to do this properly and efficiently
The example shows basic wrapping, and some facilities provided by numpy to help. Again, ctype is pretty efficient as long as you do not need to do convertion. If you call it thousand of times, it will be slow, but this is more or less inherent to python (function calls are nowhere near as fast as in a compiled language, at least for now).
SWIG may be better in your case because it is aware of C++ classes, and is *compiling* an extension, whereas ctypes does not compile anything. This means that you do not have to care about binary interfaces problemes between foreign object codes. SWIG parses a prett good deal of C++, and is aware of classes (template is another matter, obviously). numpy sources contain swig code to process automatically numpy arrays (that is convert C representation of a numpy array to a simple C array and vice et versa), if I remember correctly.
There is also boost.python, but I don't know if its situation towards numpy array has changed (eg there was some code lying around to represent numpy arrays in C++).
If I were you, and if there are only a few classes and a few member functions, I would try the C wrapper called from python first.
David
Doing the wrapping in an object oriented way is difficult, and maybe not that useful. This does not prevent the API exposed to python to be OO, of course.
I have some difficulties to do this in an automated way... I'm trying now to make a derived object from my function, without templates and, I'm hoping so, with a correct interface i.e. double + 2*int to save make the conversions with numpy arrays.
That works great for C then, not that well for C++...
Well, this is an inherent problem of C++ when you try to use it from other languages, but let's not start this (again :) ),
:D
The example shows basic wrapping, and some facilities provided by numpy to help. Again, ctype is pretty efficient as long as you do not need to do convertion. If you call it thousand of times, it will be slow, but this is more or less inherent to python (function calls are nowhere near as fast as in a compiled language, at least for now).
It's for optimization, so the function will be called several hundreds of times, I suppose, and I tried porting the whole function to Python, but I'm not sure that the Python version behaves like the C++ version - the results are not identic, so... -, thus the wrapping.
SWIG may be better in your case because it is aware of C++ classes,
and is *compiling* an extension, whereas ctypes does not compile anything. This means that you do not have to care about binary interfaces problemes between foreign object codes. SWIG parses a prett good deal of C++, and is aware of classes (template is another matter, obviously). numpy sources contain swig code to process automatically numpy arrays (that is convert C representation of a numpy array to a simple C array and vice et versa), if I remember correctly.
Yes, I will try to use this solution, I have trouble figuring how passing the output numpy array, Bill Baxter asked the same question today, at exactly the same time ;) Well, I saw on the docs that such arrays must be passed to the function, and already allocated, and that is a problem for me...
There is also boost.python, but I don't know if its situation towards
numpy array has changed (eg there was some code lying around to represent numpy arrays in C++).
That will be my next quest during the summer ;)
If I were you, and if there are only a few classes and a few member
functions, I would try the C wrapper called from python first.
It's only one class and one method + the constructor. Not much but I'm a real beginner in that domain. True, I could use the C API directly...
Matthieu
This may be of no help at all but I see mentions of C++/Python/OO/SWIG and it triggers me to mention something I heard about recently called "PyCXX": http://cxx.sourceforge.net/ I /think/ the idea behind it is to basically make a C++ version of the Python C API. You still do all the wrapping yourself, but it's much less painful.
--bb
On 4/20/07, Matthieu Brucher matthieu.brucher@gmail.com wrote:
Doing the wrapping in an object oriented way is difficult, and maybe not that useful. This does not prevent the API exposed to python to be OO, of course.
I have some difficulties to do this in an automated way... I'm trying now to make a derived object from my function, without templates and, I'm hoping so, with a correct interface i.e. double + 2*int to save make the conversions with numpy arrays.
That works great for C then, not that well for C++...
Well, this is an inherent problem of C++ when you try to use it from other languages, but let's not start this (again :) ),
:D
The example shows basic wrapping, and some facilities provided by numpy to help. Again, ctype is pretty efficient as long as you do not need to do convertion. If you call it thousand of times, it will be slow, but this is more or less inherent to python (function calls are nowhere near as fast as in a compiled language, at least for now).
It's for optimization, so the function will be called several hundreds of times, I suppose, and I tried porting the whole function to Python, but I'm not sure that the Python version behaves like the C++ version - the results are not identic, so... -, thus the wrapping.
SWIG may be better in your case because it is aware of C++ classes, and is *compiling* an extension, whereas ctypes does not compile anything. This means that you do not have to care about binary interfaces problemes between foreign object codes. SWIG parses a prett good deal of C++, and is aware of classes (template is another matter, obviously). numpy sources contain swig code to process automatically numpy arrays (that is convert C representation of a numpy array to a simple C array and vice et versa), if I remember correctly.
Yes, I will try to use this solution, I have trouble figuring how passing the output numpy array, Bill Baxter asked the same question today, at exactly the same time ;) Well, I saw on the docs that such arrays must be passed to the function, and already allocated, and that is a problem for me...
There is also boost.python, but I don't know if its situation towards numpy array has changed (eg there was some code lying around to represent numpy arrays in C++).
That will be my next quest during the summer ;)
If I were you, and if there are only a few classes and a few member functions, I would try the C wrapper called from python first.
It's only one class and one method + the constructor. Not much but I'm a real beginner in that domain. True, I could use the C API directly...
Matthieu _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Matthieu Brucher wrote:
Doing the wrapping in an object oriented way is difficult, and maybe not that useful. This does not prevent the API exposed to python to be OO, of course.I have some difficulties to do this in an automated way... I'm trying now to make a derived object from my function, without templates and, I'm hoping so, with a correct interface i.e. double + 2*int to save make the conversions with numpy arrays.
> That works great for C then, not that well for C++... Well, this is an inherent problem of C++ when you try to use it from other languages, but let's not start this (again :) ),:D
The example shows basic wrapping, and some facilities provided by numpy to help. Again, ctype is pretty efficient as long as you do not need to do convertion. If you call it thousand of times, it will be slow, but this is more or less inherent to python (function calls are nowhere near as fast as in a compiled language, at least for now).It's for optimization, so the function will be called several hundreds of times, I suppose, and I tried porting the whole function to Python, but I'm not sure that the Python version behaves like the C++ version
- the results are not identic, so... -, thus the wrapping.
A couple of hundred times is OK if the called function is doing something. I once benchmarked ctypes with respect to function calling, and I could do several hundred of thousand of calls/s, if I remember correctly.
It's only one class and one method + the constructor. Not much but I'm a real beginner in that domain. True, I could use the C API directly...
Ok, I have a simple working example. It is actually much easier than I thought, because no C compiler is involved at all (I was afraid about object layout, vtables and other horrors), only C++.
I attached the example: it shows how to mimic a C++ class in python through a C-like interface. The main difficulty is to be sure that your object get deleted by the python interpreter, which means having a __del__ function. The problem is that you cannot control how python will destroy its things: it may "destroy" ctypes module before your object, which is problematic since you need it to destroy your object. The idea is to "force" python to keep everything in the namespace through a fake second argument to the destructor.
http://docs.python.org/ref/customization.html (comments on __del__ ).
David
CC = colorgcc CXX = colorgcc
LD = g++
PYTHONINC = -I/usr/include/python2.5 NUMPYINC = -I/usr/lib/python2.5/site-packages/numpy/core/include
WARN = -W -Wall
CFLAGS = $(WARN) $(PYTHONINC) $(NUMPYINC)
hellocpp.so: hellocpp.o $(LD) -shared -o $@ $< -Wl,-soname,$@
hellocpp.o: hellocpp.cpp $(CXX) -c $(CFLAGS) -fPIC $<
clean: rm -f *.o rm -f *.so
Well, that's easy ;) OK, I have to digg in for the transformations of numpy arrays, knowing that I have other parameters. But for this, the Cookbook at scipy should help me a lot. Thanks for the help ;)
Matthieu
Ok, I have a simple working example. It is actually much easier than I
thought, because no C compiler is involved at all (I was afraid about object layout, vtables and other horrors), only C++.
I attached the example: it shows how to mimic a C++ class in python through a C-like interface. The main difficulty is to be sure that your object get deleted by the python interpreter, which means having a __del__ function. The problem is that you cannot control how python will destroy its things: it may "destroy" ctypes module before your object, which is problematic since you need it to destroy your object. The idea is to "force" python to keep everything in the namespace through a fake second argument to the destructor.
http://docs.python.org/ref/customization.html (comments on __del__ ).
David
CC = colorgcc CXX = colorgcc
LD = g++
PYTHONINC = -I/usr/include/python2.5 NUMPYINC = -I/usr/lib/python2.5/site-packages/numpy/core/include
WARN = -W -Wall
CFLAGS = $(WARN) $(PYTHONINC) $(NUMPYINC)
hellocpp.so: hellocpp.o $(LD) -shared -o $@ $< -Wl,-soname,$@
hellocpp.o: hellocpp.cpp $(CXX) -c $(CFLAGS) -fPIC $<
clean: rm -f *.o rm -f *.so
Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
2007/4/20, Matthieu Brucher matthieu.brucher@gmail.com:
Well, that's easy ;) OK, I have to digg in for the transformations of numpy arrays, knowing that I have other parameters. But for this, the Cookbook at scipy should help me a lot. Thanks for the help ;)
Matthieu
Some news, I finished the wrappers, I had some troubles with the allocator, it's not something you want to use every time, I'd say, it's not intuitive to pass the function to the C wrapped function, calling it to get a pointer to a new array and not returning it because it was saved in the allocator... Now, I think I can say that the ported Python function does not function as well as the wrapped function, I must have made an error somewhere, but now I don't have to care about it, it works well.
Many thanks to David and Bill !
Matthieu
-
Bill Baxter -
David Cournapeau -
David Cournapeau -
Matthieu Brucher