Moving "decorator" into C++ part of module
Hello all I have a class with various methods that return uBLAS matrices. I've wrapped various instantiations of these matrices with: template<typename T> struct matrix_wrapper { static void wrap(const char* python_name) { py::class_<T, boost::noncopyable>(python_name, py::no_init) .add_property("__array_struct__", array_struct__) ; } static void cleanup(void* obj) { PyArrayInterface* ai = (PyArrayInterface*) obj; delete [] ai->shape; delete [] ai->strides; delete ai; } static PyObject* array_struct__(T const& self) { // http://numpy.scipy.org/array_interface.shtml PyArrayInterface* ai = new PyArrayInterface; // ... more code here ... ai->data = (void*) self.data().begin(); return PyCObject_FromVoidPtr(ai, cleanup); } } In Python this allows me to do: foo = WrappedClass() cmatrix = foo.somefunc() numpyarr = numpy.asarray(cmatrix) without a copy of cmatrix's contents having to be made. Now I'd just like to get rid of the asarray bit so that it looks like my wrapped functions return NumPy arrays. In Python I can "decorate" this function like so: def return_asarray(func): from numpy import asarray def call(*args, **kwargs): return asarray(func(*args, **kwargs)) return call WrappedClass.somefunc = return_asarray(WrappedClass.somefunc) but I would like to move this "decoration" into the C++ code, so that my module doesn't need to have the extra Python bits. Any thoughts on how this could be done? I looked at the documentation for class_, but nothing jumped out at me. Regards, Albert
On 5/21/07, Albert Strasheim <fullung@gmail.com> wrote:
Hello all ... without a copy of cmatrix's contents having to be made. Now I'd just like to get rid of the asarray bit so that it looks like my wrapped functions return NumPy arrays. In Python I can "decorate" this function like so:
def return_asarray(func): from numpy import asarray def call(*args, **kwargs): return asarray(func(*args, **kwargs)) return call WrappedClass.somefunc = return_asarray(WrappedClass.somefunc)
but I would like to move this "decoration" into the C++ code, so that my module doesn't need to have the extra Python bits.
Any thoughts on how this could be done? I looked at the documentation for class_, but nothing jumped out at me.
Even call policies? If I understand right, you can create new call policy, implement postcall only member function and that's all. Am I missing something? -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/
Albert Strasheim wrote:
Hello all
I have a class with various methods that return uBLAS matrices. I've wrapped various instantiations of these matrices with:
[...] I'm very interested in this. I thought array_interface was only a proposal at this point. You are using it? If I have python2.5, do I need something to try this out? (Like, where is the array_interface include file?) Could you post a complete code that shows how you convert ublas::matrix to numpy? Do you have converters in the other direction?
Howdy On Mon, 21 May 2007, Neal Becker wrote:
Albert Strasheim wrote:
Hello all
I have a class with various methods that return uBLAS matrices. I've wrapped various instantiations of these matrices with:
[...]
I'm very interested in this.
I thought array_interface was only a proposal at this point. You are using it? If I have python2.5, do I need something to try this out? (Like, where is the array_interface include file?)
NumPy's array interface is ready to go. I think you might be thinking about the enhanced buffer protocol PEP thing.
Could you post a complete code that shows how you convert ublas::matrix to numpy? Do you have converters in the other direction?
Code attached. This is still very much a work in progress. Test it with: import numpy as N import os from numpy.testing import * # directory where you compiled the module set_local_path(os.path.join('..', 'win_build', 'Debug')) import pyublas restore_path() at = pyublas.array_test() print at print at.ref_return() print at.const_ref_return() print at.pointer_return() print at.const_pointer_return() As you can see, I can now convert return types (just haven't implemented return by-value yet). Now that I understand call policies properly, I'm going to look at converting arguments. Cheers, Albert
This looks very good so far. I see there is c++ ublas -> python numpy conversion. Have you tried the other direction? python numpy -> c++ ublas?
Hello On Mon, 21 May 2007, Neal Becker wrote:
This looks very good so far. I see there is c++ ublas -> python numpy conversion.
With a few more templates and whatnot one can take care of things figuring out that for a matrix<int> the typecode should be 'i', etc. For now I'm just focusing on getting double working. But soon...
Have you tried the other direction? python numpy -> c++ ublas?
I'm trying to sort this out now. Here there are a few more complications: 1. Some array adaptor complications (nothing too major) to make a uBLAS matrix use the array's data without a copy 2. Conversion of NumPy array arguments to uBLAS I'm currently stuck here. I was hoping to do something like: template < class T, std::size_t arg, class BasePolicy_ = py::default_call_policies
struct convert_array : BasePolicy_ { BOOST_STATIC_ASSERT(arg > 0); template <class ArgumentPackage> static bool precall(ArgumentPackage const& args_) { unsigned int arity_ = PyTuple_GET_SIZE(args_); if (arg > arity_) { PyErr_SetString( PyExc_IndexError, "convert_array: argument index out of range"); return false; } PyObject* obj = PyTuple_GetItem(args_, arg); if (obj == NULL) { return false; } if (!PyArray_Check(obj)) { PyErr_SetString( PyExc_TypeError, "convert_array: ndarray argument expected"); return NULL; } // XXX magic happens here see below return BasePolicy_::precall(args_); } }; What I wanted to attempt at XXX is to create an instance of a uBLAS vector/matrix/whatever inside its associated PyObject, with the uBLAS matrix using the memory of the NumPy array. I was hoping I could then do a little bait and switch on the tuple item to put this new uBLAS-PyObject-using-a-Numpy-array into the arguments before the function gets called. You could then wrap the method something like this: py::class_<array_test, boost::noncopyable>("array_test") .def("pointer_arg", &array_test::pointer_arg, convert_array<matrix<float>, 1>()) Unfortunately, it seems checking of argument types happens before the the convert_array precall (not much of a *pre*call is it? ;-)), so this trick doesn't work (you get an ArgumentError). Any ideas would help at this point. :-) Regards, Albert
Albert Strasheim wrote:
Hello
On Mon, 21 May 2007, Neal Becker wrote:
This looks very good so far. I see there is c++ ublas -> python numpy conversion.
With a few more templates and whatnot one can take care of things figuring out that for a matrix<int> the typecode should be 'i', etc. For now I'm just focusing on getting double working. But soon...
Have you tried the other direction? python numpy -> c++ ublas?
I'm trying to sort this out now. Here there are a few more complications:
1. Some array adaptor complications (nothing too major) to make a uBLAS matrix use the array's data without a copy
2. Conversion of NumPy array arguments to uBLAS
I'm currently stuck here. I was hoping to do something like:
template < class T, std::size_t arg, class BasePolicy_ = py::default_call_policies
struct convert_array : BasePolicy_ { BOOST_STATIC_ASSERT(arg > 0);
template <class ArgumentPackage> static bool precall(ArgumentPackage const& args_) { unsigned int arity_ = PyTuple_GET_SIZE(args_); if (arg > arity_) { PyErr_SetString( PyExc_IndexError, "convert_array: argument index out of range"); return false; } PyObject* obj = PyTuple_GetItem(args_, arg); if (obj == NULL) { return false; } if (!PyArray_Check(obj)) { PyErr_SetString( PyExc_TypeError, "convert_array: ndarray argument expected"); return NULL; }
// XXX magic happens here see below
return BasePolicy_::precall(args_); } };
What I wanted to attempt at XXX is to create an instance of a uBLAS vector/matrix/whatever inside its associated PyObject, with the uBLAS matrix using the memory of the NumPy array. I was hoping I could then do a little bait and switch on the tuple item to put this new uBLAS-PyObject-using-a-Numpy-array into the arguments before the function gets called.
You could then wrap the method something like this:
py::class_<array_test, boost::noncopyable>("array_test") .def("pointer_arg", &array_test::pointer_arg, convert_array<matrix<float>, 1>())
Unfortunately, it seems checking of argument types happens before the the convert_array precall (not much of a *pre*call is it? ;-)), so this trick doesn't work (you get an ArgumentError).
Any ideas would help at this point. :-)
Mostly I'd be interested in the following. My use case is I have lots of c++ algorithms written for ublas interface (or maybe a more generic superset of that). I want to use them from python. I want to call them with numpy arrays. Someone has to convert the numpy array to a ublas interface, then call the c++ algorithm, and maybe convert the ublas object back to numpy to python.
I have a class that is similar to a smart_ptr, called RefCountPtr and I have a c++ class constructor X::X( RefCountPtr< foo >& f ) : thisfoo(f) ; When I pass a Python 'foo' object into this constructor, I want to wrap it in a c++ RefCountPtr class before passing it to the c++ constructor. I have written an from_python extration method to do this static void* extract_rcp(PyObject* o) { object boostobj = object(handle<>(borrowed( o )) ); foo *counted = extract<foo*>( boostobj ); RefCountPtr< foo > *counter = new RefCountPtr< foo > ( counted , true ); return counter; } When I extract the 'foo' object, I create a new RefCountPtr to pass to the constructor, once the constructor finishes, I am left with a reference count of 2: One for the new'd object and one for the data member of the X class. My question is, how can I get rid of the extra reference count? Is there a call policy that will do this for me? ~Sean
On 6/1/07, Sean Ross-Ross <srossross@gmail.com> wrote:
I have a class that is similar to a smart_ptr, called RefCountPtr and I have a c++ class constructor
X::X( RefCountPtr< foo >& f ) : thisfoo(f) ;
When I pass a Python 'foo' object into this constructor, I want to wrap it in a c++ RefCountPtr class before passing it to the c++ constructor.
What if the instance of foo object is already managed by other RefCountPtr?
I have written an from_python extration method to do this
static void* extract_rcp(PyObject* o) { object boostobj = object(handle<>(borrowed( o )) ); foo *counted = extract<foo*>( boostobj ); RefCountPtr< foo > *counter = new RefCountPtr< foo > ( counted , true ); return counter; }
When I extract the 'foo' object, I create a new RefCountPtr to pass to the constructor, once the constructor finishes, I am left with a reference count of 2: One for the new'd object and one for the data member of the X class.
My question is, how can I get rid of the extra reference count? Is there a call policy that will do this for me?
I think the whole approach you are taking is erroneous. It is much better to let Boost.Python to manage such things. Take a look on "custom smart pointers" guide I wrote few month ago http://www.language-binding.net/pyplusplus/troubleshooting_guide/smart_ptrs/... -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/
Hi, Sorry to unearth this thread, but I'd like convert numpy array as well, but not only for matrices (my lab uses multi-dimensional images in C++ and it could be great to have a non-copying wrapper to Python and from Python). I'm very new to Boost.Python and to the numpy C-API, I hope I'll not ask stupid questions. In your code, Albert, you wrap a uBlas matrix in array_struct__. Is there some kind of reference counting for the memory allocated in the matrix ? Or the matrix must be valid until the python variable is destroyed ? If this is the case, is there a "simple" way of using smart pointers (if the class that must be wrapped uses smart pointers for holding the data) ? Matthieu 2007/5/21, Albert Strasheim <fullung@gmail.com>:
Howdy
On Mon, 21 May 2007, Neal Becker wrote:
Albert Strasheim wrote:
Hello all
I have a class with various methods that return uBLAS matrices. I've wrapped various instantiations of these matrices with:
[...]
I'm very interested in this.
I thought array_interface was only a proposal at this point. You are using it? If I have python2.5, do I need something to try this out? (Like, where is the array_interface include file?)
NumPy's array interface is ready to go. I think you might be thinking about the enhanced buffer protocol PEP thing.
Could you post a complete code that shows how you convert ublas::matrix to numpy? Do you have converters in the other direction?
Code attached. This is still very much a work in progress. Test it with:
import numpy as N import os
from numpy.testing import * # directory where you compiled the module set_local_path(os.path.join('..', 'win_build', 'Debug')) import pyublas restore_path()
at = pyublas.array_test() print at print at.ref_return() print at.const_ref_return() print at.pointer_return() print at.const_pointer_return()
As you can see, I can now convert return types (just haven't implemented return by-value yet). Now that I understand call policies properly, I'm going to look at converting arguments.
Cheers,
Albert
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
Hello On Thu, 12 Jul 2007, Matthieu Brucher wrote:
Sorry to unearth this thread, but I'd like convert numpy array as well, but not only for matrices (my lab uses multi-dimensional images in C++ and it could be great to have a non-copying wrapper to Python and from Python). I'm very new to Boost.Python and to the numpy C-API, I hope I'll not ask stupid questions.
In your code, Albert, you wrap a uBlas matrix in array_struct__. Is there some kind of reference counting for the memory allocated in the matrix ? Or the matrix must be valid until the python variable is destroyed ? If this is the case, is there a "simple" way of using smart pointers (if the class that must be wrapped uses smart pointers for holding the data) ?
The allocation and deallocation of the array is still managed by Boost.Python. You control the management through using the existing Boost.Python call policies. I put it together as follows: 1. The type you want to wrap should expose an __array_struct__. See, for example, my ublas_matrix wrapper: http://pyspkrec.googlecode.com/svn/trunk/numpycpp/ublas_matrix.h and you might want to look at the documentation for the array interface: http://numpy.scipy.org/array_interface.shtml For your multidimensional image class you want want to be more fancy and put something useful in descr. I think you could use this to make your image object behave like a NumPy record array, for example. 2. Functions returning your type are wrapped with return_asarray combined with any Boost.Python call policy, like here: http://pyspkrec.googlecode.com/svn/trunk/numpycpp/src/ublas_matrix_test.cpp so you do something like: def("foo", &foo, return_asarray<py::return_value_policy<py::manage_new_object> >()); where py == boost::python. 3. return_asarray is defined here: http://pyspkrec.googlecode.com/svn/trunk/numpycpp/numpycpp.h To understand how it works, you should study the CallPolicies concept: http://www.boost.org/libs/python/doc/v2/CallPolicies.html#CallPolicies-conce... What I'm doing is apparently called CallPolicies composition. The line: result = BasePolicy_::postcall(args_, result); gets the PyObject* of the wrapped object which might in turn already be wrapped in some Boost.Python thingy (like a custodian with ward). This object exposes __array_struct__, which is read when we pass it on to NumPy C/API function: return PyArray_FromStructInterface(result); Take a look at that function's source to get a better idea of what it does. This function returns a NumPy array that owns a reference (if that's the right terminology?) to the wrapped object. My tests seem to indicate that this all does what I think it does, but I could be wrong. ;-) Good luck with your wrapping and feel free to contact me if you have any more issues or questions. Cheers, Albert
Hi again, A big thank you for all the indications, I'll read them and try to understand them. Matthieu 2007/7/12, Albert Strasheim <fullung@gmail.com>:
Hello
On Thu, 12 Jul 2007, Matthieu Brucher wrote:
Sorry to unearth this thread, but I'd like convert numpy array as well, but not only for matrices (my lab uses multi-dimensional images in C++ and it could be great to have a non-copying wrapper to Python and from Python). I'm very new to Boost.Python and to the numpy C-API, I hope I'll not ask stupid questions.
In your code, Albert, you wrap a uBlas matrix in array_struct__. Is there some kind of reference counting for the memory allocated in the matrix ? Or the matrix must be valid until the python variable is destroyed ? If this is the case, is there a "simple" way of using smart pointers (if the class that must be wrapped uses smart pointers for holding the data) ?
The allocation and deallocation of the array is still managed by Boost.Python. You control the management through using the existing Boost.Python call policies.
I put it together as follows:
1. The type you want to wrap should expose an __array_struct__. See, for example, my ublas_matrix wrapper:
http://pyspkrec.googlecode.com/svn/trunk/numpycpp/ublas_matrix.h
and you might want to look at the documentation for the array interface:
http://numpy.scipy.org/array_interface.shtml
For your multidimensional image class you want want to be more fancy and put something useful in descr. I think you could use this to make your image object behave like a NumPy record array, for example.
2. Functions returning your type are wrapped with return_asarray combined with any Boost.Python call policy, like here:
http://pyspkrec.googlecode.com/svn/trunk/numpycpp/src/ublas_matrix_test.cpp
so you do something like:
def("foo", &foo, return_asarray<py::return_value_policy<py::manage_new_object> >());
where py == boost::python.
3. return_asarray is defined here:
http://pyspkrec.googlecode.com/svn/trunk/numpycpp/numpycpp.h
To understand how it works, you should study the CallPolicies concept:
http://www.boost.org/libs/python/doc/v2/CallPolicies.html#CallPolicies-conce...
What I'm doing is apparently called CallPolicies composition.
The line:
result = BasePolicy_::postcall(args_, result);
gets the PyObject* of the wrapped object which might in turn already be wrapped in some Boost.Python thingy (like a custodian with ward). This object exposes __array_struct__, which is read when we pass it on to NumPy C/API function:
return PyArray_FromStructInterface(result);
Take a look at that function's source to get a better idea of what it does. This function returns a NumPy array that owns a reference (if that's the right terminology?) to the wrapped object.
My tests seem to indicate that this all does what I think it does, but I could be wrong. ;-)
Good luck with your wrapping and feel free to contact me if you have any more issues or questions.
Cheers,
Albert _______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
participants (5)
-
Albert Strasheim -
Matthieu Brucher -
Neal Becker -
Roman Yakovenko -
Sean Ross-Ross