Custom smart pointer with same behaviour as shared_ptr
Hi, I am passing Python objects to C++ through a shared_ptr smart pointer and it's work well. However I need to have the same behaviour as shared_ptr but with my own version of smart pointer. I try to register and expose my custom smart pointer but I have this compilation error: boost/python/pointee.hpp(28) : error C2039: 'element_type': is not a member of 'CountedObjPtr<T>' with [ T=PyCallEngineState ] boost/python/pointee.hpp(38) : see reference to class template instantiation 'boost::python::detail::pointee_impl<false>::apply<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] boost/python/register_ptr_to_python.hpp(17) : see reference to class template instantiation 'boost::python::pointee<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] PythonModules.cpp(113) : see reference to function template instantiation 'void boost::python::register_ptr_to_python<CountedObjPtr<T>>(void)'being compiled with [ T=PyCallEngineState ] Here is the content of the from_python file for my smart pointer: template <class T> struct CountedObjPtr_from_python { CountedObjPtr_from_python() { converter::registry::insert(&convertible, &construct, type_id<CountedObjPtr<T> >()); } static void* convertible(PyObject* p) { if (p == Py_None) { return p; } return converter::get_lvalue_from_python( p, python::converter::registered<T>::converters); } static void construct(PyObject* source, python::converter::rvalue_from_python_stage1_data* data) { void* const storage = ((python::converter::rvalue_from_python_storage<CountedObjPtr<T>
*)data)->storage.bytes; // Deal with the "None" case. if (data->convertible == source) { new (storage) CountedObjPtr<T>(); } else { new (storage) CountedObjPtr<T>(static_cast<T*>(data->convertible)); }
data->convertible = storage; } }; And here is the code where I register the smart pointer: namespace boost{ namespace python{ template <class T> struct pointee< CountedObjPtr<const T> > { typedef T type; }; template <class T> inline T* get_pointer(CountedObjPtr<const T> const& p) { return const_cast<T*>(p.get()); } }} template <typename T> void register_CountedObjPtr_conversions() { boost::python::converter::registry::insert( &CountedObjPtr_from_python<T>::convertible, &CountedObjPtr_from_python<T>::construct, boost::python::type_id<CountedObjPtr<T> >()); } BOOST_PYTHON_MODULE(CallEngineExtending) { python::class_<PyCallEngineState, PyCallEngineState_Wrapped, boost::noncopyable>("PyCallEngineState") ; //register_ptr_to_python<boost::shared_ptr<PyCallEngineState> >(); python::register_ptr_to_python<CountedObjPtr<PyCallEngineState> >(); register_CountedObjPtr_conversions<PyCallEngineState>(); } Is anyone knows what is missing or wrong with this code? Also, I am not sure this code manages the ref counting properly since there is no call to borrowed or something like this? Any help would be appreciated, Francis
Ok I found my compilation problem. "Francis Moreau" <francis_moreau@hotmail.com> wrote in message news:eurnv7$5cg$1@sea.gmane.org...
Hi,
I am passing Python objects to C++ through a shared_ptr smart pointer and it's work well. However I need to have the same behaviour as shared_ptr but with my own version of smart pointer. I try to register and expose my custom smart pointer but I have this compilation error:
boost/python/pointee.hpp(28) : error C2039: 'element_type': is not a member of 'CountedObjPtr<T>' with [ T=PyCallEngineState ] boost/python/pointee.hpp(38) : see reference to class template instantiation 'boost::python::detail::pointee_impl<false>::apply<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] boost/python/register_ptr_to_python.hpp(17) : see reference to class template instantiation 'boost::python::pointee<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] PythonModules.cpp(113) : see reference to function template instantiation 'void boost::python::register_ptr_to_python<CountedObjPtr<T>>(void)'being compiled with [ T=PyCallEngineState ]
Here is the content of the from_python file for my smart pointer:
template <class T> struct CountedObjPtr_from_python { CountedObjPtr_from_python() { converter::registry::insert(&convertible, &construct, type_id<CountedObjPtr<T> >()); }
static void* convertible(PyObject* p) { if (p == Py_None) { return p; }
return converter::get_lvalue_from_python( p, python::converter::registered<T>::converters); }
static void construct(PyObject* source,
python::converter::rvalue_from_python_stage1_data* data) { void* const storage = ((python::converter::rvalue_from_python_storage<CountedObjPtr<T>
*)data)->storage.bytes; // Deal with the "None" case. if (data->convertible == source) { new (storage) CountedObjPtr<T>(); } else { new (storage) CountedObjPtr<T>(static_cast<T*>(data->convertible)); }
data->convertible = storage; } };
And here is the code where I register the smart pointer:
namespace boost{ namespace python{
template <class T> struct pointee< CountedObjPtr<const T> > { typedef T type; };
template <class T> inline T* get_pointer(CountedObjPtr<const T> const& p) { return const_cast<T*>(p.get()); }
}}
template <typename T> void register_CountedObjPtr_conversions() { boost::python::converter::registry::insert( &CountedObjPtr_from_python<T>::convertible, &CountedObjPtr_from_python<T>::construct, boost::python::type_id<CountedObjPtr<T> >()); }
BOOST_PYTHON_MODULE(CallEngineExtending) { python::class_<PyCallEngineState, PyCallEngineState_Wrapped, boost::noncopyable>("PyCallEngineState") ;
//register_ptr_to_python<boost::shared_ptr<PyCallEngineState> >();
python::register_ptr_to_python<CountedObjPtr<PyCallEngineState> >(); register_CountedObjPtr_conversions<PyCallEngineState>(); }
Is anyone knows what is missing or wrong with this code? Also, I am not sure this code manages the ref counting properly since there is no call to borrowed or something like this?
Any help would be appreciated, Francis
on Mon Apr 02 2007, "Francis Moreau" <francis_moreau-AT-hotmail.com> wrote:
Hi,
I am passing Python objects to C++ through a shared_ptr smart pointer and it's work well. However I need to have the same behaviour as shared_ptr but with my own version of smart pointer. I try to register and expose my custom smart pointer but I have this compilation error:
boost/python/pointee.hpp(28) : error C2039: 'element_type': is not a member of 'CountedObjPtr<T>' with [ T=PyCallEngineState ] boost/python/pointee.hpp(38) : see reference to class template instantiation 'boost::python::detail::pointee_impl<false>::apply<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] boost/python/register_ptr_to_python.hpp(17) : see reference to class template instantiation 'boost::python::pointee<T>' being compiled with [ T=CountedObjPtr<PyCallEngineState> ] PythonModules.cpp(113) : see reference to function template instantiation 'void boost::python::register_ptr_to_python<CountedObjPtr<T>>(void)'being compiled with [ T=PyCallEngineState ]
Please read http://www.boost.org/libs/python/doc/v2/pointee.html However, be aware that you won't get exactly the "magic" shared_ptr behavior described in http://www.boost.org/libs/python/doc/v2/faq.html#xref unless your pointer has a deleter just as shared_ptr does AND you do some extra work to define special converters for it (and perhaps not even then; I haven't looked into the details). -- Dave Abrahams Boost Consulting www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com
David Abrahams wrote:
on Mon Apr 02 2007, "Francis Moreau" <francis_moreau-AT-hotmail.com> wrote:
Please read http://www.boost.org/libs/python/doc/v2/pointee.html
However, be aware that you won't get exactly the "magic" shared_ptr behavior described in http://www.boost.org/libs/python/doc/v2/faq.html#xref unless your pointer has a deleter just as shared_ptr does AND you do some extra work to define special converters for it (and perhaps not even then; I haven't looked into the details).
Could you please explain a bit more what you mean by 1) deleter 2) which converters 3) what more is needed I've tried to copy boost::shared_ptr and rename it, to see whether the "magic" was coming from boost::share_ptr, but it looks like the "magic" is inside Boost.Python. I've tried also tr1::shared_ptr and they all fail where boost::shared_ptr succeeds. My library is just too huge to change all smart pointers to boost::shared_ptr... Cheers
I've tried to copy boost::shared_ptr and rename it, to see whether the "magic" was coming from boost::share_ptr, but it looks like the "magic" is inside Boost.Python. I've tried also tr1::shared_ptr and they all fail where boost::shared_ptr succeeds.
My library is just too huge to change all smart pointers to boost::shared_ptr...
we had similar problems when wrapping a library where we used boost::intrusive_ptr as our smart pointer of choice. we managed to work around most of the problems using the magic incantations in this file : http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/Intrusive... you can see an example of how we use that in this file : http://cortex-vfx.googlecode.com/svn/trunk/src/IECore/bindings/ObjectBinding... hopefully those might give you some clue as to how to fix your problems too... cheers... john _________________________________________________________________
john haddon wrote:
we had similar problems when wrapping a library where we used boost::intrusive_ptr as our smart pointer of choice. we managed to work around most of the problems using the magic incantations in this file :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/Intrusive...
you can see an example of how we use that in this file :
http://cortex-vfx.googlecode.com/svn/trunk/src/IECore/bindings/ObjectBinding...
hopefully those might give you some clue as to how to fix your problems too...
cheers... john
Thanks for the suggestion. I end up debugging until I found a place where the *magic* boost::shared_ptr was created (out of the blue) and I ended up in the file <boost/python/converter/shared_ptr_from_python.hpp> It is the same file you sent me, and I've seen what you did to change it. So I am going to try it out. Could you please tell me what you meant by
"we managed to work around most of the problems"
since it looks like I will be following the same steps as you. Maybe this whole approach should be documented (if it is not too dodgy) Andrea
we had similar problems when wrapping a library where we used boost::intrusive_ptr as our smart pointer of choice. we managed to work around most of the problems using the magic incantations in this file :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/Intrusive...
Could you please tell me what you meant by
"we managed to work around most of the problems"
since it looks like I will be following the same steps as you.
Our main problem was that when returning an intrusive_ptr<Base> from C++ to python, it would create a Python object of type Base, when what we really wanted to do was create a Python object of type Derived, by downcasting to the most derived type. The code I posted deals with that successfully (and I think one other issue I don't recall). We still have the problem that passing the same object to python twice results in two different objects : # x always returns an intrusive pointer to the same c++ object a = x() b = x() # assertion fails - two python objects refer to one c++ object and don't know it assert( a is b ) We are working around this problem in two ways. Firstly we have an ugly isSame() method bound for object to be used instead of "is". Secondly we have another to_python converter for use when we wrap classes. This solves the object identity problem by keeping a mapping from C++ objects to python objects. You can see that here : http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo... I suspect the two solutions could be combined into one general solution, but currently it's working just well enough so we've left it there... Hope that helps... Cheers... John _________________________________________________________________
john haddon wrote:
It works... at least in my small proof of concept...
Our main problem was that when returning an intrusive_ptr<Base> from C++ to python, it would create a Python object of type Base, when what we really wanted to do was create a Python object of type Derived, by downcasting to the most derived type. The code I posted deals with that successfully (and I think one other issue I don't recall). We still have the problem that passing the same object to python twice results in two different objects :
I am not sure I follow here. My hierarchy is base class A B : A C : B then I call a function returning an intrusive_ptr<A>(new C()) into c
type(c) <class 'bo.C'>
So Python can tell that the object is of type C even if it was returned as a pointer to the base class A. Other point: please correct me if I am wrong. I need to register every class I want to be able to cast to (i.e. every class I want to accept as argument). So in that case I need to add both INTRUSIVE_PTR_PATCH(B, B_class); and INTRUSIVE_PTR_PATCH(C, C_class); so that my functions can take intrusive_ptr of type B and C. No need to do it with A.
We are working around this problem in two ways. Firstly we have an ugly isSame() method bound for object to be used instead of "is". Secondly we have another to_python converter for use when we wrap classes. This solves the object identity problem by keeping a mapping from C++ objects to python objects. You can see that here :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo...
I suspect the two solutions could be combined into one general solution, but currently it's working just well enough so we've left it there...
Here, first I need to understand the problem, then I pass to your solution. Is it a general problem, or something relative to intrusive_ptr, that would not be there, had you used boost::shared_ptr? Thanks very much. Andrea
I am not sure I follow here.
My hierarchy is
base class A B : A C : B
then I call a function returning an intrusive_ptr(new C()) into c
type(c)
So Python can tell that the object is of type C even if it was returned as a pointer to the base class A.
Other point: please correct me if I am wrong. I need to register every class I want to be able to cast to (i.e. every class I want to accept as argument). So in that case I need to add both
INTRUSIVE_PTR_PATCH(B, B_class); and INTRUSIVE_PTR_PATCH(C, C_class);
so that my functions can take intrusive_ptr of type B and C. No need to do it with A.
correct - that should be all you need to solve the downcasting problem. everything else i mentioned relates to other problems you might see later - i've expanded on those a bit below.
We are working around this problem in two ways. Firstly we have an ugly isSame() method bound for object to be used instead of "is". Secondly we have another to_python converter for use when we wrap classes. This solves the object identity problem by keeping a mapping from C++ objects to python objects. You can see that here :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo...
I suspect the two solutions could be combined into one general solution, but currently it's working just well enough so we've left it there...
Here, first I need to understand the problem, then I pass to your solution. Is it a general problem, or something relative to intrusive_ptr, that would not be there, had you used boost::shared_ptr?
the problem is this - say you bind this function : intrusive_ptr x( intrusive_ptr b ) { return b; } then this python code fails : b = B() b2 = x( b ) assert( b is b2 ) # fails although both b and b2 correctly refer to the same c++ object, they refer to two different python instances - what x should really do is return the same PyObject that was being used to hold b in the first place. i believe there's some magic in boost::python that solves this problem as long as you're using shared_ptr (although you might want to test that), but the patch i posted for the downcasting of intrusive_ptr doesn't address it... we've addressed the issue only for classes we wrap during binding - as we can easily store a mapping between the c++ pointer and the PyObject * and use that in our to_python function...this is what you see in this file : http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo... that file also refers to the WrapperGarbageCollector class we use in the same project. we found there were memory management issues using the boost::wrapper mechanism - i don't recall the specifics but it was either that wrapped objects would never be garbage collected, or that the python half of the wrapped object could die before the c++ half. to tie the lifetimes of the two halves of the objects together they each must have a reference to the other, but that results in a circular reference and hence no collection - WrapperGarbageCollector periodically trawls through the wrapped instances and breaks any such cycles when necessary. depending on whether or not you use wrapping you may want to incorporate something like that too... cheers... john _________________________________________________________________
Other point: please correct me if I am wrong. I need to register every class I want to be able to cast to (i.e. every class I want to accept as argument). So in that case I need to add both
INTRUSIVE_PTR_PATCH(B, B_class); and INTRUSIVE_PTR_PATCH(C, C_class);
so that my functions can take intrusive_ptr of type B and C. No need to do it with A.
Taking a look at our bindings again, I think it would be wise to do the same for A too. Sorry if I'm a little vague on this subject - it was a while ago and I didn't actually write all the code... _________________________________________________________________
john haddon wrote:
Other point: please correct me if I am wrong. I need to register every class I want to be able to cast to (i.e. every class I want to accept as argument). So in that case I need to add both
INTRUSIVE_PTR_PATCH(B, B_class); and INTRUSIVE_PTR_PATCH(C, C_class);
so that my functions can take intrusive_ptr of type B and C. No need to do it with A.
Taking a look at our bindings again, I think it would be wise to do the same for A too. Sorry if I'm a little vague on this subject - it was a while ago and I didn't actually write all the code...
Yes, you are right. It is needed when I have a pointer to a derived class and I need to pass is as base class. It seems that I have to pair INTRUSIVE_PTR_PATCH with each class_ declaration. It would be nice if it could be added to class_ so it is done automatically. Thanks
john haddon wrote:
the problem is this - say you bind this function :
intrusive_ptr x( intrusive_ptr b ) { return b; }
then this python code fails :
b = B() b2 = x( b ) assert( b is b2 ) # fails
although both b and b2 correctly refer to the same c++ object, they refer to two different python instances - what x should really do is return the same PyObject that was being used to hold b in the first place. i believe there's some magic in boost::python that solves this problem as long as you're using shared_ptr (although you might want to test that), but the patch i posted for the downcasting of intrusive_ptr doesn't address it...
Yes, boost::shared_ptr<> achieve this
b=bo.newB() c=bo.sameB(b) b <bo.B object at 0xb7e9102c> c <bo.B object at 0xb7e9102c> b is c True
but any other pointer fails. It looks that there is some special handling of boost::shared_ptr<> when they are returned to python. Has anybody found where this happens. Maybe the fix is as easy allowing the downcasting of custom pointers... I've tried to follow the function calls, but I got lost after a few layer of templates... Andrea
we had similar problems when wrapping a library where we used boost::intrusive_ptr as our smart pointer of choice. we managed to work around most of the problems using the magic incantations in this file :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/Intrusive...
Could you please tell me what you meant by
"we managed to work around most of the problems"
since it looks like I will be following the same steps as you.
Our main problem was that when returning an intrusive_ptr<Base> from C++ to python, it would create a Python object of type Base, when what we really wanted to do was create a Python object of type Derived, by downcasting to the most derived type. The code I posted deals with that successfully (and I think one other issue I don't recall). We still have the problem that passing the same object to python twice results in two different objects : # x always returns an intrusive pointer to the same c++ object a = x() b = x() # assertion fails - two python objects refer to one c++ object and don't know it assert( a is b ) We are working around this problem in two ways. Firstly we have an ugly isSame() method bound for object to be used instead of "is". Secondly we have another to_python converter for use when we wrap classes. This solves the object identity problem by keeping a mapping from C++ objects to python objects. You can see that here : http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo... I suspect the two solutions could be combined into one general solution, but currently it's working just well enough so we've left it there... Hope that helps... Cheers... John _________________________________________________________________
we had similar problems when wrapping a library where we used boost::intrusive_ptr as our smart pointer of choice. we managed to work around most of the problems using the magic incantations in this file :
http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/Intrusive...
Could you please tell me what you meant by
"we managed to work around most of the problems"
since it looks like I will be following the same steps as you.
Our main problem was that when returning an intrusive_ptr<Base> from C++ to python, it would create a Python object of type Base, when what we really wanted to do was create a Python object of type Derived, by downcasting to the most derived type. The code I posted deals with that successfully (and I think one other issue I don't recall). We still have the problem that passing the same object to python twice results in two different objects : # x always returns an intrusive pointer to the same c++ object a = x() b = x() # assertion fails - two python objects refer to one c++ object and don't know it assert( a is b ) We are working around this problem in two ways. Firstly we have an ugly isSame() method bound for object to be used instead of "is". Secondly we have another to_python converter for use when we wrap classes. This solves the object identity problem by keeping a mapping from C++ objects to python objects. You can see that here : http://cortex-vfx.googlecode.com/svn/trunk/include/IECore/bindings/WrapperTo... I suspect the two solutions could be combined into one general solution, but currently it's working just well enough so we've left it there... Hope that helps... Cheers... John _________________________________________________________________
participants (4)
-
Andrea -
David Abrahams -
Francis Moreau -
john haddon