[C++-sig] using intrusive_ptr objects with boost python

[Jeff Webb]

Let me first say that I am extremely impressed with boost python's elegant and powerful design.  I would not have thought that it was possible to integrate C++ and python code in such a nice manner.  Thanks for all the hard work and thoughtful design.

I am writing a C++ library from scratch, and I want to develop boost python bindings for it simultaneously.  My C++ library will instantiate many small objects that hold references to other objects.  I had come up a consistent scheme for managing the objects using C++, but when I started trying to wrap the library using boost, I found that this scheme didn't mesh well with python's reference counting form of memory management.  After quite a bit of reading, experimentation, and contemplation, it appears to me that using reference-counting smart pointers in C++ would be the most consistent and seamless way to provide a rich python binding for this type of library.  Do others agree with this assertion?

I wrote a simple test library using shared_ptr objects and found that wrapping it was very natural.  Things 'just worked' as I would have hoped.  The source code for this library is attached to this email as 's_ptr_lib.cpp'.  A sample ipython session is shown below.

 In [1]: from s_ptr_lib import *
 In [2]: c1 = C('c1'); c2 = C('c2')
 C(c1)
 C(c2)
 In [3]: c1.set_ptr(c2)
 In [4]: ptr = c1.get_ptr()
 In [5]: ptr is c2
 Out[5]: True  
 In [6]: del(c2)
 In [7]: del(ptr)
 In [8]: del(c1)
 ~C(c1)
 ~C(c2)

Very nice!  Unfortunately, I think the overhead of using a shared_ptr will be too high, since my library will instantiate many small objects.  I then discovered the intrusive_ptr, and I think it would be a better choice.  I then converted my library to use intrusive_ptrs instead of shared_ptrs.  The source code for this library is attached as 'i_ptr_lib.cpp'.  When I tried to use this library from python, I found that the intrusive_ptr was not as well supported as the shared_ptr.  A sample ipython session is shown below:

 In [1]: from i_ptr_lib import *
 In [2]: c1 = C('c1'); c2 = C('c2')
 C(c1)
 C(c2)
 In [3]: c1.set_ptr(c2)
 -------------------------------
 Boost.Python.ArgumentError                             
 Traceback (most recent call last)
 ArgumentError: Python argument types in
     C.set_ptr(C, C)
 did not match C++ signature:
     set_ptr(C {lvalue}, boost::intrusive_ptr<C>)

It seems that I will need to supply all the necessary type conversions for the intrusive_ptr.  I was hoping to not have to dig that deep into boost python's template magic at this point...  Is there any good documentation on how to do this?  I found the following post that seemed promising:

 http://mail.python.org/pipermail/c++-sig/2007-February/011961.html

I downloaded the 'register_intrusive_ptr_from_python.hpp' and 'intrusive_ptr_from_python.hpp' file attachments from the above link and then modified my i_ptr_lib.cpp program by adding '#include register_intrusive_ptr_from_python.hpp' near the top, and this line after the end of the class_<C> declaration:

 boostPatch::register_intrusive_ptr_from_python_and_casts( (C *)0, class_<C>::metadata::bases() );

I then tested the library again:

 In [1]: from i2_ptr_lib import *
 In [2]: c1 = C('c1'); c2 = C('c2')
 C(c1)
 C(c2)
 In [3]: c1.set_ptr(c2)
 In [4]: del(c2)
 ~C(c2)
 In [5]: c1.get_ptr()
 ------------------------
 exceptions.TypeError      
 Traceback (most recent call last)
 TypeError: No to_python (by-value) converter found for C++ type: boost::intrusive_ptr<C>

The set_ptr method no longer generates an exception, but c2's lifetime is not tied to c1 like it is for the shared_ptr version (the destructor gets called when c2 gets deleted).  Another type conversion problem pops up in 'In[5]'.

Am I on the right path here?  Is it possible to make the intrusive_ptr work as seamlessly as the shared_ptr, or is there some fundamental problem with what I hope to achieve?  Is there some technical reason why this is not already implemented, or is it just that there hasn't been a need up to this point?  If I should continue down this path, can someone give me a skeleton of the code I need to write (and how to get it 'registered') to provide all the functionality that exists for the shared_ptr class?

Thanks again,

Jeff Webb


s_ptr_lib.cpp
--------------------------------------------------------------
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <string>
#include <iostream>

using namespace std;
using namespace boost;
using namespace boost::python;

class C;
typedef shared_ptr<C> CPtr;

class C
{
public:
 C(const string &s) : name(s)
 {
   cout << "C(" << name << ")" << endl;
 }
 ~C() 
 {
   cout << "~C(" << name << ")" << endl;
 }
 void set_ptr(CPtr ptr)
 {
   this->ptr = ptr;
 }
 void clear_ptr()
 {
   ptr.reset();
 }
 CPtr get_ptr()
 {
   return ptr;
 }
private:
 string name;
 CPtr ptr;
};

BOOST_PYTHON_MODULE(s_ptr_lib)
{
 class_<C>("C", init<const string&>())
   .def("set_ptr", &C::set_ptr)
   .def("clear_ptr", &C::set_ptr)
   .def("get_ptr", &C::get_ptr)
   ;
}



i_ptr_lib.cpp
--------------------------------------------------------------
#include <boost/python.hpp>
#include <boost/intrusive_ptr.hpp>
#include <string>
#include <iostream>

using namespace std;
using namespace boost;
using namespace boost::python;

class C;
typedef intrusive_ptr<C> CPtr;

namespace boost
{
 void intrusive_ptr_add_ref(C * ptr);
 void intrusive_ptr_release(C * ptr);
}

class C
{
public:
 C(const string &s) : name(s)
 {
   cout << "C(" << name << ")" << endl;
 }
 ~C() 
 {
   cout << "~C(" << name << ")" << endl;
 }
 void set_ptr(CPtr ptr)
 {
   this->ptr = ptr;
 }
 void clear_ptr()
 {
   intrusive_ptr_release((C*) ptr.get());
 }
 CPtr get_ptr()
 {
   return ptr;
 }
private:
 friend void boost::intrusive_ptr_add_ref(C * ptr);
 friend void boost::intrusive_ptr_release(C * ptr);

 string name;
 CPtr ptr;
 int ref_count;
};

namespace boost
{
 inline void intrusive_ptr_add_ref(C * ptr)
 {
   ++(ptr->ref_count);
 }

 void intrusive_ptr_release(C * ptr)
 {
   if (--(ptr->ref_count) == 0)
     delete ptr;
 }
}

BOOST_PYTHON_MODULE(i_ptr_lib)
{
class_<C>("C", init<const string&>())
   .def("set_ptr", &C::set_ptr)
   .def("clear_ptr", &C::set_ptr)
   .def("get_ptr", &C::get_ptr)
   ;
}