[C++-sig] Boost.Python to_python_indirect and intrusive_ptr.

[Nicolas Colombe]

Hi everyone.

I'm trying to use Boost.Python to expose some of my C++ classes and I
encountered an odd behaviour of the to_python_indirect result converter for
intrusive_ptr with specific classes.
Here is the deal:

I'm using a component-based design for my classes, meaning an aggregation of
objects with different features. These objects are ref-counted, and if they
are in the same aggregation, they share the same lifetime.
That is to say, as long as one of the components in the "ring" has a
strictly positive reference counter, every component in the ring survive
even if their ref-count drops to zero.
I use boost::intrusive_ptr to manipulate them easily.

There is the basic interface:

class Component
{
    void AddRef();
    void Release();
    unsigned int GetRefCount();
    bool AddComponent(Component*);
    bool RemoveComponent(Component*);
    Component* ComponentIterator(Component*);    //-->Used to parse the
ring.

    virtual some_virtual_fn();
};

To avoid lifetime problems when manipulating an object both from C++ and
python side, i want python to manipulate components only through
boost::intrusive_ptr.

There is the exposition code:

//-->Object wrapper
struct Component_wrapper : SandBoxProj::Component, bp::wrapper<
SandBoxProj::Component > {

    Component_wrapper( ) :Component( ), bp::wrapper< SandBoxProj::Component
>(){}

    virtual some_virtual_fn() {
          //overriding code...
    }

    default_some_virtual_fn() {
       Component::some_virtual_fn();
    }
};

//-->Some declarations that boost.python needs to manipulate
boost::intrusive_ptr
namespace boost { namespace python {

   template <class T> struct pointee< intrusive_ptr<T> >   //-->pointee
struct for intrusive_ptr holder
     {
       typedef T type;
     };

 struct make_owning_intrusive_ptr_holder    //-->ResultConverter to make an
intrusive_ptr from a raw one, inspired from to_python_indirect.hpp:79
  {
    template <class T>
    static PyObject* execute(T* p)
    {
      typedef objects::pointer_holder<boost::intrusive_ptr<T>, T> holder_t;

      boost::intrusive_ptr<T> ptr(p);
      return boost::python::objects::make_ptr_instance<T,
holder_t>::execute(ptr);
    }
  };

  struct return_by_intrusive_ptr    //-->corresponding
ResultConverterGenerator
  {
    template <class T>
    struct apply
    {
      typedef to_python_indirect<T, make_owning_intrusive_ptr_holder> type;
//--> here is the usage of to_python_indirect
    };
  };

}}

bp::class_<
Component_wrapper,boost::intrusive_ptr<Component_wrapper>,boost::noncopyable
>( "Component", bp::init< >() )
    .def( "AddComponent", &Component::AddComponent, ( bp::arg("arg0") ) )

    .def( "ComponentIterator",&Component::ComponentIterator, (
bp::arg("arg0") ), bp::return_value_policy< bp::return_by_intrusive_ptr >()
)    //-->this class uses the converter.
    .def(
"some_virtual_fn",&Component::some_virtual_fn,&Component_wrapper::default_some_virtual_fn)
    .def( "GetRefCount",&Component::GetRefCount)
    .def( "RemoveComponent",&Component::RemoveComponent, ( bp::arg("arg0") )
) ;


And finally there is a scripts that uses this class and its output:

>>>from pythonbinder import *
>>>import gc

>>>Comp1=Component()
>>>Comp2=Component()
>>>print Comp2.GetRefCount()
1
//-->Ok, I manipulate an intrusive_ptr
>>>Comp2.AddComponent(Comp1)
>>>iter=None
>>>iter=Comp1.ComponentIterator(iter)
>>>print iter
<pythonbinder.Component object at 0x026DA630>           //-->Ok, There is my
C++ object
>>>iter=Comp1.ComponentIterator(iter)
>>>print iter
<pythonbinder.Component object at 0x026DA600>
>>>print iter.GetRefCount()
1
//-->What the ? iter is supposed to be an intrusive ptr. After a look at
to_python_indirect, it seems that iter==Comp2, the python object wrapping
the intrusive_ptr was just addref'ed. Ok, fair enough.

>>>iter=Comp1.ComponentIterator(iter)
>>>print iter
None
    //-->Now the troubles begin.
>>>del Comp2
>>>gc.collect()    //-->ensure deletion, Comp2 is still alive from the C++
side, because Comp1 is alive.
>>>iter=Comp1.ComponentIterator(iter)  //May crash
>>>print iter
    //-->if we made it through, a weak_ptr to some random structure.

After that, i figured out that i never went through the ResultConverters i
declared. Let's look at the execute function from to_python_indirect,

 template <class U>
    inline PyObject* execute(U const& x, mpl::false_) const
    {
        U* const p = &const_cast<U&>(x);
        if
(is_polymorphic<U>::value)
//-->My type is polymorphic
        {
            if (PyObject* o = detail::wrapper_base_::owner(p))
//-->Ok, that's why I have a refcount of 1, no matter how many python
intrusive_ptr wrappers I seem to have
                return incref(o);
        }
        return
MakeHolder::execute(p);                                         //-->There
is the call to my result converter.
    }

What happen is that after deletion of the last python wrapping of the
intrusive_ptr, which is o in the code above, the Component_wrapper, which is
partly a python object, is not deleted (the component ring still holds it),
but it still reference its last owner which is the last instance_holder that
just got deleted, so the object returned is a random memory garbage.

Changing the holder type fromintrusive_ptr<Component_wrapper> to
intrusive_ptr<Component> solves the above problems but then it's not
possible to use the class Component_wrapper as a base in python anymore.
I tried to create a class derived from Component in python and the
constructor of Component_wrapper was not called. (purpose is overriding
virtual functions in python)

I wondered if I'm going the wrong way, or if I discovered a bug.
It may be a limitation of the library, which cannot understand that an
object survived the deletion of its last holder.

Maybe if i made my own version of to_python_indirect but removing the
is_polymorphic case, it would work?
Is there a fundamental reason to this test, other than saving memory?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100403/4324afe8/attachment.html>