[C++-sig] boost.python - C++ class overridden in python causes slicing

[Jean-Sébastien Guay]

Hi all,

Thanks to previous help (a while ago) I was able to make considerable 
progress wrapping a pretty complex library (OpenSceneGraph) with 
boost.python. I've been working on this a bit more lately in my free 
time, and I've been able to wrap pretty much everything I wanted to. 
It's starting to look really good.

But I've got a problem overriding classes with python code. The 
overridden method gets called, but when (from python code) it calls some 
other C++ method using the arguments it was given, the arguments are 
sliced to the base type.

I'll give some example code, but unfortunately I have to give a lot for 
it to make sense... Sorry about that. The definitions below are just for 
context:

   class osg::Node
   {
     public:
       //...
       // does nothing in base class
       virtual void traverse(NodeVisitor& nv) {}
       // ...
   };

   class osg::Group
   {
     public:
       //...
       // overridden to traverse children
       virtual void traverse(NodeVisitor& nv) {...}
       // ...
   };

class osg::NodeVisitor
{
   public:
     // ...
     virtual void apply(osg::Node& node) { traverse(node); }
     virtual void apply(osg::Group& node) { 
apply(static_cast<Node&>(node); }
     // ... versions for other subclasses of osg::Node and osg::Group
     inline void traverse(osg::Node& node)
     {
       // ...
       node.traverse(*this);
     }
     // ...
}

This is a classic visitor double-dispatch implementation, which in C++ 
is used like this:

class DerivedVisitor : public osg::NodeVisitor
{
   public:
     DerivedVisitor() :
       osg::NodeVisitor(osg::NodeVisitor::TRAVERSE_ALL_CHILDREN)
     {}

     // Override the version of apply you want to do the work.
     void apply(osg::Node& node)
     {
       // ... Do something
       // method must call traverse(node) to continue traversal.
       traverse(node);
     }
};

main()
{
   osg::Group* g1 = new osg::Group;
   g1->setName("g1");
   osg::Group* g2 = new osg::Group;
   g2->setName("g2");
   g1->addChild(g2);
   osg::Node* n = new osg::Node;
   n->setName("n");
   g2->addChild(n);

   DerivedVisitor v;
   g1->accept(v);
}

This will call apply(osg::Node&) 3 times, because the first two times, 
traverse(node) will call osg::Group's version of traverse(NodeVisitor&) 
which traverses its children. (note that NodeVisitor's default 
implementation of apply(Group&) is to call apply(Node&) and the same for 
all other subclasses which call their parent class's apply() )

Now, the problem I'm having with this in boost.python is that apply_Node 
(which is the name I've given my wrapped apply(Node&) so that python 
knows which version to call) is being called only once (at the 
parent-most node). So it seems that it's osg::Node's version of 
traverse(NodeVisitor&) that's being called since it's not traversing to 
its children. I've got another example of this with NodeCallback, where 
a method traverse(Node*) should traverse to its children, but it doesn't 
and I can see that because if it did, a certain 3D model would get drawn 
to the screen, but it doesn't (and it does if I remove my 
python-overridden NodeCallback).

My wrapper code is mostly based on the example here:
http://wiki.python.org/moin/boost.python/OverridableVirtualFunctions

struct NodeVisitor_wrapper : public NodeVisitor
{
     // NodeVisitor constructors storing initial self parameter
     NodeVisitor_wrapper(PyObject *p,
                         NodeVisitor::TraversalMode tm =
                             NodeVisitor::TRAVERSE_NONE)
         : NodeVisitor(tm), self(p) {}

     NodeVisitor_wrapper(PyObject *p, NodeVisitor::VisitorType type,
                         NodeVisitor::TraversalMode tm =
                             NodeVisitor::TRAVERSE_NONE)
         : NodeVisitor(type, tm), self(p) {}

     // In case NodeVisitor is returned by-value from a wrapped function
     NodeVisitor_wrapper(PyObject *p, const NodeVisitor& x)
         : NodeVisitor(x), self(p) {}

     // Override apply to call back into Python
     void apply(Node& node)
     {
         //std::cout << "in apply(Node&)" << std::endl;
         try {
             //std::cout << "Calling override" << std::endl;
             call_method<void>(self, "apply_Node", node);
         }
         // Catch boost::python exception, means method was not
         // overridden in subclass.
         catch (error_already_set) {
             NodeVisitor::apply(node);
         }
     }

     // Supplies the default implementation of apply
     void default_apply_Node(NodeVisitor& self_, Node& node)
     {
         //std::cout << "in default_apply(Node&)" << std::endl;
         self_.NodeVisitor::apply(node);
     }

     // Override apply to call back into Python
     void apply(Group& node)
     {
         //std::cout << "in apply(Group&)" << std::endl;
         try {
             //std::cout << "Calling override" << std::endl;
             call_method<void>(self, "apply_Group", node);
         }
         // Catch boost::python exception, means method was not
         // overridden in subclass.
         catch (error_already_set) {
             NodeVisitor::apply(node);
         }
     }

     // Supplies the default implementation of apply
     void default_apply_Group(NodeVisitor& self_, Group& node)
     {
         //std::cout << "in default_apply(Group&)" << std::endl;
         self_.NodeVisitor::apply(node);
     }

  private:
     PyObject* self;
};

BOOST_PYTHON_MODULE(_osg)
{
     // ...

     {
         scope in_NodeVisitor = class_<NodeVisitor, NodeVisitor_wrapper,
                                       bases<Referenced>,
                                       ref_ptr<NodeVisitor> >
                                       ("NodeVisitor")
             .def(init<NodeVisitor::TraversalMode>())
             .def(init<NodeVisitor::VisitorType,
                      NodeVisitor::TraversalMode>())
             .def("traverse", &NodeVisitor::traverse)
             .def("apply_Node", &NodeVisitor_wrapper::default_apply_Node)
             .def("apply_Group",
                      &NodeVisitor_wrapper::default_apply_Group)
             .add_property("traversalMode",
                               &NodeVisitor::getTraversalMode,
                               &NodeVisitor::setTraversalMode)
         ;

         enum_<NodeVisitor::TraversalMode>("TraversalMode")
             .value("TRAVERSE_NONE",
                    NodeVisitor::TRAVERSE_NONE)
             .value("TRAVERSE_PARENTS",
                    NodeVisitor::TRAVERSE_PARENTS)
             .value("TRAVERSE_ALL_CHILDREN",
                    NodeVisitor::TRAVERSE_ALL_CHILDREN)
             .value("TRAVERSE_ACTIVE_CHILDREN",
                    NodeVisitor::TRAVERSE_ACTIVE_CHILDREN)
         ;

         enum_<NodeVisitor::VisitorType>("VisitorType")
             .value("NODE_VISITOR",
                    NodeVisitor::NODE_VISITOR)
             .value("UPDATE_VISITOR",
                    NodeVisitor::UPDATE_VISITOR)
             .value("EVENT_VISITOR",
                    NodeVisitor::EVENT_VISITOR)
             .value("COLLECT_OCCLUDER_VISITOR",
                    NodeVisitor::COLLECT_OCCLUDER_VISITOR)
             .value("CULL_VISITOR",
                    NodeVisitor::CULL_VISITOR)
         ;

     }

     // ...
}

Note a few things about this:

1. I couldn't use the method for wrapping a class which you want to 
override in python code that's given here:
 
http://www.boost.org/doc/libs/1_40_0/libs/python/doc/tutorial/doc/html/python/exposing.html#python.class_virtual_functions

because I would always get crashes at the get_override() line. But the 
above technique works well (apart from the slicing problem). I'm using 
boost 1.35 if that helps explain that.

2. I'm using try { ... } catch (error_already_set) { ... } because in 
the library, all the virtual methods are optional to override. 
call_method() bombs if my python subclass didn't have the method 
overridden, so that's the way I found to make that work. If there's a 
better way (perhaps to check if the method was indeed overridden or not 
in the subclass, which is that the get_override() line I referred to 
above does I know) please let me know.

Finally, my python code is:

     class DerivedVisitor(osg.NodeVisitor):
         def __init__(self):
             # call parent class constructor with argument, as in C++
             osg.NodeVisitor.__init__(self,
                 osg.NodeVisitor.TraversalMode.TRAVERSE_ALL_CHILDREN)
         def apply_Node(self, node):
             print "python apply_Node - node name:", node.name
             # call traverse(node) as in C++ to continue traversal.
             self.traverse(node)

     g1 = osg.Group()
     g1.name = "g1"
     g2 = osg.Group()
     g2.name = "g2"
     g1.addChild(g2)
     n = osg.Node()
     n.name = "n"
     g2.addChild(n)

     nv = DerivedVisitor()
     g1.accept(nv)

Note that the code is quasi-identical to the C++ version, right down to 
the need to call the base class constructor with the right traversal 
mode. That's what I want.

Now, if I provide overridden versions of both apply_Node() and 
apply_Group(), I get correct results, but not when I only override 
apply_Node(). So it would seem the slicing happens when call_method 
calls that method with the osg::Node& argument. Considering when I 
override osg::NodeVisitor in C++ (and only override apply(Node&) ) it 
works as it should, why does the boost.python version slice off the 
derived parts of objects? I can see nowhere where objects are passed by 
value, so (IIRC) no slicing should occur...

In fact, no pass-by-value could occur, because all osg::Referenced 
subclasses (which includes osg::Node and osg::Group) have protected 
destructors so they can't be constructed on the stack.

If it helps, here's the top part of my osg::Node and osg::Group wrappers:

using namespace osg;

namespace boost {
namespace python {
   template <class T> struct pointee< ref_ptr<T> >
   {
      typedef T type;
   };
}
}

BOOST_PYTHON_MODULE(_osg)  // repeated just for context
{
   class_<Referenced, ref_ptr<Referenced> >("Referenced")
   ;

   {
     scope in_Object = class_<Object, bases<Referenced>, ref_ptr<Object>,
                              boost::noncopyable >("Object", no_init)
       // ...
     ;
     // ...
   }

   {
     scope in_Node = class_<Node, bases<Object>, ref_ptr<Node> >("Node")
       // ...
     ;
     // ...
   }

   class_<Group, bases<Node>, ref_ptr<Group> >("Group")
     // ...
   ;
}

All this (in theory) works, in the sense that I can make hierarchies of 
nodes, groups, geodes, geometry, etc. all in Python code, also assign 
textures, start a viewer to see all that, etc. Apart from another 
related problem (which I'll start a separate thread to get help for) 
this is the only part that's giving me problems for now.

Any help would be appreciated. Sorry for the length of this message, I 
wanted to be thorough so the problem would be clear (and perhaps the 
solution easy to see?). I hope it's not something too simple though, or 
I'll have wasted a lot of time and a lot of your time too...

Thanks in advance,

J-S
-- 
______________________________________________________
Jean-Sebastien Guay    jean-sebastien.guay at cm-labs.com
                                http://www.cm-labs.com/
                         http://whitestar02.webhop.org/