[C++-sig] Pointer to pointer class_ held type, arguments to functions, and polymorphism

[English, Mark]

Problem
=======
Using smart pointers to smart pointers as held type of class_ loses argument matching capabilities
for functions which take a smart pointer to held type.
Also lose implicit casting ability.

Setup
=====
Windows 2000 SP4
MSVS 2003 (no service pack)
Python 2.3.6
Boost 1.32

Background
==========
--Original problem--
Constrained to using a third-party smart pointer.
Using this as the held type of a class_ loses Python polymorphic capabilities.
For example:
given a class "base_i", derived class "derived_t" and a function "base_ptr_to_base_ptr" exposed through boost python:
// cpp code snippet start
class base_i{};
class derived_t : public base_i {};
smart_pointers::shared_ptr_t<base_i> base_ptr_to_base_ptr(smart_pointers::shared_ptr_t<base_i> ptr)
{ return ptr; }
// cpp code snippet end

#python code snippet start
class pyderived(derived_t):
	def pytest(self): return 'pyderived test'
pyd = pyderived()
bfrompyd = base_ptr_to_base_ptr(pyd)
bfrompyd.pytest() #FAILS with c++ held type third party smart pointer
#python code snippet end

This seems to be a common problem and no solution has been presented that I'm aware of.
The following posts seem related:
http://mail.python.org/pipermail/c++-sig/2006-March/010423.html (I haven't tried using has_back_reference - I'm not sure it would help here)
http://mail.python.org/pipermail/c++-sig/2003-December/006481.html (doesn't deal with python derived classes)

There is example code here:
http://www.language-binding.net/pyplusplus/troubleshooting_guide/shared_ptr/shared_ptr.html
which demonstrates using a third party smart pointer.
The snippets above are actually name-changed code to use the same names as this example,
and I'll continue to do that for the remainder of this email so if you're interested in it you
might want to look at the sample code here:
http://www.language-binding.net/pyplusplus/troubleshooting_guide/shared_ptr/shared_ptr.zip


-- Potential solution to original problem--
Boost Python relies on boost::shared_ptr (and in particular the boost python checked deleter ?) to support
Python subclasses. The code snippets above would work fine if boost::shared_ptr were used, and as a result the original object
would be returned by "base_ptr_to_base_ptr()".

So one approach is to use boost::shared_ptr because Boost::Python "requires" it, wrapped around the third party
smart_ptr because the library being wrapped requires it.

I've put example code at the bottom of this mail tagged with
big c++ code snippet to use boost::shared_ptr<smart_pointers::smart_ptr_t<T> >

This allows the code snippets above to work (substituting the ptr class names for the original class names in the snippets).

Functions with raw pointers arguments will work, as I understand it, because of the pointee and get_pointer specializations.
Functions with boost::shared_ptr <smart_pointers::smart_ptr_t<T> > arguments will work because that's the held type.

Problems
==========
==New Problem 1==
Can't implicitly convert from base held type to derived held type.
i.e.
 bp::implicitly_convertible< 
               boost::shared_ptr < smart_pointers::smart_ptr_t< derived_t > >
             , boost::shared_ptr < smart_pointers::smart_ptr_t< base_i > > >();
Doesn't work.

==Potential solution to Problem 1==
- Write a converter for boost::shared_ptr < smart_pointers::smart_ptr_t< T > >.
  This is in the example code at the bottom named "ptr_ptr_from_python".
- Register a converter for base_i

As commented this is naively based on the code in shared_ptr_from_python.hpp
I'd really like some input on how to solve this problem.

I also tried specialising 
 template <class Source, class Target>
 struct implicit<boost::shared_ptr<smart_pointers::smart_ptr_t <Source> >
                ,boost::shared_ptr<smart_pointers::smart_ptr_t <Target> >
                >
but this seemed like a really bad idea.

==New Problem 2==
Can't call functions like "base_ptr_to_base_ptr()" because held type
boost::shared_ptr < smart_pointers::smart_ptr_t< T > >
won't match argument signature smart_pointers::smart_ptr_t< T >

==New problem 2 Solution==
I have no idea.

At the moment using this approach I have to write wrappers for all code which takes
smart_pointers::smart_ptr_t< T >
so that the wrapper takes
boost::shared_ptr < smart_pointers::smart_ptr_t< T > >
and dereferences the boost::shared_ptr calling the wrapped function.

I thought there might be a way to register a conversion but I can't find it.
The type of the passed in argument seems to be the result of pointee (i.e. T *)
and this can't find a registered match for the held type of the class_.
For example it seems that where T is derived_t,
the registered conversions/initialisers/??? are:
 smart_pointers::smart_ptr_t< derived_t >
 smart_pointers::smart_ptr_t< base_wrapper_t >
 smart_pointers::smart_ptr_t< base_i >

Changing pointee to return smart_pointers::smart_ptr_t<T> breaks all kinds of other code.

So my question is: is there a relatively easy,
not-having-to-specialise-arg_from_python-and-numerous-other-bits-of-implementation-dependant-code
way to support conversion of the wrapped Python type to a smart_pointers::smart_ptr_t<T> ?
Possibly involving visitors ?

Obviously that question is based on the assumption that this whole pointer to pointer approach is useful and valid.
Other suggestions for supporting Python polymorphism (other than stop using smart_pointers::smart_ptr_t) welcome.

Related links:
I think this is the inverse of the question in:
http://mail.python.org/pipermail/c++-sig/2006-August/011080.html
which is why I ended up with a "from_python()" style "ptr_ptr_from_python" converter.

Could register_conversion help ?
http://mail.python.org/pipermail/c++-sig/2005-August/009487.html

Thanks,
Mark
X-Replace-Address: mark.ignorethisbit.english at xxxrbccmxxx.ignorethisbittoo_and_removethosepreceding_xxxs_.com

CODE EXAMPLE FOLLOWS:
(sorry for the size):
// ==Start big c++ code snippet to use boost::shared_ptr<smart_pointers::smart_ptr_t<T> >==
namespace boost{ 
// boost accessor function to get raw pointer from shared_ptr <smart_pointers::smart_ptr_t <T> >
template<class T>
inline T* get_pointer( boost::shared_ptr <smart_pointers::smart_ptr_t<T> > const& p )
{
  return const_cast< T* >( p.get()->get() );
}

// Expose to Python
namespace python{
    // Expose nested shared_ptr <smart_pointers::smart_ptr_t <T> > as pointee
    template<class T>
    struct pointee< boost::shared_ptr <smart_pointers::smart_ptr_t<T> > >
    {
        typedef T type;
    };
} // Ends namespace python
} // Ends namespace boost

// Converter to allow extraction of
// shared_ptr <smart_pointers::smart_ptr_t <base_i> >
// (based on shared_ptr_from_python.hpp)
namespace boost { namespace python { namespace converter {
template <class T>
struct ptr_ptr_from_python
{
 private:
    typedef smart_pointers::smart_ptr_t <T> TargetType;
    typedef boost::shared_ptr <TargetType>  PtrPtrType;
public:
    ptr_ptr_from_python()
    {
        converter::registry::insert(&convertible, &construct, type_id< PtrPtrType >());
    }

 private:
    static void* convertible(PyObject* p)
    {
        if (p == Py_None)
            return p;
        
        return converter::get_lvalue_from_python(p, registered<T>::converters);
    }
    
    static void construct(PyObject* source, rvalue_from_python_stage1_data* data)
    {
        void* const storage = ((converter::rvalue_from_python_storage<PtrPtrType >*)data)->storage.bytes;
        // Deal with the "None" case.
        if (data->convertible == source)
            new (storage) PtrPtrType();
        else
            new (storage) PtrPtrType(
                static_cast<TargetType*>(data->convertible),
                shared_ptr_deleter(handle<>(borrowed(source)))
                );
        
        data->convertible = storage;
    }
};
}}} // Ends namespace boost::python::converter

// Example holder class to demonstrate Python polymorphism
typedef std::vector<boost::shared_ptr <smart_pointers::smart_ptr_t<base_i> > > vec_smart_base_i;
struct hold_base_i
{
  vec_smart_base_i  m_vecHold;

  void add(vec_smart_base_i::value_type valAdd)
  {
    m_vecHold.push_back(valAdd);
  }

  vec_smart_base_i::value_type get(int nPos)
  {
    return m_vecHold.at(nPos);
  }
};

// Example constructor function
boost::shared_ptr <smart_pointers::smart_ptr_t <derived_wrapper_t> >
makeDerived()
{
  return boost::shared_ptr< smart_pointers::smart_ptr_t <derived_wrapper_t> >
    (new smart_pointers::smart_ptr_t <derived_wrapper_t>(new derived_wrapper_t));
}

// <snip> all the other code from the original example apart from the BOOST_PYTHON_MODULE declaration</snip>

BOOST_PYTHON_MODULE( custom_sptr ){
/// <snip> Code from original sample code goes here</snip>
  // For demonstration purposes expose same C++ types with different names,
  // but use boost::shared_ptr<smart_pointers::smart_ptr_t<T> > as held type
    bp::class_<base_wrapper_t
              ,boost::shared_ptr<smart_pointers::smart_ptr_t <base_wrapper_t> >
              // ^^^^^^Note use of shared_ptr <smart_ptr_t <T> > here
              ,boost::noncopyable
              >
      ("base_i_ptr"
      ,bp::no_init
      )
      .def( "get_value", bp::pure_virtual( &base_i::get_value ) );
      ;

    bp::class_<derived_wrapper_t
              ,boost::shared_ptr<smart_pointers::smart_ptr_t <derived_wrapper_t> >
              // ^^^^^^Note use of shared_ptr <smart_ptr <T> > here
              ,bp::bases< base_i >
              ,boost::noncopyable
              >
      ("derived_t_ptr"
      ,bp::no_init
      )
      .def("__init__", bp::make_constructor(&makeDerived))
          // ^^^^^^Constructor for shared_ptr<smart_ptr<derived_t> >
      .def( "get_value", &derived_t::get_value, &derived_wrapper_t::default_get_value );
      ;

    // Class used to demonstrate storing smart_ptr<derived_t>
    bp::class_<hold_base_i>
      ("hold_base_i")
      .def("add", &hold_base_i::add)
      .def("get", &hold_base_i:get)
      ;

    // Register a base converter
    boost::python::converter::ptr_ptr_from_python<base_i>();
}
// ==End big c++ code snippet to use boost::shared_ptr<smart_pointers::smart_ptr_t<T> >==

Oversized disclaimer follows:
________________________________________

This E-Mail (including any attachments) may contain privileged or confidential information.  It is intended only for the addressee(s) indicated above.

The sender does not waive any of its rights, privileges or other protections respecting this information.  

Any distribution, copying or other use of this E-Mail or the information it contains, by other than an intended recipient, is not sanctioned and is prohibited.

If you received this E-Mail in error, please delete it and advise the sender (by return E-Mail or otherwise) immediately. 

This E-Mail (including any attachments) has been scanned for viruses. 

It is believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. 

However, it is the responsibility of the recipient to ensure that it is virus free. 

The sender accepts no responsibility for any loss or damage arising in any way from its use.

E-Mail received by or sent from RBC Capital Markets is subject to review by Supervisory personnel. 

Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.

IRS CIRCULAR 230 NOTICE:  TO COMPLY WITH U.S. TREASURY REGULATIONS, WE ADVISE YOU THAT ANY U.S. FEDERAL TAX ADVISE INCLUDED IN THIS COMMUNICATION IS NOT INTENDED OR WRITTEN TO BE USED, AND CANNOT BE USED, TO AVOID ANY U.S. FEDERAL TAX PENALTIES OR TO PROMOTE, MARKET, OR RECOMMEND TO ANOTHER PARTY ANY TRANSACTION OR MATTER.