[C++-sig] Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

[Holger Brandsmeier]

Adam,

>> So, before the function exits your object `arg` exists at least in three
>> places:
>>  1) somewhere in C++ where it was created
>>  2) in the python context
>>  3) in the context of addCallback
>>
>
> Technically, it was declared and constructed from Python, but everything you
> say is a consequence of this is consistent with what I'm seeing.  We could
> get into semantics here.  If I create an object implementing a C++
> interface, do we consider that created in Python or would it be regarded as
> created in the C++ runtime?

Regard it as beein created from C++, as it has been created from boost
python. If it would be a pure python object, then it would be a
different story as there would not be a shared_ptr in the first place.

>> Assume you have class `Parent` and a class `Child` derived from it.
>> Now you can do:
>>  - create an instance of Child C++ and bring it to python as
>> shared_ptr<Child>
>>  - pass that instance to C++ (via shared_ptr<Child> or shared_ptr<Parent>)
>>  - get it later back from C++ but as a shared_ptr<Parent>
>>  - magic: you can treat that instance as a shared_ptr<Child>
>> In C++ you would need to do a dynamic cast to get this functionallity,
>> but because that object has been known to python to be an instance of
>> Child, boost::python automatically makes it an instance of Child, nice
>> right?
>>
>> Unfortunately your (and my) problem are a consequence of this. When
>> you go from 2->3 boost::python prepares for doing its magic. It
>> doesn't just return a copy of the shared_ptr from 1), it creates a new
>> shared_ptr with a special Deallocator object. The use_count at that
>> moment is 2: one for python 2) and one for addCallback() 3). When the
>> function addCallback() finishes, the use_count=1 (from python) and
>> weak_count=1 from 3). Once the python context ends then use_count=0
>> and weak_count=1, and I believe that is exactly what you observe.
>>
>> In this case as use_count drops to 0 the boost custom Deallocator gets
>> called. This is usally not bad, as he just deregisters (decreases the
>> use cound by 1) in the shared_ptr for context 1.Only if that use_count
>> in context 1) would drop to 0 the object would get deleted (that's why
>> you don't observe it to be deleted). The problem is now, that the two
>> weak/shared ptrs (which still point to a healthy and alive object) are
>> now disconnected. So when you try to turn the weak_ptr in context3 to
>> a strong pointer you would get serious problems.
>>
>
> Sounds about like what I'm dealing with here.  Perhaps of particular note is
> that the pointers I'm moving around are typed for a parent class, but the
> actual reference is to a child.

I don't think that it matters if you actually use that feature or not.
Just because the functionality is there, you get the problem. I just
wanted to give you an idea of why it is very nice that we have that
functionality.

>> I hope this is a correct explanation of the sitation. To solve this
>> would need to revist the magic for shared_ptrs in boost::python. I
>> plan to try and solve it some point later, but I am no regular
>> developer for boost::python and I can not promise that I will succeed,
>> nor when.
>>
>
> Originally I was using shared_ptr instead of weak_ptr for the callback
> managers, and found some stuff never got deleted.  Could this process also
> cause the disconnect there?

If you are storing the result as a shared_ptr in the problem, then you
will not run into problem, that is what I am doing at the moment. This
implies that my class never gets deleted, which I can accept at the
moment, but its actually quite bad.

Maybe someone else on the list has a solution for this. At the moment
I can only explain what I believe is causing the error.

-Holger