[Python-Dev] Weak references: dereference notification

[Ronald Oussoren]

On 9-nov-2005, at 23:44, Gustavo J. A. M. Carneiro wrote:

> On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote:
>> On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote:
>>
>>> Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu:
>>>>>> Gustavo J. A. M. Carneiro wrote:
>>>>>>>   I have come across a situation where I find the current weak
>>>>>>> references interface for extension types insufficient.
>>>>>>>
>>>>>>>   Currently you only have a tp_weaklistoffset slot, pointing  
>>>>>>> to a
>>>>>>> PyObject with weak references.  However, in my case[1] I
>>>>>>> _really_ need
>>>>>>> to be notified when a weak reference is dereferenced.
>>>>
>>>> I find reading through the bug discussion a bit difficult to
>>>> understand your use case. Could you explain it here? If you can't
>>>> explain it you certainly won't get your problem solved! :-)
>>>
>>>   This is a typical PyObject wrapping C object (GObject) problem.
>>> Both
>>> PyObject and GObject have independent reference counts.  For each
>>> GObject there is at most one PyObject wrapper.
>>>
>>>   When the refcount on the wrapper drops to zero, tp_dealloc is
>>> called.
>>> In tp_dealloc, and if the GObject refcount is > 1, I do something
>>> slightly evil: I 'resurect' the PyObject (calling PyObject_Init),
>>> create
>>> a weak reference to the GObject, and drop the "strong" reference.  I
>>> call this a 'hibernation state'.
>>
>> Why do you do that? The only reasons I can think of are that you hope
>> to gain
>> some speed from this or that you want to support weak references to
>> the GObject.
>
>   We want to support weak references to GObjects.  Mainly because that
> support has always been there and we don't want/can't break API.   
> And it
> does have some uses...
>
>>
>> For what its worth, in PyObjC we don't support weak references to the
>> underlying
>> Objective-C object and delete the proxy object when it is garbage
>> collected.
>> Objective-C also has reference counts, we increase that in the
>> constructor for
>> the proxy object and decrease it again in the destroctor.
>
>   OK, but what if it is a subclass of a builtin type, with instance
> variables?  What if the PyObject is GC'ed but the ObjC object remains
> alive, and later you get a new reference to it?  Do you create a new
> PyObject wrapper for it?  What happened to the instance variables?

Our main goal is that there is at most one wrapper for a python object
alive at any one time. And likewise there is at most one Objective-C
wrapper for a python object.

If a PyObject is GC'ed and the ObjC object remains alive you will get
a new PyObject when a reference to the ObjC object passes into python
space again. That is no problem because the proxy object contains no
state other than the pointer to the ObjC object.

ObjC's runtime might be more flexible than that of GObject. If you  
create
a subclass of an ObjC class the PyObjC runtime will create a real ObjC
class for you and all object state, including Python instance variables,
are stored on the ObjC side.
>
>   Our goal in wrapping GObject is that, once a Python wrapper for a
> GObject instance is created, it never dies until the GObject dies too.
> At the same time, once the python wrapper loses all references, it
> should not stop keeping the GObject alive.

I tried that too, but ran into some very ugly issues and decided that
weak references are not important enough for that. There's also the  
problem
that this will keep the python proxy alive even when it is not needed  
anymore,
which gives significant overhead of you traverse a large datastructure.


What I don't quite understand is how you know that your python wrapper
is the last reference to the GObject and your wrapper should not be
forcefully kept alive.

>
>   What happens currently, which is what I'm trying to change, is that
> there is a reference loop between PyObject and GObject, so that
> deallocation only happens with the help of the cyclic GC.  But relying
> on the GC for _everything_ causes annoying problems:

At one time I used Python's reference counts as the reference count  
of the
Objective-C object (ObjC's reference count management is done through  
method
calls and can therefore be overridden in subclasses). That did work, but
getting the semantics completely correct turned the code into a mess.  
Our
current solution is much more satisfying.

>
> 	1- The GC runs only once in a while, not soon enough if eg. you  
> have an
> image object with several megabytes;
>
> 	2- It makes it hard to debug reference counting bugs, as the symptom
> only appears when the GC runs, far away from the code that cause the
> problem in the first place;
>
> 	3- Generally the GC has a lot more work, since every PyGTK object  
> needs
> it, and a GUI app can have lots of PyGTK objects.
>
>   Regards.
>
> -- 
> Gustavo J. A. M. Carneiro
> <gjc at inescporto.pt> <gustavo at users.sourceforge.net>
> The universe is always one step beyond logic
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com