Design thought for callbacks

[Cem Karan]

On Feb 21, 2015, at 8:15 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>> OK, so it would violate the principle of least surprise for you.  Interesting.  Is this a general pattern in python?  That is, callbacks are owned by what they are registered with?
>> 
>> In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me.  If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations.  Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.
>> 
>> So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?
> 
> I don't know about Python specifically, but it's certainly a general
> pattern in other languages. They most definitely are owned, and it's
> the only model that makes sense when you use closures (which won't
> have any other references anywhere).

I agree about closures; its the only way they could work.  When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects.  The callable objects may pass themselves, or one of their methods to the library, or may do something really weird.  

Although I just realized that closures may cause another problem.  In my code, I expect that many different callbacks can be registered for the same event.  Unregistering means you request to be unregistered for the event. How do you do that with a closure?  Aren't they anonymous?

> If you're expecting 'del foo' to destroy the object, then you have a
> bigger problem than callbacks, because that's simply not how Python
> works. You can't _ever_ assume that deleting something from your local
> namespace will destroy the object, because there can always be more
> references. So maybe you need a more clear way of saying "I'm done
> with this, get rid of it".

Agreed about 'del', and I don't assume that the object goes away at the point.  The problem is debugging and determining WHY your object is still around.  I know a combination of logging and gc.get_referrers() will probably help you figure out why something is still around, but I'm trying to avoid that headache.  

I guess the real problem is how this creates cycles in the call graph.  User code effectively owns the library code, which via callbacks owns the user code.  I have no idea what the best point the cycle is to break it, and not surprise someone down the road.  The only idea I have is to redesign the library a little, and make anything that accepts a callback actually be a subclass of collections.abc.Container, or even collections.abc.MutableSet.  That makes it very obvious that the object owns the callback, and that you will need to remove your object to unregister it.  The only problem is how to handle closures; since they are anonymous, how do you decide which one to remove?

Thanks,
Cem Karan