Design thought for callbacks

[Cem Karan]

Hi all, I apologize for taking so long to reply, but neither my work schedule nor the weather have been kind in the past week.  That said, I've been thinking long and hard about what everyone has said, and have decided that it would be useful to write a wrap-up email that attempts to encapsulate everything everyone has said, as a record of sorts if nothing else.  As such, this email is fairly long and involved.

=======================
Analysis of the problem
=======================

My original question was 'what is the least surprising/most pythonic way to write a callback API?'  Through reading what everyone has said, I realized that I wasn't being specific enough, simply because callback APIs can be quite different.  At the very least, the following questions need to be answered:

1) When a callback is registered, does it replace the prior callback?
2) If more than one callback can be registered, is there an ordering to them?
3) Can a callback be registered more than once?
4) When and how are callbacks deregistered?
5) Who is responsible for maintaining a strong reference to the callback?

As far as I know, there isn't a standard method to indicate to the caller that one callback replaces another one except via well-written documentation.  My personal feeling is that callbacks that replace other callbacks should be properties of the library.  By implementing a setter, getter, and deleter for each callback, the library makes it obvious that there is one and only one callback active at a time.  The only difficulty is making sure the user knows that the library retains the callback, but this is a documentation problem.  

I realized that ordering could be a problem when I read through the documentation to asyncio.call_soon().  It promises that callbacks will be called in the order in which they were registered.  However, there are cases where the order doesn't matter.  Registration in both of these cases is fairly simple; the former appends the callback to a list, while the latter adds it to a set.  The list or set can be a property of the library, and registration is simply a matter of either inserting or adding.  But this brings up point 3; if a callback can be registered at most once and ordering matters, then we need something that is both a sequence and a set.  Subclassing either (or both) collections.abc.MutableSequence or collections.abc.MutableSet will lead to confusion due to unexpected violations of PEP 3119 (https://www.python.org/dev/peps/pep-3119/).  Once again, the only option appears to be careful documentation.

Registration is only half the problem.  The other half is determining when a callback should be unregistered.  Some callbacks are one-shots and are automatically unregistered as soon as they are called.  Others will be called each time an event occurs until they are explicitly unregistered from the library.  Which happens is another design choice that needs to be carefully documented.

Finally, we come to the part that started my original question; who retains the callback.  I had originally asked everyone if it would be surprising to store callbacks as weak references.  The idea was that unless someone else maintained a strong reference to the callback, it would be garbage collected, which would save users from 'surprising' results such as the following:

"""
#! /usr/bin/env python

class Callback_object(object):
    def __init__(self, msg):
        self._msg = msg
    def callback(self, stuff):
        print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
    def __init__(self):
        self._callbacks = list()
    def register_callback(self, callback):
        self._callbacks.append(callback)
    def execute_callbacks(self):
        for thing in self._callbacks:
            thing('Surprise!')

if __name__ == "__main__":
    cbo = Callback_object("Evil Zombie")
    lib = Fake_library()
    lib.register_callback(cbo.callback)

    # Way later, after the user forgot all about the callback above
    cbo = Callback_object("Your Significant Other")
    lib.register_callback(cbo.callback)

    # And finally getting around to running all those callbacks.
    lib.execute_callbacks()
"""

However, as others pointed out using a weak reference could actually increase confusion rather than decrease it.  The problem is that if there is a reference cycle elsewhere in the code, it is possible that the zombie object is still alive when it is supposed to be dead.  This will likely be difficult to debug. In addition, different types of callables have different requirements in order to correctly store weak references to them.  Both Ian Kelly and Fabio Zadrozny provided solutions to this, with Fabio providing a link to his code at http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html.

====================================
Solution to my problem in particular
====================================

After considering all the comments above, I've decided to do the following for my API:

- All callbacks will be strongly retained (no weakrefs).
- Callbacks will be stored in a list, and the list will be exposed as a read-only property of the library.  This will let users reorder callbacks as necessary, add them multiple times in a row, etc.  I'm also hoping that by making it a list, it becomes obvious that the callback is strongly retained.
- Finally, callbacks are not one-shots.  This just happens to make sense for my code, but others may find other methods make more sense.


Thanks again to everyone for providing so many comments on my question, and I apologize again for taking so long to wrap things up.

Thanks,
Cem Karan