Boost.Python: Callbacks to class functions
I have an `EventManager` class written in C++ and exposed to Python. This is how I intended for it to be used from the Python side: class Something: def __init__(self): EventManager.addEventHandler(FooEvent, self.onFooEvent) def __del__(self): EventManager.removeEventHandler(FooEvent, self.onFooEvent) def onFooEvent(self, event): pass (The `add-` and `remove-` are exposed as static functions of `EventManager`.) The problem with the above code is that the callbacks are captured inside `boost::python::object` instances; when I do `self.onFooEvent` these will increase the reference count of `self`, which will prevent it from being deleted, so the destructor never gets called, so the event handlers never get removed (except at the end of the application). The code works well for functions that don't have a `self` argument (i.e. free or static functions). How should I capture Python function objects such that I won't increase their reference count? I only need a weak reference to the objects. -- View this message in context: http://boost.2283326.n4.nabble.com/Boost-Python-Callbacks-to-class-functions... Sent from the Python - c++-sig mailing list archive at Nabble.com.
On Aug 1, 2011, at 10:50 AM, diego_pmc <paulc.mnt@gmail.com> wrote:
How should I capture Python function objects such that I won't increase their reference count? I only need a weak reference to the objects.
http://docs.python.org/library/weakref.html#module-weakref I don't know how to access a Python weakref from Boost.Python.
On 08/01/2011 07:50 AM, diego_pmc wrote:
I have an `EventManager` class written in C++ and exposed to Python. This is how I intended for it to be used from the Python side:
class Something: def __init__(self): EventManager.addEventHandler(FooEvent, self.onFooEvent) def __del__(self): EventManager.removeEventHandler(FooEvent, self.onFooEvent) def onFooEvent(self, event): pass
(The `add-` and `remove-` are exposed as static functions of `EventManager`.)
The problem with the above code is that the callbacks are captured inside `boost::python::object` instances; when I do `self.onFooEvent` these will increase the reference count of `self`, which will prevent it from being deleted, so the destructor never gets called, so the event handlers never get removed (except at the end of the application).
The code works well for functions that don't have a `self` argument (i.e. free or static functions). How should I capture Python function objects such that I won't increase their reference count? I only need a weak reference to the objects.
Are these cycles actually a problem in practice? Python does do garbage collection, so it might be that it knows about all these dependencies and just hasn't bothered to try to delete them because it doesn't need the memory yet. Anyhow, as the other reply suggested, one option is clearly weakref (in addition, look at http://docs.python.org/c-api/weakref.html for how to use those in C/C++; there's no special Boost.Python interface for them). Unfortunately, what could have been a better option - implementing the special functions that tell Python how to break cycles in your C++ classes (http://docs.python.org/c-api/gcsupport.html) - is pretty low level, and probably impossible with Boost.Python. Good luck! Jim Bosch
On Tue, Aug 2, 2011 at 3:00 AM, Nat Goodspeed <nat@lindenlab.com> wrote:
Unfortunately, I can't use `weakref` I already tried that, but the problem is that the weak references get deleted (and their value set to `None`) before `__del__` is called. So when `removeEventHandler` is called, it won't be able to find the weakref it's supposed to remove (because its value has been changed to `None`). To fix, this would require some special handling of Python function objects in the `EventManager` — I don't want that, I wish to keep the manager agnostic to Python since I'm going to be pushing events into it from both C++ and Python. I'd like to solve this problem exclusively through the `EventManager` wrapper class I wrote. On Tue, Aug 2, 2011 at 3:24 AM, Jim Bosch-2 [via Boost] < ml-node+3711102-1593261807-256468@n4.nabble.com> wrote:
Are these cycles actually a problem in practice? Python does do garbage collection, so it might be that it knows about all these dependencies and just hasn't bothered to try to delete them because it doesn't need the memory yet.
Yes, they are problematic. When the object gets removed it should not receive any more events or it will very likely result in some odd behavior on the screen. These objects are entity states in a game. A state dictates how an entity reacts to game events and when the entity changes its state, the old one should stop giving the entity instructions, or they will conflict with the instructions given by the new state. -- View this message in context: http://boost.2283326.n4.nabble.com/Boost-Python-Callbacks-to-class-functions... Sent from the Python - c++-sig mailing list archive at Nabble.com.
On 08/01/2011 09:44 PM, diego_pmc wrote:
Are these cycles actually a problem in practice? Python does do garbage collection, so it might be that it knows about all these dependencies and just hasn't bothered to try to delete them because it doesn't need the memory yet.
Yes, they are problematic. When the object gets removed it should not receive any more events or it will very likely result in some odd behavior on the screen. These objects are entity states in a game. A state dictates how an entity reacts to game events and when the entity changes its state, the old one should stop giving the entity instructions, or they will conflict with the instructions given by the new state.
Hmm. That might mean you need to do a big design change; while it often works, one really isn't supposed to rely on __del__ being called when a Python object first could be garbage-collected - when cycles are involved, Python doesn't even guarantee that it will ever call __del__. It sounds like you'd be much better off with a named destructor-like method that would be called explicitly when you want to remove an object from the game. Jim
On Tue, Aug 2, 2011 at 8:24 AM, Jim Bosch <talljimbo@gmail.com> wrote:
Hmm. That might mean you need to do a big design change; while it often works, one really isn't supposed to rely on __del__ being called when a Python object first could be garbage-collected - when cycles are involved, Python doesn't even guarantee that it will ever call __del__. It sounds like you'd be much better off with a named destructor-like method that would be called explicitly when you want to remove an object from the game.
I am aware of that. :) Still, I'd like to leave it only as a last resort, I don't like complicating the API (who does? :P ). If I get rid of the strong reference that the `EventManager` has over the callback, there will be no cycles. My dependency graph would look something like this: ► http://i.imgur.com/wkxUn.jpg
participants (4)
-
diego_pmc -
Jim Bosch -
Nat Goodspeed -
Paul-Cristian Manta