Better support for finalization with weakrefs
I would like to see better support for the use of weakrefs callbacks for object finalization. The current issues with weakref callbacks include: 1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles). When it is an option, using a __del__ method is far less hassle. 2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None. 3. Sometimes you want the callback to be called at program exit even if the referent is still alive. 4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?) (Note that problems 2-4 also apply to using __del__ methods.) If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be >>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny! Prototype at https://gist.github.com/3208245 Cheers, Richard
Hi, On Mon, 30 Jul 2012 17:44:09 +0100 Richard Oudkerk <shibturn@gmail.com> wrote:
2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None.
A solution was proposed a long time ago at http://bugs.python.org/issue812369
4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?)
No, it's just because PyErr_WriteUnraisable() doesn't print the traceback. We could just fix that.
(Note that problems 2-4 also apply to using __del__ methods.)
If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be
>>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny!
Looks good! Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
On Mon, Jul 30, 2012 at 12:44 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
I would like to see better support for the use of weakrefs callbacks for object finalization.
The current issues with weakref callbacks include:
1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles).
When it is an option, using a __del__ method is far less hassle.
2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None.
3. Sometimes you want the callback to be called at program exit even if the referent is still alive.
4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?)
(Note that problems 2-4 also apply to using __del__ methods.)
If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be
>>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny!
Prototype at https://gist.github.com/3208245
I like how simple it is, but it might be too simple. That is, shouldn't we have a way to unregister the callback?
Cheers,
Richard
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
On Tue, Jul 31, 2012 at 9:28 AM, Calvin Spealman <ironfroggy@gmail.com>wrote:
On Mon, Jul 30, 2012 at 12:44 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
I would like to see better support for the use of weakrefs callbacks for object finalization.
The current issues with weakref callbacks include:
1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles).
When it is an option, using a __del__ method is far less hassle.
2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None.
3. Sometimes you want the callback to be called at program exit even if the referent is still alive.
4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?)
(Note that problems 2-4 also apply to using __del__ methods.)
If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be
>>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny!
Prototype at https://gist.github.com/3208245
I like how simple it is, but it might be too simple. That is, shouldn't we have a way to unregister the callback?
You can't unregister a __del__ method, so why should that stop this from moving forward? And if you really need that then you just have the cleanup code check something to see if it should run or not or just use some form of delegation.
If finalizations are objects, is there any reason you can't make the finalization object have an unregister method, like below? Or does this not address the problem? >>> class Kenny: pass ... >>> kenny = Kenny() >>> f = finalize(kenny, print, "you killed kenny!") >>> f <finalize.finalize object at 0xffed4f2c> >>> f.unregister() >>> del kenny >>> Matthew Lefavor From: Brett Cannon <brett@python.org<mailto:brett@python.org>> Date: Wednesday, August 1, 2012 11:25 AM To: "ironfroggy@gmail.com<mailto:ironfroggy@gmail.com>" <ironfroggy@gmail.com<mailto:ironfroggy@gmail.com>> Cc: Richard Oudkerk <shibturn@gmail.com<mailto:shibturn@gmail.com>>, "python-ideas@python.org<mailto:python-ideas@python.org>" <python-ideas@python.org<mailto:python-ideas@python.org>> Subject: Re: [Python-ideas] Better support for finalization with weakrefs On Tue, Jul 31, 2012 at 9:28 AM, Calvin Spealman <ironfroggy@gmail.com<mailto:ironfroggy@gmail.com>> wrote: On Mon, Jul 30, 2012 at 12:44 PM, Richard Oudkerk <shibturn@gmail.com<mailto:shibturn@gmail.com>> wrote:
I would like to see better support for the use of weakrefs callbacks for object finalization.
The current issues with weakref callbacks include:
1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles).
When it is an option, using a __del__ method is far less hassle.
2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None.
3. Sometimes you want the callback to be called at program exit even if the referent is still alive.
4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?)
(Note that problems 2-4 also apply to using __del__ methods.)
If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be
>>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny!
Prototype at https://gist.github.com/3208245
I like how simple it is, but it might be too simple. That is, shouldn't we have a way to unregister the callback? You can't unregister a __del__ method, so why should that stop this from moving forward? And if you really need that then you just have the cleanup code check something to see if it should run or not or just use some form of delegation.
On 01/08/2012 4:41pm, Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC] wrote:
If finalizations are objects, is there any reason you can't make the finalization object have an unregister method, like below? Or does this not address the problem?
>>> class Kenny: pass ... >>> kenny = Kenny() >>> f = finalize(kenny, print, "you killed kenny!") >>> f <finalize.finalize object at 0xffed4f2c> >>> f.unregister() >>> del kenny >>>
Matthew Lefavor
I have a patch (with tests and docs) at http://bugs.python.org/issue15528 It has a pop() method for unregistering the callback. So if f was created using f = finalize(obj, func, *args, **kwds) then f.pop() returns a tuple (wr, func, args, kwds) where wr is a weakref to obj. There is also a get() method which returns the same info, but does not unregister the callback. Once the finalizer is dead, f(), f.get(), f.pop() all return None. Richard
On Wed, Aug 1, 2012 at 12:27 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
On 01/08/2012 4:41pm, Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC] wrote:
If finalizations are objects, is there any reason you can't make the finalization object have an unregister method, like below? Or does this not address the problem?
>>> class Kenny: pass ... >>> kenny = Kenny() >>> f = finalize(kenny, print, "you killed kenny!") >>> f <finalize.finalize object at 0xffed4f2c> >>> f.unregister() >>> del kenny >>>
Matthew Lefavor
I have a patch (with tests and docs) at
http://bugs.python.org/issue15528
It has a pop() method for unregistering the callback. So if f was created using
f = finalize(obj, func, *args, **kwds)
then f.pop() returns a tuple
(wr, func, args, kwds)
where wr is a weakref to obj.
There is also a get() method which returns the same info, but does not unregister the callback.
Once the finalizer is dead, f(), f.get(), f.pop() all return None.
I don't like reusing the method names of containers, which this isn't. I liked "peek" and "detach". Also, why not fully hide the weakref adn return the actual object when you call one fo them, but have them fail if the object no longer exists? Or, what if they were just properties that looked up the weakref'ed object on access but you could just get at the callback and args directly?
Richard
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
On 01/08/2012 9:56pm, Calvin Spealman wrote:
I don't like reusing the method names of containers, which this isn't. I liked "peek" and "detach".
Yeah, maybe so.
Also, why not fully hide the weakref adn return the actual object when you call one fo them, but have them fail if the object no longer exists?
I did consider that but worried about the "race" where the object is garbage collected just after you pop from the registry. Thinking about it again, you can just try resolving the weakref before popping from the registry.
Or, what if they were just properties that looked up the weakref'ed object on access but you could just get at the callback and args directly?
The finalizer itself is just a light weight dictionary key. It should not own references to anything non-trivial. Otherwise storing it could keep alive big objects (or cause ref-cycles) even after the finalizer is dead. But all the information could be available from properties which return None when the finalizer is dead. Richard
participants (5)
-
Antoine Pitrou -
Brett Cannon -
Calvin Spealman -
Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC] -
Richard Oudkerk