Hi there. Some months back I mentioned that I had other stuff in store for GC. Here is an idea for you. This particular idea is just a generalization of a system we've used in EVE for years now: garbage collection callbacks. The need, originally, was to be able to quantify the time spent in garbage collection runs. Since they occur out of direct control of the application, we needed to have python tell us about it somehow. We added gc.register_callback(), a function that added a callable to an internal list of functions that get called on two occations: 1) When a garbage collection run Is about to start 2) When a garbage collection run has finished. The cases are distinguished using an integer argument. The callbacks are invoked from gc with gc inhibited from reentry, so that the callbacks cannot themselves cause another gc run to commence. What we traditionally use this for is to start and stop a performance timer and other stats. More recently though, we have found another very important use for this. When gc finds uncollectable objects, they are put in the gc.garbage list. This then needs to be handled by the application. However, there is no particularly good way for the application to do this, except to periodically check this list. With the above callback, modules that create uncollectable objects, such as classes with __del__ methods, can register their callback. At the end of a gc run, they can then walk gc.garbage and take appropriate action for those objects it recognizes. So, what do you think? This is a very simple addition to gc, orthogonal to everything and easily implemented. I also think it is very useful. K
Sounds great. One could quibble over the details of the callback
signature, but I'll leave the bikeshedding to others. Do you have code
that could be a basis for a patch? Is it encumbered by copyrights?
--Guido
2010/11/28 Kristján Valur Jónsson
Hi there.
Some months back I mentioned that I had other stuff in store for GC.
Here is an idea for you. This particular idea is just a generalization of a system we’ve used in EVE for years now: garbage collection callbacks.
The need, originally, was to be able to quantify the time spent in garbage collection runs. Since they occur out of direct control of the application, we needed to have python tell us about it somehow.
We added gc.register_callback(), a function that added a callable to an internal list of functions that get called on two occations:
1) When a garbage collection run Is about to start
2) When a garbage collection run has finished.
The cases are distinguished using an integer argument. The callbacks are invoked from gc with gc inhibited from reentry, so that the callbacks cannot themselves cause another gc run to commence.
What we traditionally use this for is to start and stop a performance timer and other stats.
More recently though, we have found another very important use for this. When gc finds uncollectable objects, they are put in the gc.garbage list. This then needs to be handled by the application. However, there is no particularly good way for the application to do this, except to periodically check this list.
With the above callback, modules that create uncollectable objects, such as classes with __del__ methods, can register their callback. At the end of a gc run, they can then walk gc.garbage and take appropriate action for those objects it recognizes.
So, what do you think? This is a very simple addition to gc, orthogonal to everything and easily implemented. I also think it is very useful.
K
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
I'll submit our current code as a patch to the tracker. Like all our contributions it comes with no strings attached :)
K
-----Original Message-----
From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum
Sent: Monday, November 29, 2010 13:43
To: Kristján Valur Jónsson
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] gc callbacks
Sounds great. One could quibble over the details of the callback signature, but I'll leave the bikeshedding to others. Do you have code that could be a basis for a patch? Is it encumbered by copyrights?
--Guido
2010/11/28 Kristján Valur Jónsson
Hi there.
Some months back I mentioned that I had other stuff in store for GC.
Here is an idea for you. This particular idea is just a generalization of a system we've used in EVE for years now: garbage collection callbacks.
The need, originally, was to be able to quantify the time spent in garbage collection runs. Since they occur out of direct control of the application, we needed to have python tell us about it somehow.
We added gc.register_callback(), a function that added a callable to an internal list of functions that get called on two occations:
1) When a garbage collection run Is about to start
2) When a garbage collection run has finished.
The cases are distinguished using an integer argument. The callbacks are invoked from gc with gc inhibited from reentry, so that the callbacks cannot themselves cause another gc run to commence.
What we traditionally use this for is to start and stop a performance timer and other stats.
More recently though, we have found another very important use for this. When gc finds uncollectable objects, they are put in the gc.garbage list. This then needs to be handled by the application. However, there is no particularly good way for the application to do this, except to periodically check this list.
With the above callback, modules that create uncollectable objects, such as classes with __del__ methods, can register their callback. At the end of a gc run, they can then walk gc.garbage and take appropriate action for those objects it recognizes.
So, what do you think? This is a very simple addition to gc, orthogonal to everything and easily implemented. I also think it is very useful.
K
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
FYI:
http://bugs.python.org/issue10576
K
-----Original Message-----
From: python-ideas-bounces+kristjan=ccpgames.com@python.org [mailto:python-ideas-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Kristján Valur Jónsson
Sent: Monday, November 29, 2010 14:02
To: Guido van Rossum
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] gc callbacks
I'll submit our current code as a patch to the tracker. Like all our contributions it comes with no strings attached :) K
-----Original Message-----
From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum
Sent: Monday, November 29, 2010 13:43
To: Kristján Valur Jónsson
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] gc callbacks
Sounds great. One could quibble over the details of the callback signature, but I'll leave the bikeshedding to others. Do you have code that could be a basis for a patch? Is it encumbered by copyrights?
--Guido
2010/11/28 Kristján Valur Jónsson
Hi there.
Some months back I mentioned that I had other stuff in store for GC.
Here is an idea for you. This particular idea is just a generalization of a system we've used in EVE for years now: garbage collection callbacks.
The need, originally, was to be able to quantify the time spent in garbage collection runs. Since they occur out of direct control of the application, we needed to have python tell us about it somehow.
We added gc.register_callback(), a function that added a callable to an internal list of functions that get called on two occations:
1) When a garbage collection run Is about to start
2) When a garbage collection run has finished.
The cases are distinguished using an integer argument. The callbacks are invoked from gc with gc inhibited from reentry, so that the callbacks cannot themselves cause another gc run to commence.
What we traditionally use this for is to start and stop a performance timer and other stats.
More recently though, we have found another very important use for this. When gc finds uncollectable objects, they are put in the gc.garbage list. This then needs to be handled by the application. However, there is no particularly good way for the application to do this, except to periodically check this list.
With the above callback, modules that create uncollectable objects, such as classes with __del__ methods, can register their callback. At the end of a gc run, they can then walk gc.garbage and take appropriate action for those objects it recognizes.
So, what do you think? This is a very simple addition to gc, orthogonal to everything and easily implemented. I also think it is very useful.
K
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 11/29/2010 12:12 AM, Kristján Valur Jónsson wrote:
Hi there.
Some months back I mentioned that I had other stuff in store for GC.
Here is an idea for you. This particular idea is just a generalization of a system we’ve used in EVE for years now: garbage collection callbacks.
The need, originally, was to be able to quantify the time spent in garbage collection runs. Since they occur out of direct control of the application, we needed to have python tell us about it somehow.
We added gc.register_callback(), a function that added a callable to an internal list of functions that get called on two occations:
1)When a garbage collection run Is about to start
2)When a garbage collection run has finished.
The cases are distinguished using an integer argument. The callbacks are invoked from gc with gc inhibited from reentry, so that the callbacks cannot themselves cause another gc run to commence.
What we traditionally use this for is to start and stop a performance timer and other stats.
More recently though, we have found another very important use for this. When gc finds uncollectable objects, they are put in the gc.garbage list. This then needs to be handled by the application. However, there is no particularly good way for the application to do this, except to periodically check this list.
With the above callback, modules that create uncollectable objects, such as classes with __del__ methods, can register their callback. At the end of a gc run, they can then walk gc.garbage and take appropriate action for those objects it recognizes.
So, what do you think? This is a very simple addition to gc, orthogonal to everything and easily implemented. I also think it is very useful.
I presume the slowdown in the normal case consists of a very quick 'if callback list is not empty' check. If others agree and you open an issue with a ready-to-go patch (code, doc, test, what's-new), getting in 3.2b1 might be possible. -- Terry Jan Reedy
participants (3)
-
Guido van Rossum -
Kristján Valur Jónsson -
Terry Reedy