A way to tell the gc it's OK to delete an object with a __del__
I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works. In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method. The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work. Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete. However -- some __del__ methods are perfectly safe regardless of delete order. So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time. Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway. Is there a technical complication I'm not thinking of? It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
2014-11-13 19:18 GMT+01:00 Chris Barker <chris.barker@noaa.gov>:
Did you try Python 3.4? https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization -- Amaury Forgeot d'Arc
On Thu, Nov 13, 2014 at 10:48 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...) I hadn't noticed that in any of the py3.4 announcements (nor had I looked hard), but defiantly didn't find in in any of my googling about this issue. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker <chris.barker@noaa.gov> wrote:
This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...)
Well, if you're coming here with language improvement suggestions, you'll definitely want to be on Py3. There won't be any features added to Py2. :) ChrisA
On Thu, Nov 13, 2014 at 3:52 PM, Chris Angelico <rosuav@gmail.com> wrote:
yes, I'm fully aware of that -- still the the real kick to get me there -- maybe this is it. I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker <chris.barker@noaa.gov> wrote:
The next minor version of Py3 is 3.5, which is currently scheduled for a first alpha in February. If the feature misses that, the next minor release will be I think about 18 months later, give or take. The next minor version of Py2 is 2.8 and will not be happening. http://legacy.python.org/dev/peps/pep-0404/ ChrisA
On 2014-11-13 18:18, Chris Barker wrote:
There's some code in this that you might find useful: http://permalink.gmane.org/gmane.comp.python.ideas/20334
On Thu, Nov 13, 2014 at 10:50 AM, MRAB <python@mrabarnett.plus.com> wrote:
There's some code in this that you might find useful:
thanks -- that does look kind of like what I was thinking I'd need to do -- but in the long run what the gc to do it for me (which maybe it does in py3.4) But maybe I'll patch the netCDF lib with something like that -- folks will be using py < 3.4 for a long time yet :-( -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 14 November 2014 09:20, Chris Barker <chris.barker@noaa.gov> wrote:
PyPy's GC design is (very) different, and allows finalisation of reference cycles involving __del__ even in Python 2. That path naturally comes with its own C extension compatibility challenges, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Nov 13, 2014 at 11:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
yup -- as does IronPython and Jython. But particularly for the scientific stack, switching to PyPy is a LOT harder than switching to 3.4 ! I certainly wouldn't do it for the garbage collection ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
2014-11-13 19:18 GMT+01:00 Chris Barker <chris.barker@noaa.gov>:
Did you try Python 3.4? https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization -- Amaury Forgeot d'Arc
On Thu, Nov 13, 2014 at 10:48 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...) I hadn't noticed that in any of the py3.4 announcements (nor had I looked hard), but defiantly didn't find in in any of my googling about this issue. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker <chris.barker@noaa.gov> wrote:
This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...)
Well, if you're coming here with language improvement suggestions, you'll definitely want to be on Py3. There won't be any features added to Py2. :) ChrisA
On Thu, Nov 13, 2014 at 3:52 PM, Chris Angelico <rosuav@gmail.com> wrote:
yes, I'm fully aware of that -- still the the real kick to get me there -- maybe this is it. I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker <chris.barker@noaa.gov> wrote:
The next minor version of Py3 is 3.5, which is currently scheduled for a first alpha in February. If the feature misses that, the next minor release will be I think about 18 months later, give or take. The next minor version of Py2 is 2.8 and will not be happening. http://legacy.python.org/dev/peps/pep-0404/ ChrisA
On 2014-11-13 18:18, Chris Barker wrote:
There's some code in this that you might find useful: http://permalink.gmane.org/gmane.comp.python.ideas/20334
On Thu, Nov 13, 2014 at 10:50 AM, MRAB <python@mrabarnett.plus.com> wrote:
There's some code in this that you might find useful:
thanks -- that does look kind of like what I was thinking I'd need to do -- but in the long run what the gc to do it for me (which maybe it does in py3.4) But maybe I'll patch the netCDF lib with something like that -- folks will be using py < 3.4 for a long time yet :-( -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 14 November 2014 09:20, Chris Barker <chris.barker@noaa.gov> wrote:
PyPy's GC design is (very) different, and allows finalisation of reference cycles involving __del__ even in Python 2. That path naturally comes with its own C extension compatibility challenges, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Nov 13, 2014 at 11:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
yup -- as does IronPython and Jython. But particularly for the scientific stack, switching to PyPy is a LOT harder than switching to 3.4 ! I certainly wouldn't do it for the garbage collection ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (6)
-
Amaury Forgeot d'Arc -
Chris Angelico -
Chris Barker -
MRAB -
Nick Coghlan -
Terry Reedy