Mailman 3 A way to tell the gc it's OK to delete an object with a __del__ - Python-ideas

A way to tell the gc it's OK to delete an object with a del

older
including psutil in the standard...

Chris Barker

13 Nov 2014 13 Nov '14

7:18 p.m.

I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works. In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method. The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work. Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete. However -- some __del__ methods are perfectly safe regardless of delete order. So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time. Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway. Is there a technical complication I'm not thinking of? It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Attachments:

attachment.htm (text/html — 2.5 KB)

Show replies by date

Amaury Forgeot d'Arc

13 Nov 13 Nov

7:48 p.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

2014-11-13 19:18 GMT+01:00 Chris Barker :

...

I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works.

In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method.

The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work.

Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete.

However -- some __del__ methods are perfectly safe regardless of delete order.

So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time.

Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway.

Is there a technical complication I'm not thinking of?

It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management.

Did you try Python 3.4? https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization -- Amaury Forgeot d'Arc

Chris Barker

14 Nov 14 Nov

12:18 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Thu, Nov 13, 2014 at 10:48 AM, Amaury Forgeot d'Arc wrote:

...

So it would be nice to be able to somehow let the gc know that a

...
particular object is safe to delete at any time.

Did you try Python 3.4?

https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization

no, I haven't -- very cool!

This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...) I hadn't noticed that in any of the py3.4 announcements (nor had I looked hard), but defiantly didn't find in in any of my googling about this issue. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Chris Angelico

12:52 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker wrote:

...

This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...)

Well, if you're coming here with language improvement suggestions, you'll definitely want to be on Py3. There won't be any features added to Py2. :) ChrisA

Chris Barker

1:58 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Thu, Nov 13, 2014 at 3:52 PM, Chris Angelico wrote:

...

On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker wrote:

...
This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...)

Well, if you're coming here with language improvement suggestions, you'll definitely want to be on Py3. There won't be any features added to Py2. :)

yes, I'm fully aware of that -- still the the real kick to get me there -- maybe this is it. I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally. -CHB

...

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Chris Angelico

2:25 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker wrote:

...

I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally.

The next minor version of Py3 is 3.5, which is currently scheduled for a first alpha in February. If the feature misses that, the next minor release will be I think about 18 months later, give or take. The next minor version of Py2 is 2.8 and will not be happening. http://legacy.python.org/dev/peps/pep-0404/ ChrisA

Terry Reedy

6:23 a.m.

On 11/13/2014 8:25 PM, Chris Angelico wrote:

...

On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker wrote:

...
I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally.

The next minor version of Py3 is 3.5, which is currently scheduled for a first alpha in February. If the feature misses that, the next minor release will be I think about 18 months later, give or take.

Beta1, scheduled for late May, is the new feature deadline, though sooner is better. -- Terry Jan Reedy

MRAB

13 Nov 13 Nov

7:50 p.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On 2014-11-13 18:18, Chris Barker wrote:

...

I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works.

In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method.

The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work.

Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete.

However -- some __del__ methods are perfectly safe regardless of delete order.

So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time.

Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway.

Is there a technical complication I'm not thinking of?

It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management.

There's some code in this that you might find useful: http://permalink.gmane.org/gmane.comp.python.ideas/20334

Chris Barker

14 Nov 14 Nov

12:20 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Thu, Nov 13, 2014 at 10:50 AM, MRAB wrote:

...

There's some code in this that you might find useful:

http://permalink.gmane.org/gmane.comp.python.ideas/20334

thanks -- that does look kind of like what I was thinking I'd need to do -- but in the long run what the gc to do it for me (which maybe it does in py3.4) But maybe I'll patch the netCDF lib with something like that -- folks will be using py < 3.4 for a long time yet :-( -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Nick Coghlan

8:45 a.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On 14 November 2014 09:20, Chris Barker wrote:

...

On Thu, Nov 13, 2014 at 10:50 AM, MRAB wrote:

...
There's some code in this that you might find useful:

http://permalink.gmane.org/gmane.comp.python.ideas/20334

thanks -- that does look kind of like what I was thinking I'd need to do -- but in the long run what the gc to do it for me (which maybe it does in py3.4)

But maybe I'll patch the netCDF lib with something like that -- folks will be using py < 3.4 for a long time yet :-(

PyPy's GC design is (very) different, and allows finalisation of reference cycles involving __del__ even in Python 2. That path naturally comes with its own C extension compatibility challenges, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Chris Barker

7:01 p.m.

New subject: A way to tell the gc it's OK to delete an object with a __del__

On Thu, Nov 13, 2014 at 11:45 PM, Nick Coghlan wrote:

...

PyPy's GC design is (very) different, and allows finalisation of reference cycles involving __del__ even in Python 2. That path naturally comes with its own C extension compatibility challenges, though.

yup -- as does IronPython and Jython. But particularly for the scientific stack, switching to PyPy is a LOT harder than switching to 3.4 ! I certainly wouldn't do it for the garbage collection ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

3449

Age (days ago)

3450

Last active (days ago)

List overview

Download

10 comments

6 participants

participants (6)

Amaury Forgeot d'Arc
Chris Angelico
Chris Barker
MRAB
Nick Coghlan
Terry Reedy

A way to tell the gc it's OK to delete an object with a __del__

tags

participants (6)

A way to tell the gc it's OK to delete an object with a del