On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov wrote:

__del__ is very useful not for interpreter shutdown but as a regular destructor in object's lifecycle.

The reason we should warn people against ever implementing __del__ is that people rarely actually understand object lifecycle. Its presence guarantees delayed garbage collection and that its code will code execute in a context way outside of normal control flow that all other methods are invoked from. -gps

Action on the shutdown is another beast. Personally, I prefer to do all finalization works by explicit calls instead.

On Wed, Jan 1, 2020 at 2:39 AM Yonatan Zunger wrote:

...

Hey everyone,

I just encountered yet another reason to beware of __del__: when it's

called during interpreter shutdown, for reasons which are kind of obvious in retrospect, if it calls notify() on a threading.Condition, the waiting thread may or may not ever actually receive it, and so if it does that and then tries to join() the thread the interpreter may hang in a hard-to-debug way.

...

This isn't something that can reasonably be fixed, and (like in most

cases involving __del__) there's a very simple fix of using weakref.finalize instead. My question for the dev list: How would people feel about changing the documentation for the method to more bluntly warn people against using it, and refer them to weakref.finalize and/or atexit.register as an alternative? The text already has an undertone of "lasciate ogni speranza, voi ch'entrate" but it may be helpful to be more explicit to avoid people getting caught in unexpected pitfalls.

...

Yonatan _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/AAZQBWD6...

...

Code of Conduct: http://python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/57CDW4NI... Code of Conduct: http://python.org/psf/codeofconduct/

Oh, I'm absolutely thinking about clarity. Something like: ---- This method is called when the instance is about to be destroyed. Because it may be called by either the ordinary Python execution flow or by the garbage collector, it has very unusual semantics, and must be treated with great care. In almost all cases, you should instead use either weakref.finalize (if you want logic to be called when an object is no longer in use) or atexit.register (if you want to clean up before the interpreter exits). The precise semantics of when __del__ is called on an object are implementation-dependent. For example: * It might be invoked during the normal interpreter flow at a moment like function return, if the reference count for an object has dropped to zero; * It might be invoked by the garbage collector, which happens during the normal interpreter flow but is called from an arbitrary thread; * It might be invoked during interpreter shutdown, after various global variables (including modules!) have already been deleted, or other threads' interpreters have already stopped. * It may not be invoked at all; it is *not guaranteed* that __del__() methods are called for objects that still exist when the interpreter exits. In particular, this means that, unless your program is structured so that you can deterministically control when an object will be destroyed, implementations of __del__(): * Must not assume that any variables outside self exist, including modules or global variables; * Must not attempt to acquire locks or otherwise block, since they may (e.g.) be invoked inside a thread which already holds those resources; * Must not rely on being able to signal other threads and wait for their responses (even thread joins), since those threads' interpreters may have exited; * Must not cause failures (e.g. leaving external devices in an invalid state) if they are *never* invoked. Note that del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero. Depending on the implementation, it is possible for a reference cycle to prevent the reference count of an object from going to zero. (e.g., in CPython, a common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback.) In this case, the cycle will be later detected and deleted by the cyclic garbage collector. If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance. It is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection. It is implementation-dependent whether __del__() is called a second time when a resurrected object is about to be destroyed; the current CPython implementation only calls it once. On Wed, Jan 1, 2020 at 12:57 PM Andrew Svetlov wrote:

If the warning text tells about the delayed execution -- I'm +1. -1 for something like "just don't use __del__, it is dangerous".

On Wed, Jan 1, 2020, 21:51 Gregory P. Smith wrote:

...

On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov wrote:

...

__del__ is very useful not for interpreter shutdown but as a regular destructor in object's lifecycle.

The reason we should warn people against ever implementing __del__ is that people rarely actually understand object lifecycle. Its presence guarantees delayed garbage collection and that its code will code execute in a context way outside of normal control flow that all other methods are invoked from.

-gps

...

Action on the shutdown is another beast. Personally, I prefer to do all finalization works by explicit calls instead.

On Wed, Jan 1, 2020 at 2:39 AM Yonatan Zunger wrote:

...

Hey everyone,

I just encountered yet another reason to beware of __del__: when it's

called during interpreter shutdown, for reasons which are kind of obvious in retrospect, if it calls notify() on a threading.Condition, the waiting thread may or may not ever actually receive it, and so if it does that and then tries to join() the thread the interpreter may hang in a hard-to-debug way.

...

This isn't something that can reasonably be fixed, and (like in most

cases involving __del__) there's a very simple fix of using weakref.finalize instead. My question for the dev list: How would people feel about changing the documentation for the method to more bluntly warn people against using it, and refer them to weakref.finalize and/or atexit.register as an alternative? The text already has an undertone of "lasciate ogni speranza, voi ch'entrate" but it may be helpful to be more explicit to avoid people getting caught in unexpected pitfalls.

...

Yonatan _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/AAZQBWD6...

...

Code of Conduct: http://python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/57CDW4NI... Code of Conduct: http://python.org/psf/codeofconduct/

I would say that the "recommended" weakref.finalize() shares very many limitations of __del__(), that's why hard to buy it. atexit.register() is not a common thing, the recommendation of using atexit for file descriptor closing *in general* looks weird, while it can be a good solution in some particular case. On Thu, Jan 2, 2020 at 1:05 PM Armin Rigo wrote:

Hi,

On Thu, 2 Jan 2020 at 03:59, Yonatan Zunger wrote:

...

It is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection. It is implementation-dependent whether __del__() is called a second time when a resurrected object is about to be destroyed; the current CPython implementation only calls it once.

"...in most cases."

Armin Rigo

-- Thanks, Andrew Svetlov

On Fri., 3 Jan. 2020, 3:52 am Yonatan Zunger, wrote:

weakref.finalize always executes during the ordinary Python flow, IIUC -- e.g., it happens before the interpreter is stopping. I guess it does still have the "arbitrary thread" limitation -- which is a good point, I may have some code bugs of my own to fix. But it's a huge difference in terms of predictability of the interpreter state, and in particular both the issues around deleted modules and the ones around threading no longer working don't apply to it.

atexit is a pretty corner-case thing -- I agree that it shouldn't be the common thing at all, and the recommendation should be specifically for those use cases where it makes sense.

And in the context of this particular discussion, has the limitation that atexit handlers run *after* the interpreter waits for non-daemon threads to stop. Cheers, Nick.

...

Jeff, I like all of your proposals. It does get a bit long, but I think it's not so long that it feels out-of-place in the data model docs, which is where you expect weird stuff to live. (I mean, look how long the discussion of metaclasses is -- and I certainly wouldn't want to eliminate that!) On Fri, Jan 3, 2020 at 3:56 AM Jeff Allen wrote:

On 02/01/2020 02:53, Yonatan Zunger wrote:

Oh, I'm absolutely thinking about clarity. ...

Could any revision also be clear what is *required of Python the language* vs. what is a CPython implementation detail? I always appreciate this care. There is good practice here and elsewhere in the existing documentation, but drift is easy for those steeped in CPython implementation. In the present case, it's a matter of avoiding an explicit requirement for a reference-counting approach to lifecycle management. Here the places in the proposal a change could achieve that.

The precise semantics of when __del__ is called on an object are implementation-dependent. For example: * It might be invoked during the normal interpreter flow at a moment like function return, ...

We should continue here "... immediately the object is no nonger referenced;"

(It might not be called immediately, but that's implied by your implementation-dependent "might be".)

Note that del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero. Depending on the implementation, it is possible for a reference cycle to prevent the reference count of an object from going to zero. (e.g., in CPython, a common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback.) In this case, the cycle will be later detected and deleted by the cyclic garbage collector.

I realise that most of this paragraph is existing text rearranged, and currently it fails to make the distinction I'm looking for in the "note" part. But it is clear in the next paragraph. I think it better to say, closer to the current text:

"""Note:: ``del x`` does not call ``x.__del__()`` directly. After ``del x``, variable ``x`` is undefined (or unbound). If the object it referenced is now no longer referenced at all, that object's ``__del__()`` might be called, immediately or later, subject to the caveats already given.

*CPython implementation detail:* ``del x`` decrements the reference count of the object by one, and if that makes it zero, ``x.__del__()`` will be called immediately. It is possible for a reference cycle to prevent the reference count of any object in it from going to zero. A common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback. In this case, the cycle will be detected later and its objects deleted by the cyclic garbage collector."""

If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance.

Possibly this thought belings with the "implementations of __del__(): * Must ..." paragraph.

But also, while I think there is scope for a better guidance, this is getting a bit long. Should there be a "HOW TO write a __del__ method (and how to avoid it)" to contain the advisory points being made? In-lining advice here, on how to survive the infernal circles of __del__, dilutes the scariness of the warning not to enter at all.

---

Jeff Allen

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/A32QEUP4... Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, 1 Jan 2020 11:51:41 -0800 "Gregory P. Smith" wrote:

On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov wrote:

...

__del__ is very useful not for interpreter shutdown but as a regular destructor in object's lifecycle.

The reason we should warn people against ever implementing __del__ is that people rarely actually understand object lifecycle. Its presence guarantees delayed garbage collection and that its code will code execute in a context way outside of normal control flow that all other methods are invoked from.

For the latter (executes in an arbitrary context): agreed. For the former (guarantees delayed garbage collection): what do you mean? Regardless, +1 for pointing to `weakref.finalize` as a better alternative for things that may survive until interpreter shutdown. Most people don't know about this kind of detail, and it can bite hard when working on complex systems. Regards Antoine.

On Sun, 05 Jan 2020 12:37:24 +1300 Greg Ewing wrote:

On 5/01/20 11:41 am, Antoine Pitrou wrote:

...

+1 for pointing to `weakref.finalize` as a better alternative for things that may survive until interpreter shutdown.

I'm not sure that weakref.finalize is much better. It can still be called at an arbitrary time from an arbitrary thread with the rest of the system in an unknown state, so it needs to be written very defensively to cope with that.

I didn't say otherwise. The main thing going for `weakref.finalize` is that it is called at the beginning of the interpreter shutdown while __del__ may be called much later in the process.

I don't think a blanket warning against using __del__ would be very helpful, because many of its pitfalls also apply to the suggested alternatives. This is an area where there's no substitute for an in-depth understanding of the issues.

I didn't suggest a blanket warning either. I do believe there should be a discussion, somewhere in the docs, of the various options available for finalization and their respective pitfalls. However, many (most?) Python developers know about __del__, while few know about `weakref.finalize`, so it makes sense to add a pointer to the latter in the docs for the former. Regards Antoine.

On Sat, Jan 4, 2020 at 5:43 AM Inada Naoki wrote:

I don't want to recommend atexit or weakref because they have their pitfall.

I think the context manager should be considered at first when people want to use __del__.

I support this proposal :)

-- Inada Naoki _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RSG6DHNG... Code of Conduct: http://python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov