Design question: call __del__ only after successful __init__?
I was looking at the code that invokes __del__, with the intent to implement a feature from Java: in Java, a finalizer is only called once per object, even if calling it makes the object live longer. To implement this, we need a flag in each instance that means "__del__ was called". I opened the creation code for instances, looking for the right place to set the flag. I then realized that it might be smart, now that we have this flag anyway, to set it to "true" during initialization. There are a number of exits from the initialization where the object is created but not fully initialized, where the new object is DECREF'ed and NULL is returned. When such an exit is taken, __del__ is called on an incompletely initialized object! Example: >>> class C: def __del__(self): print "deleting", self >>> x = C(1) !--> deleting <__main__.C instance at 1686d8> Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: this constructor takes no arguments >>> Now I have a choice to make. If the class has an __init__, should I clear the flag only after __init__ succeeds? This means that if __init__ raises an exception, __del__ is never called. This is an incompatibility. It's possible that someone has written code that relies on __del__ being called even when __init__ fails halfway, and then their code would break. But it is just as likely that calling __del__ on a partially uninitialized object is a bad mistake, and I am doing all these cases a favor by not calling __del__ when __init__ failed! Any opinions? If nobody speaks up, I'll make the change. --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Now I have a choice to make. If the class has an __init__, GvR> should I clear the flag only after __init__ succeeds? This GvR> means that if __init__ raises an exception, __del__ is never GvR> called. This is an incompatibility. It's possible that GvR> someone has written code that relies on __del__ being called GvR> even when __init__ fails halfway, and then their code would GvR> break. It reminds me of the separation between object allocation and initialization in ObjC. GvR> But it is just as likely that calling __del__ on a partially GvR> uninitialized object is a bad mistake, and I am doing all GvR> these cases a favor by not calling __del__ when __init__ GvR> failed! GvR> Any opinions? If nobody speaks up, I'll make the change. I think you should set the flag right before you call __init__(), i.e. after (nearly all) the C level initialization has occurred. Here's why: your "favor" can easily be accomplished by Python constructs in the __init__(): class MyBogo: def __init__(self): self.get_delified = 0 do_sumtin_exceptional() self.get_delified = 1 def __del__(self): if self.get_delified: ah_sweet_release() -Barry
[Barry]
It reminds me of the separation between object allocation and initialization in ObjC.
Is that good or bad?
But the other behavior (call __del__ even when __init__ fails) can also easily be accomplished in Python: class C: def __init__(self): try: ...stuff that may fail... except: self.__del__() raise def __del__(self): ...cleanup... I believe that in almost all cases the programmer would be happier if __del__ wasn't called when their __init__ fails. This makes it easier to write a __del__ that can assume that all the object's fields have been properly initialized. In my code, typically when __init__ fails, this is a symptom of a really bad bug (e.g. I just renamed one of __init__'s arguments and forgot to fix all references), and I don't care much about cleanup behavior. --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> But the other behavior (call __del__ even when __init__ GvR> fails) can also easily be accomplished in Python: It's a fair cop. GvR> I believe that in almost all cases the programmer would be GvR> happier if __del__ wasn't called when their __init__ fails. GvR> This makes it easier to write a __del__ that can assume that GvR> all the object's fields have been properly initialized. That's probably fine; I don't have strong feelings either way. -Barry P.S. Interesting what X-Oblique-Strategy was randomly inserted in this message (but I'm not sure which approach is more "explicit" :). -Barry
On Thu, 2 Mar 2000, Guido van Rossum wrote:
+1 on calling __del__ IFF __init__ completes successfully. Cheers, -g -- Greg Stein, http://www.lyra.org/
On Thu, 2 Mar 2000, Greg Stein wrote:
That would be my vote as well. What convinced me of this is the following: If it's up to the implementation of __del__ to deal with a problem that happened during initialization, you only know about the problem with very coarse granularity. It's a pain (or even impossible) to then rediscover the information you need to recover adequately. If on the other hand you deal with the problem in __init__, then you have much better control over what is happening, because you can position try/except blocks precisely where you need them to deal with specific potential problems. Each block can take care of its case appropriately, and re-raise if necessary. In general, it seems to me that what you want to do when __init__ runs afoul is going to be different from what you want to do to take care of object cleanup in __del__. So it doesn't belong there -- it belongs in an except: clause in __init__. Even though it's an incompatibility, i really think this is the right behaviour. -- ?!ng "To be human is to continually change. Your desire to remain as you are is what ultimately limits you." -- The Puppet Master, Ghost in the Shell
[Guido]
Why? That is, in what way is this an improvement over current behavior? Note that Java is a bit subtle: a finalizer is only called once by magic; explicit calls "don't count". The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer. I deal with possible exceptions in Python constructors the same way I do in C++ and Java: if there's a destructor, don't put anything in __init__ that may raise an uncaught exception. Anything dangerous is moved into a separate .reset() (or .clear() or ...) method. This works well in practice.
To implement this, we need a flag in each instance that means "__del__ was called".
At least <wink>.
I agree *that* isn't good. Taken on its own, though, it argues for adding an "instance construction completed" flag that __del__ later checks, as if its body were: if self.__instance_construction_completed: body That is, the problem you've identified here could be addressed directly.
I'd be in favor of fixing the actual problem; I don't understand the point to the rest of it, especially as it has the potential to break existing code and I don't see a compensating advantage (surely not compatibility w/ JPython -- JPython doesn't invoke __del__ methods at all by magic, right? or is that changing, and that's what's driving this?). too-much-magic-is-dizzying-ly y'rs - tim
"TP" == Tim Peters <tim_one@email.msn.com> writes:
TP> (surely not compatibility w/ JPython -- JPython doesn't invoke TP> __del__ methods at all by magic, right? or is that changing, TP> and that's what's driving this?). No, JPython doesn't invoke __del__ methods by magic, and I don't have any plans to change that. -Barry
[Tim]
Of course. Same in my proposal. But I wouldn't call it "by magic" -- just "on behalf of the garbage collector".
The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer.
I don't find the Java rules confusing. It seems quite useful that the GC promises to call the finalizer at most once -- this can simplify the finalizer logic. (Otherwise it may have to ask itself, "did I clean this already?" and leave notes for itself.) Explicit finalizer calls are always a mistake and thus "don't count" -- the response to that should in general be "don't do that" (unless you have particularly stupid callers -- or very fearful lawyers :-).
Sure, but the rule "if __init__ fails, __del__ won't be called" means that we don't have to program our __init__ or __del__ quite so defensively. Most people who design a __del__ probably assume that __init__ has run to completion. The typical scenario (which has happened to me! And I *implemented* the damn thing!) is this: __init__ opens a file and assigns it to an instance variable; __del__ closes the file. This is tested a few times and it works great. Now in production the file somehow unexpectedly fails to be openable. Sure, the programmer should've expected that, but she didn't. Now, at best, the failed __del__ creates an additional confusing error message on top of the traceback generated by IOError. At worst, the failed __del__ could wreck the original traceback. Note that I'm not proposing to change the C level behavior; when a Py<Object>_New() function is halfway its initialization and decides to bail out, it does a DECREF(self) and you bet that at this point the <object>_dealloc() function gets called (via self->ob_type->tp_dealloc). Occasionally I need to initialize certain fields to NULL so that the dealloc() function doesn't try to free memory that wasn't allocated. Often it's as simple as using XDECREF instead of DECREF in the dealloc() function (XDECREF is safe when the argument is NULL, DECREF dumps core, saving a load-and-test if you are sure its arg is a valid object).
Sure -- but I would argue that when __del__ returns, __instance_construction_completed should be reset to false, because the destruction (conceptually, at least) cancels out the construction!
JPython's a red herring here. I think that the proposed change probably *fixes* much morecode that is subtly wrong than it breaks code that is relying on __del__ being called after a partial __init__. All the rules relating to __del__ are confusing (e.g. what __del__ can expect to survive in its globals). Also note Ping's observation: | If it's up to the implementation of __del__ to deal with a problem | that happened during initialization, you only know about the problem | with very coarse granularity. It's a pain (or even impossible) to | then rediscover the information you need to recover adequately. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Tim]
Note that Java is a bit subtle: a finalizer is only called once by magic; explicit calls "don't count".
[Guido]
Of course. Same in my proposal.
OK -- that wasn't clear.
But I wouldn't call it "by magic" -- just "on behalf of the garbage collector".
Yup, magically called <wink>.
The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer.
I don't find the Java rules confusing.
"add up" == "taken as a whole"; include the Java spec's complex state machine for cleanup semantics, and the later complications added by three (four?) distinct flavors of weak reference, and I doubt 1 Java programmer in 1,000 actually understands the rules. This is why I'm wary of moving in the Java *direction* here. Note that Java programmers in past c.l.py threads have generally claimed Java's finalizers are so confusing & unpredictable they don't use them at all! Which, in the end, is probably a good idea in Python too <0.5 wink>.
It seems quite useful that the GC promises to call the finalizer at most once -- this can simplify the finalizer logic.
Granting that explicit calls are "use at your own risk", the only user-visible effect of "called only once" is in the presence of resurrection. Now in my Python experience, on the few occasions I've resurrected an object in __del__, *of course* I expected __del__ to get called again if the object is about to die again! Typical: def __del__(self): if oops_i_still_need_to_stay_alive: resurrect(self) else: # really going away release(self.critical_resource) Call __del__ only once, and code like this is busted bigtime. OTOH, had I written __del__ logic that relied on being called only once, switching the implementation to call it more than once would break *that* bigtime. Neither behavior is an obvious all-cases win to me, or even a plausibly most-cases win. But Python already took a stand on this & so I think you need a *good* reason to change semantics now.
This is (or can easily be made) a separate issue, & I agreed the first time this seems worth fixing (although if nobody has griped about it in a decade of use, it's hard to call it a major bug <wink>).
In the __del__ above (which is typical of the cases of resurrection I've seen), there is no such implication. Perhaps this is philosophical abuse of Python's intent, but if so it relied only on trusting its advertised semantics.
Yes, again, I have no argument against refusing to call __del__ unless __init__ succeeded. Going beyond that to a new "called at most once" rule is indeed going beyond that, *will* break reasonable old code, and holds no particular attraction that I can see (it trades making one kind of resurrection scenario easier at the cost of making other kinds harder). If there needs to be incompatible change here, curiously enough I'd be more in favor of making resurrection illegal period (which could *really* simplify gc's headaches).
All the rules relating to __del__ are confusing (e.g. what __del__ can expect to survive in its globals).
Problems unique to final shutdown don't seem relevant here.
Also note Ping's observation: ...
I can't agree with that yet another time without being quadruply redundant <wink>.
OK, so we're down to this one point: if __del__ resurrects the object, should __del__ be called again later? Additionally, should resurrection be made illegal? I can easily see how __del__ could *accidentally* resurrect the object as part of its normal cleanup -- e.g. you make a call to some other routine that helps with the cleanup, passing self as an argument, and this other routine keeps a helpful cache of the last argument for some reason. I don't see how we could forbid this type of resurrection. (What are you going to do? You can't raise an exception from instance_dealloc, since it is called from DECREF. You can't track down the reference and replace it with a None easily.) In this example, the helper routine will eventually delete the object from its cache, at which point it is truly deleted. It would be harmful, not helpful, if __del__ was called again at this point. Now, it is true that the current docs for __del__ imply that resurrection is possible. The intention of that note was to warn __del__ writers that in the case of accidental resurrection __del__ might be called again. The intention certainly wasn't to allow or encourage intentional resurrection. Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this. [Jack just finds a snag]
Yes, that's a problem. But there are other ways for the subclass to break the base class's invariant (e.g. it could override __del__ without calling the base class' __del__). So I think it's a red herring. In Python 3000, typechecked classes may declare invariants that are enforced by the inheritance mechanism; then we may need to keep track which base class constructors succeeded and only call corresponding destructors. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Yes and no :-) One example comes to mind: implementations of weak references, which manage weak object references themselves (as soon as __del__ is called the weak reference implementation takes over the object). Another example is that of free list like implementations which reduce object creation times by implementing smart object recycling, e.g. objects could keep allocated dictionaries alive or connections to databases open, etc. As for the second point: Calling __del__ again is certainly needed to keep application logic sane... after all, __del__ should be called whenever the refcount reaches 0 -- and that can happend more than once in the objects life-time if reanimation occurs.
I'd say this is an application logic error -- nothing that the mechanism itself can help with automagically. OTOH, turning multi calls to __del__ off, would make certain techniques impossible.
I don't think that docs are the right argument here ;-) It is simply the reference counting logic that plays its role: __del__ is called when refcount reaches 0, which usually means that the object is about to be garbage collected... unless the object is rereferenced by some other object and thus gets reanimated.
Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this.
BTW, I can't see what the original question has to do with this discussion ... calling __del__ only after successful __init__ is ok, IMHO, but what does this have to do with the way __del__ itself is implemented ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Guido van Rossum wrote:
[much stuff] Just a random note: What if we had a __del__ with zombie behavior? Assume an instance that is about to be destructed. Then __del__ is called via normal method lookup. What we want is to let this happen only once. Here the Zombie: After method lookup, place a dummy __del__ into the to-be-deleted instance dict, and we are sure that this does not harm. Kinda "yes its there, but a broken link ". The zombie always works by doing nothing. Makes some sense? ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Guido]
I give up on the latter, so it really is just one.
If this is something that happens easily, and current behavior is harmful, don't you think someone would have complained about it by now? That is, __del__ *is* "called again at this point" now, and has been for years & years. And if it happens easily, it *is* happening now, and in an unknown amount of existing code. (BTW, I doubt it happens at all <wink> -- people tend to write very simple __del__ methods, so far as I've ever seen)
Now, it is true that the current docs for __del__ imply that resurrection is possible.
"imply" is too weak. The Reference Manual's "3.3.1 Basic customization" flat-out says it's possible ("though not recommended"). The precise meaning of the word "may" in the following sentence is open to debate, though.
The intention of that note was to warn __del__ writers that in the case of accidental resurrection
Sorry, but I can't buy this: saying that *accidents* are "not recommended" is just too much of a stretch <wink/frown>.
__del__ might be called again.
That's a plausible reading of the following "may", but not the only one. I believe it's the one you intended, but it's not the meaning I took prior to this.
The intention certainly wasn't to allow or encourage intentional resurrection.
Well, I think it plainly says it's supported ("though not recommended"). I used it intentionally at KSR, and even recommended it on c.l.py in the dim past (in one of those "dark & useless" threads <wink>).
Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this.
Why would anyone tell you about something that *works*?! You rarely hear the good stuff, you know. I gave the typical pattern in the preceding msg. To flesh out the motivation more, you have some external resource that's very expensive to set up (in KSR's case, it was an IPC connection to a remote machine). Rights to use that resource are handed out in the form of an object. When a client is done using the resource, they *should* explicitly use the object's .release() method, but you can't rely on that. So the object's __del__ method looks like (for example): def __del__(self): # Code not shown to figure out whether to disconnect: the downside to # disconnecting is that it can cost a bundle to create a new connection. # If the whole app is shutting down, then of course we want to disconnect. # Or if a timestamp trace shows that we haven't been making good use of # all the open connections lately, we may want to disconnect too. if decided_to_disconnect: self.external_resource.disconnect() else: # keep the connection alive for reuse global_available_connection_objects.append(self) This is simple & effective, and it relies on both intentional resurrection and __del__ getting called repeatedly. I don't claim there's no other way to write it, just that there's *been* no problem doing this for a millennium <wink>. Note that MAL spontaneously sketched similar examples, although I can't say whether he's actually done stuff like this. Going back up a level, in another msg you finally admitted <wink> that you want "__del__ called only once" for the same reason Java wants it: because gc has no idea what to do when faced with finalizers in a trash cycle, and settles for an unprincipled scheme whose primary virtue is that "it doesn't blow up" -- and "__del__ called only once" happens to be convenient for that scheme. But toss such cycles back to the user to deal with at the Python level, and all those problems go away (along with the artificial need to change __del__). The user can break the cycles in an order that makes sense to the app (or they can let 'em leak! up to them). >>> print gc.get_cycle.__doc__ Return a list of objects comprising a single garbage cycle; [] if none. At least one of the objects has a finalizer, so Python can't determine the intended order of destruction. If you don't break the cycle, Python will neither run any finalizers for the contained objects nor reclaim their memory. If you do break the cycle, and dispose of the list, Python will follow its normal reference-counting rules for running finalizers and reclaiming memory. That this "won't blow up" either is just the least of its virtues <wink>. you-break-it-you-buy-it-ly y'rs - tim
Tim Peters wrote:
Not exactly this, but similar things in the weak reference implementation of mxProxy. The idea came from a different area: the C implementation of Python uses free lists a lot and these are basically implementations of the same idiom: save an allocated resource for reviving it at some later point. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[M.-A. Lemburg, on the resurrection/multiple-__del__ "idiom"]
Excellent analogy! Thanks. Now that you phrased it in this clarifying way, I recall that very much the same point was raised in the papers that resulted in the creation of guardians in Scheme. I don't know that anyone is actually using Python __del__ this way today (I am not), but you reminded me why I thought it was natural at one time <wink>. generally-__del__-aversive-now-except-in-c++-where-destructors-are- guaranteed-to-be-called-when-you-except-them-to-be-ly y'rs - tim
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Now I have a choice to make. If the class has an __init__, GvR> should I clear the flag only after __init__ succeeds? This GvR> means that if __init__ raises an exception, __del__ is never GvR> called. This is an incompatibility. It's possible that GvR> someone has written code that relies on __del__ being called GvR> even when __init__ fails halfway, and then their code would GvR> break. It reminds me of the separation between object allocation and initialization in ObjC. GvR> But it is just as likely that calling __del__ on a partially GvR> uninitialized object is a bad mistake, and I am doing all GvR> these cases a favor by not calling __del__ when __init__ GvR> failed! GvR> Any opinions? If nobody speaks up, I'll make the change. I think you should set the flag right before you call __init__(), i.e. after (nearly all) the C level initialization has occurred. Here's why: your "favor" can easily be accomplished by Python constructs in the __init__(): class MyBogo: def __init__(self): self.get_delified = 0 do_sumtin_exceptional() self.get_delified = 1 def __del__(self): if self.get_delified: ah_sweet_release() -Barry
[Barry]
It reminds me of the separation between object allocation and initialization in ObjC.
Is that good or bad?
But the other behavior (call __del__ even when __init__ fails) can also easily be accomplished in Python: class C: def __init__(self): try: ...stuff that may fail... except: self.__del__() raise def __del__(self): ...cleanup... I believe that in almost all cases the programmer would be happier if __del__ wasn't called when their __init__ fails. This makes it easier to write a __del__ that can assume that all the object's fields have been properly initialized. In my code, typically when __init__ fails, this is a symptom of a really bad bug (e.g. I just renamed one of __init__'s arguments and forgot to fix all references), and I don't care much about cleanup behavior. --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> But the other behavior (call __del__ even when __init__ GvR> fails) can also easily be accomplished in Python: It's a fair cop. GvR> I believe that in almost all cases the programmer would be GvR> happier if __del__ wasn't called when their __init__ fails. GvR> This makes it easier to write a __del__ that can assume that GvR> all the object's fields have been properly initialized. That's probably fine; I don't have strong feelings either way. -Barry P.S. Interesting what X-Oblique-Strategy was randomly inserted in this message (but I'm not sure which approach is more "explicit" :). -Barry
On Thu, 2 Mar 2000, Guido van Rossum wrote:
+1 on calling __del__ IFF __init__ completes successfully. Cheers, -g -- Greg Stein, http://www.lyra.org/
On Thu, 2 Mar 2000, Greg Stein wrote:
That would be my vote as well. What convinced me of this is the following: If it's up to the implementation of __del__ to deal with a problem that happened during initialization, you only know about the problem with very coarse granularity. It's a pain (or even impossible) to then rediscover the information you need to recover adequately. If on the other hand you deal with the problem in __init__, then you have much better control over what is happening, because you can position try/except blocks precisely where you need them to deal with specific potential problems. Each block can take care of its case appropriately, and re-raise if necessary. In general, it seems to me that what you want to do when __init__ runs afoul is going to be different from what you want to do to take care of object cleanup in __del__. So it doesn't belong there -- it belongs in an except: clause in __init__. Even though it's an incompatibility, i really think this is the right behaviour. -- ?!ng "To be human is to continually change. Your desire to remain as you are is what ultimately limits you." -- The Puppet Master, Ghost in the Shell
[Guido]
Why? That is, in what way is this an improvement over current behavior? Note that Java is a bit subtle: a finalizer is only called once by magic; explicit calls "don't count". The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer. I deal with possible exceptions in Python constructors the same way I do in C++ and Java: if there's a destructor, don't put anything in __init__ that may raise an uncaught exception. Anything dangerous is moved into a separate .reset() (or .clear() or ...) method. This works well in practice.
To implement this, we need a flag in each instance that means "__del__ was called".
At least <wink>.
I agree *that* isn't good. Taken on its own, though, it argues for adding an "instance construction completed" flag that __del__ later checks, as if its body were: if self.__instance_construction_completed: body That is, the problem you've identified here could be addressed directly.
I'd be in favor of fixing the actual problem; I don't understand the point to the rest of it, especially as it has the potential to break existing code and I don't see a compensating advantage (surely not compatibility w/ JPython -- JPython doesn't invoke __del__ methods at all by magic, right? or is that changing, and that's what's driving this?). too-much-magic-is-dizzying-ly y'rs - tim
"TP" == Tim Peters <tim_one@email.msn.com> writes:
TP> (surely not compatibility w/ JPython -- JPython doesn't invoke TP> __del__ methods at all by magic, right? or is that changing, TP> and that's what's driving this?). No, JPython doesn't invoke __del__ methods by magic, and I don't have any plans to change that. -Barry
[Tim]
Of course. Same in my proposal. But I wouldn't call it "by magic" -- just "on behalf of the garbage collector".
The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer.
I don't find the Java rules confusing. It seems quite useful that the GC promises to call the finalizer at most once -- this can simplify the finalizer logic. (Otherwise it may have to ask itself, "did I clean this already?" and leave notes for itself.) Explicit finalizer calls are always a mistake and thus "don't count" -- the response to that should in general be "don't do that" (unless you have particularly stupid callers -- or very fearful lawyers :-).
Sure, but the rule "if __init__ fails, __del__ won't be called" means that we don't have to program our __init__ or __del__ quite so defensively. Most people who design a __del__ probably assume that __init__ has run to completion. The typical scenario (which has happened to me! And I *implemented* the damn thing!) is this: __init__ opens a file and assigns it to an instance variable; __del__ closes the file. This is tested a few times and it works great. Now in production the file somehow unexpectedly fails to be openable. Sure, the programmer should've expected that, but she didn't. Now, at best, the failed __del__ creates an additional confusing error message on top of the traceback generated by IOError. At worst, the failed __del__ could wreck the original traceback. Note that I'm not proposing to change the C level behavior; when a Py<Object>_New() function is halfway its initialization and decides to bail out, it does a DECREF(self) and you bet that at this point the <object>_dealloc() function gets called (via self->ob_type->tp_dealloc). Occasionally I need to initialize certain fields to NULL so that the dealloc() function doesn't try to free memory that wasn't allocated. Often it's as simple as using XDECREF instead of DECREF in the dealloc() function (XDECREF is safe when the argument is NULL, DECREF dumps core, saving a load-and-test if you are sure its arg is a valid object).
Sure -- but I would argue that when __del__ returns, __instance_construction_completed should be reset to false, because the destruction (conceptually, at least) cancels out the construction!
JPython's a red herring here. I think that the proposed change probably *fixes* much morecode that is subtly wrong than it breaks code that is relying on __del__ being called after a partial __init__. All the rules relating to __del__ are confusing (e.g. what __del__ can expect to survive in its globals). Also note Ping's observation: | If it's up to the implementation of __del__ to deal with a problem | that happened during initialization, you only know about the problem | with very coarse granularity. It's a pain (or even impossible) to | then rediscover the information you need to recover adequately. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Tim]
Note that Java is a bit subtle: a finalizer is only called once by magic; explicit calls "don't count".
[Guido]
Of course. Same in my proposal.
OK -- that wasn't clear.
But I wouldn't call it "by magic" -- just "on behalf of the garbage collector".
Yup, magically called <wink>.
The Java rules add up to quite a confusing mish-mash. Python's rules are *currently* clearer.
I don't find the Java rules confusing.
"add up" == "taken as a whole"; include the Java spec's complex state machine for cleanup semantics, and the later complications added by three (four?) distinct flavors of weak reference, and I doubt 1 Java programmer in 1,000 actually understands the rules. This is why I'm wary of moving in the Java *direction* here. Note that Java programmers in past c.l.py threads have generally claimed Java's finalizers are so confusing & unpredictable they don't use them at all! Which, in the end, is probably a good idea in Python too <0.5 wink>.
It seems quite useful that the GC promises to call the finalizer at most once -- this can simplify the finalizer logic.
Granting that explicit calls are "use at your own risk", the only user-visible effect of "called only once" is in the presence of resurrection. Now in my Python experience, on the few occasions I've resurrected an object in __del__, *of course* I expected __del__ to get called again if the object is about to die again! Typical: def __del__(self): if oops_i_still_need_to_stay_alive: resurrect(self) else: # really going away release(self.critical_resource) Call __del__ only once, and code like this is busted bigtime. OTOH, had I written __del__ logic that relied on being called only once, switching the implementation to call it more than once would break *that* bigtime. Neither behavior is an obvious all-cases win to me, or even a plausibly most-cases win. But Python already took a stand on this & so I think you need a *good* reason to change semantics now.
This is (or can easily be made) a separate issue, & I agreed the first time this seems worth fixing (although if nobody has griped about it in a decade of use, it's hard to call it a major bug <wink>).
In the __del__ above (which is typical of the cases of resurrection I've seen), there is no such implication. Perhaps this is philosophical abuse of Python's intent, but if so it relied only on trusting its advertised semantics.
Yes, again, I have no argument against refusing to call __del__ unless __init__ succeeded. Going beyond that to a new "called at most once" rule is indeed going beyond that, *will* break reasonable old code, and holds no particular attraction that I can see (it trades making one kind of resurrection scenario easier at the cost of making other kinds harder). If there needs to be incompatible change here, curiously enough I'd be more in favor of making resurrection illegal period (which could *really* simplify gc's headaches).
All the rules relating to __del__ are confusing (e.g. what __del__ can expect to survive in its globals).
Problems unique to final shutdown don't seem relevant here.
Also note Ping's observation: ...
I can't agree with that yet another time without being quadruply redundant <wink>.
OK, so we're down to this one point: if __del__ resurrects the object, should __del__ be called again later? Additionally, should resurrection be made illegal? I can easily see how __del__ could *accidentally* resurrect the object as part of its normal cleanup -- e.g. you make a call to some other routine that helps with the cleanup, passing self as an argument, and this other routine keeps a helpful cache of the last argument for some reason. I don't see how we could forbid this type of resurrection. (What are you going to do? You can't raise an exception from instance_dealloc, since it is called from DECREF. You can't track down the reference and replace it with a None easily.) In this example, the helper routine will eventually delete the object from its cache, at which point it is truly deleted. It would be harmful, not helpful, if __del__ was called again at this point. Now, it is true that the current docs for __del__ imply that resurrection is possible. The intention of that note was to warn __del__ writers that in the case of accidental resurrection __del__ might be called again. The intention certainly wasn't to allow or encourage intentional resurrection. Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this. [Jack just finds a snag]
Yes, that's a problem. But there are other ways for the subclass to break the base class's invariant (e.g. it could override __del__ without calling the base class' __del__). So I think it's a red herring. In Python 3000, typechecked classes may declare invariants that are enforced by the inheritance mechanism; then we may need to keep track which base class constructors succeeded and only call corresponding destructors. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Yes and no :-) One example comes to mind: implementations of weak references, which manage weak object references themselves (as soon as __del__ is called the weak reference implementation takes over the object). Another example is that of free list like implementations which reduce object creation times by implementing smart object recycling, e.g. objects could keep allocated dictionaries alive or connections to databases open, etc. As for the second point: Calling __del__ again is certainly needed to keep application logic sane... after all, __del__ should be called whenever the refcount reaches 0 -- and that can happend more than once in the objects life-time if reanimation occurs.
I'd say this is an application logic error -- nothing that the mechanism itself can help with automagically. OTOH, turning multi calls to __del__ off, would make certain techniques impossible.
I don't think that docs are the right argument here ;-) It is simply the reference counting logic that plays its role: __del__ is called when refcount reaches 0, which usually means that the object is about to be garbage collected... unless the object is rereferenced by some other object and thus gets reanimated.
Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this.
BTW, I can't see what the original question has to do with this discussion ... calling __del__ only after successful __init__ is ok, IMHO, but what does this have to do with the way __del__ itself is implemented ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Guido van Rossum wrote:
[much stuff] Just a random note: What if we had a __del__ with zombie behavior? Assume an instance that is about to be destructed. Then __del__ is called via normal method lookup. What we want is to let this happen only once. Here the Zombie: After method lookup, place a dummy __del__ into the to-be-deleted instance dict, and we are sure that this does not harm. Kinda "yes its there, but a broken link ". The zombie always works by doing nothing. Makes some sense? ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home
[Guido]
I give up on the latter, so it really is just one.
If this is something that happens easily, and current behavior is harmful, don't you think someone would have complained about it by now? That is, __del__ *is* "called again at this point" now, and has been for years & years. And if it happens easily, it *is* happening now, and in an unknown amount of existing code. (BTW, I doubt it happens at all <wink> -- people tend to write very simple __del__ methods, so far as I've ever seen)
Now, it is true that the current docs for __del__ imply that resurrection is possible.
"imply" is too weak. The Reference Manual's "3.3.1 Basic customization" flat-out says it's possible ("though not recommended"). The precise meaning of the word "may" in the following sentence is open to debate, though.
The intention of that note was to warn __del__ writers that in the case of accidental resurrection
Sorry, but I can't buy this: saying that *accidents* are "not recommended" is just too much of a stretch <wink/frown>.
__del__ might be called again.
That's a plausible reading of the following "may", but not the only one. I believe it's the one you intended, but it's not the meaning I took prior to this.
The intention certainly wasn't to allow or encourage intentional resurrection.
Well, I think it plainly says it's supported ("though not recommended"). I used it intentionally at KSR, and even recommended it on c.l.py in the dim past (in one of those "dark & useless" threads <wink>).
Would there really be someone out there who uses *intentional* resurrection? I severely doubt it. I've never heard of this.
Why would anyone tell you about something that *works*?! You rarely hear the good stuff, you know. I gave the typical pattern in the preceding msg. To flesh out the motivation more, you have some external resource that's very expensive to set up (in KSR's case, it was an IPC connection to a remote machine). Rights to use that resource are handed out in the form of an object. When a client is done using the resource, they *should* explicitly use the object's .release() method, but you can't rely on that. So the object's __del__ method looks like (for example): def __del__(self): # Code not shown to figure out whether to disconnect: the downside to # disconnecting is that it can cost a bundle to create a new connection. # If the whole app is shutting down, then of course we want to disconnect. # Or if a timestamp trace shows that we haven't been making good use of # all the open connections lately, we may want to disconnect too. if decided_to_disconnect: self.external_resource.disconnect() else: # keep the connection alive for reuse global_available_connection_objects.append(self) This is simple & effective, and it relies on both intentional resurrection and __del__ getting called repeatedly. I don't claim there's no other way to write it, just that there's *been* no problem doing this for a millennium <wink>. Note that MAL spontaneously sketched similar examples, although I can't say whether he's actually done stuff like this. Going back up a level, in another msg you finally admitted <wink> that you want "__del__ called only once" for the same reason Java wants it: because gc has no idea what to do when faced with finalizers in a trash cycle, and settles for an unprincipled scheme whose primary virtue is that "it doesn't blow up" -- and "__del__ called only once" happens to be convenient for that scheme. But toss such cycles back to the user to deal with at the Python level, and all those problems go away (along with the artificial need to change __del__). The user can break the cycles in an order that makes sense to the app (or they can let 'em leak! up to them). >>> print gc.get_cycle.__doc__ Return a list of objects comprising a single garbage cycle; [] if none. At least one of the objects has a finalizer, so Python can't determine the intended order of destruction. If you don't break the cycle, Python will neither run any finalizers for the contained objects nor reclaim their memory. If you do break the cycle, and dispose of the list, Python will follow its normal reference-counting rules for running finalizers and reclaiming memory. That this "won't blow up" either is just the least of its virtues <wink>. you-break-it-you-buy-it-ly y'rs - tim
Tim Peters wrote:
Not exactly this, but similar things in the weak reference implementation of mxProxy. The idea came from a different area: the C implementation of Python uses free lists a lot and these are basically implementations of the same idiom: save an allocated resource for reviving it at some later point. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[M.-A. Lemburg, on the resurrection/multiple-__del__ "idiom"]
Excellent analogy! Thanks. Now that you phrased it in this clarifying way, I recall that very much the same point was raised in the papers that resulted in the creation of guardians in Scheme. I don't know that anyone is actually using Python __del__ this way today (I am not), but you reminded me why I thought it was natural at one time <wink>. generally-__del__-aversive-now-except-in-c++-where-destructors-are- guaranteed-to-be-called-when-you-except-them-to-be-ly y'rs - tim
participants (8)
-
Barry A. Warsaw -
bwarsaw＠cnri.reston.va.us
-
Christian Tismer -
Greg Stein -
Guido van Rossum -
Ka-Ping Yee -
M.-A. Lemburg -
Tim Peters