On Sat, Jun 28, 2008 at 5:39 PM, Greg Ewing wrote:

Nick Coghlan wrote:

...

It's a fact of Python development: __del__ methods cannot safely reference module globals, because those globals may be gone by the time that method is invoked.

Speaking of this, has there been any more thought given to the idea of dropping the module clearing and just relying on cyclic GC?

No, but it is an intriguing thought nevertheless. The module clearing causes nothing but trouble... -- --Guido van Rossum (home page: http://www.python.org/~guido/)

eyal.lotem+pyutils <at> gmail.com writes:

This is exactly what my post tried to address. I assumed it was clear that module clearing is the wrong solution, and that it was also clear that due to the cycles I mentioned (global.__class__.__dict__['any_method'].func_globals['global'] is global), all globals that have a __del__ will not be collectible. Therefore, I proposed a solution to cycles with a __del__ in them. Only with this solution it is possible to replace module clearing with normal garbage collection.

A more generic solution would be to pass a special flag to the GC when it is called as part of interpreter shutdown. This flag would instruct it to not put aside objects with finalizers, but first call all their __del__ methods and then collect them all at once regardless of whether they are part of a cycle or not.

eyal.lotem+pyutils <at> gmail.com writes:

That would be no worse than what happens now - but its still not perfect (__del__ ordering issues). Also, you would need to temporarily revive the cycles as mentioned above (to avoid accessibility of partially destructed objects).

The idea is to call all __del__'s *before* any object in the cycle is deallocated (that is, call them manually rather than as part of deallocating them). That way you shouldn't have the issues mentioned above.

On Jun 29, 5:12 pm, "eyal.lotem+pyut...@gmail.com" wrote:

On Jun 29, 3:36 pm, Antoine Pitrou wrote:

...

eyal.lotem+pyutils <at> gmail.com writes:

...

...

That would be no worse than what happens now - but its still not perfect (__del__ ordering issues). Also, you would need to temporarily revive the cycles as mentioned above (to avoid accessibility of partially destructed objects).

...

The idea is to call all __del__'s *before* any object in the cycle is deallocated (that is, call them manually rather than as part of deallocating them). That way you shouldn't have the issues mentioned above.

Firstly, as I said above: you will still have __del__ ordering issues. Secondly, the destructor itself currently calls __del__, so if you call __del__ before any deallocation, it will get called again as part of the deallocation.  Might be a technicality but it will still probably require some code restructuring to work around (or making that code even more hairy).

Additionally, there is another problem: If the cycle is not temporarily revived, and you call __del__ manually, it may break the cycle by removing the references. Thus, objects in the cycle will go down to refcount=0 during your attempt to call __del__'s on the objects in the cycle. The only sane thing is to temporarily revive the entire cycle so you can safely call __del__'s on it. Then, you might want to disable the normal __del__ calling that occurs as part of the later destruction of the cycle.

...

_______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

_______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

Antoine Pitrou writes:

I was thinking a flag in the PyObject header would do the trick but there aren't any flags in the PyObject header... *gasp*.

Actually, we only care about GC-tracked objects (the others are deallocated simply when their refcount falls to zero), so this could be another special value in the gc_refs field, together with a specific macro.

Firstly, as I said above: you will still have __del__ ordering issues.

Can you please elaborate? What would such __del__ ordering issues be?

Secondly, the destructor itself currently calls __del__, so if you call __del__ before any deallocation, it will get called again as part of the deallocation. Might be a technicality but it will still probably require some code restructuring to work around (or making that code even more hairy).

There could be a global barricade for calling __del__: you first call all __del__s of existing objects, then set the barricade, and then start breaking cycles. This could even be done with the current approach to module clearing. Regards, Martin

That would be no worse than what happens now - but its still not perfect (__del__ ordering issues). Also, you would need to temporarily revive the cycles as mentioned above (to avoid accessibility of partially destructed objects).

I don't quite understand what you mean by "revive cycles". There is not need to revive any object in a cycle, as all objects are still alive (or else they wouldn't be garbage). Regards, Martin

On Jun 28, 6:21 pm, Nick Coghlan wrote:

Eyal Lotem wrote:

...

Example:

...

import os class RunningFile(object):     filename = '/tmp/running'     def __init__(self):         open(self.filename, 'wb')     def __del__(self):         os.unlink(self.filename) running_file = RunningFile()

...

The deller object is in a cycle as described above [as well as the Deller class itself].  When Python exits, it could call deller.__del__() and then collect the cycle. But Python does the wrong thing here, and gets rid of the globals before calling __del__: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'unlink'" in > ignored

I don't know what you're trying to get at with this example. There isn't any cyclic GC involved at all, just referencing counting. And before the module globals are cleared, running_file is still referenced, so calling its __del__ method early would be an outright error in the interpreter (as far as I know, getting __del__ methods to run is one of the *reasons* for clearing the module globals).

It's a fact of Python development: __del__ methods cannot safely reference module globals, because those globals may be gone by the time that method is invoked. That's because globals are being cleaned up in an ad-hoc manner, instead of just using cyclic GC that supports cycles.

Cheers, Nick.

-- Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia ---------------------------------------------------------------              http://www.boredomandlaziness.org _______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

On Jun 28, 6:32 pm, "Martin v. Löwis" wrote:

...

Example:

...

import os class RunningFile(object):     filename = '/tmp/running'     def __init__(self):         open(self.filename, 'wb')     def __del__(self):         os.unlink(self.filename) running_file = RunningFile()

...

The deller object is in a cycle as described above [as well as the Deller class itself].

I think you are mistaken here. The RunningFile instance in above code is *not* part of a cycle. It doesn't have any instance variables (i.e. its __dict__ is empty), and it only refers to its class, which (AFAICT) doesn't refer back to the instance.

As I explained above, it *is* part of a cycle: """including the class objects themselves: class->dict->function->func_globals""". Note: running_file.__class.__.__dict__['__init__'].func_globals['running_file'] is running_file.

...

When Python exits, it could call deller.__del__() and then collect the cycle. But Python does the wrong thing here, and gets rid of the globals before calling __del__: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'unlink'" in > ignored

This is a different issue. For shutdown, Python doesn't rely on cyclic garbage collection (only). Instead, all modules get forcefully cleared, causing this problem.

I know. I assumed Python does not rely on cyclic garbage collectioin for shutdown, because it wouldn't work, as _all_ globals that have *any* instance method will be part of a cycle, and any of them which have a __del__ will not be collected.

...

I believe applying the above enhancement would solve these problems.

No, they wouldn't.

To work around the real problem in your case, put everything that the destructor uses into an instance or class attribute:

class RunningFile(object):     filename = '/tmp/running'     _unlink = os.unlink     def __init__(self):         open(self.filename, 'wb')     def __del__(self):         self._unlink(self.filename) I *mentioned* this workaround. What I propose is not a workaround but a solution. You wouldn't need to clean up module globals ad-hoc'ishly, because the cyclic collection would collect your object, even with its __del__.

Regards, Martin Please read my entire mail before replying to it. Thanks! Eyal

_______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

On Sun, Jun 29, 2008 at 9:00 PM, "Martin v. Löwis" wrote:

...

As I explained above, it *is* part of a cycle: """including the class objects themselves: class->dict->function->func_globals""".

Ah, right. I must have missed that explanation.

...

I know. I assumed Python does not rely on cyclic garbage collectioin for shutdown, because it wouldn't work, as _all_ globals that have *any* instance method will be part of a cycle, and any of them which have a __del__ will not be collected.

No. The mechanism for cleaning modules at shutdown time predates cyclic GC, and was not removed because "it wouldn't work".

This specific issue certainly contributes to the fact that it doesn't work, but there might be other problems as well (such as extension modules holding onto objects participating in cycles).

...

I *mentioned* this workaround. What I propose is not a workaround but a solution. You wouldn't need to clean up module globals ad-hoc'ishly, because the cyclic collection would collect your object, even with its __del__.

I don't think it solves the problem. You seem to be assuming that any such cycle will contain only a single global object with an __del__. However, as modules refer to each other, I very much doubt that is the case.

...

Please read my entire mail before replying to it. Thanks!

I really, really tried. I read it three times before replying.

However, I found it really, really difficult to follow your writing, as it was mixing problem statement and solution, so that I couldn't tell what paragraph was about what. English is not my native language, complicating communication further. Please accept my apologies.

I apologize for my tone as well, it just seemed frustrating that all the replies seemed to completely ignore the core point I was trying to make and claim that the problem I was trying to solve was not a problem at all, but a behavior I was unaware of... Mixing another quote from another mail:

...

That would be no worse than what happens now - but its still not perfect (__del__ ordering issues). Also, you would need to temporarily revive the cycles as mentioned above (to avoid accessibility of partially destructed objects).

I don't quite understand what you mean by "revive cycles". There is not need to revive any object in a cycle, as all objects are still alive (or else they wouldn't be garbage). By "revive cycles", I mean make sure that they are referenced by an independent referrer (one that won't go away as part of the __del__ calling process). This is similar to how the tp_dealloc code increases the refcount (actually sets it to 1, because it was certainly 0 when entering the destructor) before calling the __del__ slot. Without reviving the object before calling its __del__ in the destructor, and without reviving the objects of a cycle before calling its __del__'s, the __del__ Pythonic code may be exposed to "dead objects" (refcount==0).

Consider the cycle: a.x = b b.x = a Lets suppose the a object has a __del__. Lets assume each object in the cycle has a refcount of 1 (and the cycle should die). Now lets say this is a's __del__ code: def __del__(self): self.x = None Running it will set 'b's refcount to 0 and call its destructor, which will set 'a's refcount to 0 and also call its destructor. But its __del__ is currently running - so "self" must not have a refcount of 0. If you only incref on 'a' before calling __del__, then you are probably alright, as long as there is only one __del__.

Can you please elaborate? What would such __del__ ordering issues be?

class A(object): def __del__(self): print self.x.attribute class B(object): def __del__(self): print "B is going down!" del self.attribute a = A() b = B() a.x = b b.attribute = 1 If you call b's __del__ first then a's __del__ will fail. If you call a's __del__ first, then all is well. Ofcourse you can create true cyclic dependencies that no order will work, and its pretty clear there is no way to deduce the right order anyway. This is what I mean by "ordering issues".

There could be a global barricade for calling __del__: you first call all __del__s of existing objects, then set the barricade, and then start breaking cycles. This could even be done with the current approach to module clearing.

Note that the __del__'s themselves may be breaking cycles and refcounts will go to 0 - unless you temporarily revive (incref) the entire cycle first.

I still don't understand what "revive the cycle" means. You will need to incref the object for which you call __del__, that's all.

Unless there are multiple __del__'s in the cycle.

Regards, Martin