Oliver Schoenborn wrote:

...

This is a module I made to test a concept of finalization *not* based on __del__. I'm curious to find out if the techniques look like they could be implemented directly in the Python interpreter to lessen the burden on

----- Original Message ----- From: "Martin v. Löwis" <martin@v.loewis.de> the

...

user, which is why I post this here.

Yes, it could be implemented directly on the user. However, I notice that the algorithm you implement has nothing to do with finalization. ... In summary: to avoid confusion, you should not use the term "finalization" when talking about the mechanisms you have implemented in the module.

Granted, it's not finalization in the technical sense but it is in the logical sense: when the function returns, the try/finally calls a method which must assume that the object will no longer be used after that (i.e. no other references will be keeping the object alive past the function).

Python does provide finalization, and in a reliable way: the finalizer is called before the object goes away, guaranteed.

The python docs are a bit confusing to me in this regard, so allow me to quote a few section: From Python 2.3 docs: "An implementation is allowed to postpone garbage collection or omit it altogether -- it is a matter of implementation quality how garbage collection is implemented, as long as no objects are collected that are still reachable. (Implementation note: the current implementation uses a reference-counting scheme with (optional) delayed detection of cyclically linked garbage, which collects most objects as soon as they become unreachable, but is not guaranteed to collect garbage containing circular references. ...)"

From Python 2.3 docs, section 3.1 of Library Ref:

"Some objects contain references to external resources such as open files or windows. It is understood that these resources are freed when the object is garbage collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a close() method. Programs are strongly recommended to explicitly close such objects. The 'try...finally' statement provides a convenient way to do this."

From section 3.3.1:

__del__( self) Called when the instance is about to be destroyed. .... It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits. ...x.__del__() ... is only called when x's reference count reaches zero. And finally, from 1.10 of "Extending and Embedding...": "While Python uses the traditional reference counting implementation, it also offers a cycle detector that works to detect reference cycles. This allows applications to not worry about creating direct or indirect circular references" This seems to indicate that there are several situations to keep in mind: 1) __del__ is called automatically when reference count goes to zero; this is independent of the presence of gc; however, standard doesn't garantee that __del__ gets called for objects still alive when interpreter exits, so for critical pieces of code (like releasing mutex locks), this could be dangerous; plus the object is kept alive in the case of an exception, so can't rely on its __del__ being called if exception is not caught. 2) the (optional) garbage collector handles the cases where reference count never goes to zero due to cyclical refs involving at least one class with a __del__ defined; however it is implementation-defined whether this exists and if it does, when it does its job; so classes that have a __del__ are unlikely to get freed by the gc if they are involved in a cyclical ref. So it seems that having a __del__ is ok *most* of the time, and that if you find a mem leak, verify if you have cyclical refs, and modify algorithm to break cycle because gc won't be able to call __del__; however if an exception gets thrown, you can't rely on __del__ getting called. Since exceptions are so common, I have to conclude that you can never rely on __del__ to free resources. Where am I wrong in this reasoning?

What you want is a function that is called when the object goes of of some scope. This is something completely different, as the object may life long and prosper even after it goes out of scope.

You have an assertion

assert sys.getrefcount(self) == 5

It is very easy to trigger that assertion, so I'm uncertain what the intended meaning of that assertion is.

Was useful only during testing and has since been removed. Thanks for the feedback! Oliver

From: "Martin v. Löwis" <martin@v.loewis.de>

Oliver Schoenborn wrote:

...

So it seems that having a __del__ is ok *most* of the time, and that if you find a mem leak, verify if you have cyclical refs, and modify algorithm to break cycle because gc won't be able to call __del__; however if an exception gets thrown, you can't rely on __del__ getting called. Since exceptions are so common, I have to conclude that you can never rely on __del__ to free resources. Where am I wrong in this reasoning?

Yes, you are wrong. The Python language specification does not make guarantees. However, specific implementations do make specific guarantees beyond what the language specification guarantees.

Which means nothing because if I code in Python, code portability is a major concern. Any aspects of the language that are not in the specification must be avoided, so if specs don't guarantee s/m, no one in their right mind would use that s/t.

Your proposed mechanism cannot be added to the Python language specification because there might be Python implementations for which the mechanism is difficult to implement.

I can accept that only if there is something inherent to the technique which exploits a non-specified aspect of the language. I can't think of any. Could you be more specific about what aspect is not portable? If it is portable, then since it works as-is in Python, it could be put into the specs. At the very least, Python interpreter implementers who can't implement right into the interpreter could provide it as a module, like it is now (and as it would be anyways, so that the front end never changes, a bit like weakref). Oliver

Martin, From: "Martin v. Löwis" <martin@v.loewis.de>

Oliver Schoenborn wrote:

...

I can accept that only if there is something inherent to the technique which exploits a non-specified aspect of the language. I can't think of any. Could you be more specific about what aspect is not portable?

The idea that you can execute code at the end of a function to invoke teardown operations, even in the presence of exceptions. That cannot be done for Jython, AFAIK. Jython generates Java byte code from Python source code, so there is no Python virtual machine anywhere.

That's assuming that what you say below is true. The technique, shown in scope.py attached to the first post to python-dev, is simple (and consists in rebinding a function to a wrapper that puts a try/finally around the original function, with some special behind-the-scenes stuff for what to execute in the finally block). Scope.py works here, so I don't understand your statement that the technique doesn't work in Python.

...

If it is portable, then since it works as-is in Python, it could be put into the specs.

It doesn't work in Python. I believe your implementation also fails in Jython (but didn't test).

Unless we're talking cross-purpose here, scope.py shows it works at least in Python downloaded from python.org. Again, could you please be more specific, pointing to which part of the code is allegedly flawed or not portable? Oliver

This never got sent, sorry, may help... From: "Martin v. Löwis" <martin@v.loewis.de>

Oliver Schoenborn wrote:

...

...

It doesn't work in Python. I believe your implementation also fails in Jython (but didn't test).

Unless we're talking cross-purpose here, scope.py shows it works at least in Python downloaded from python.org. Again, could you please be more specific, pointing to which part of the code is allegedly flawed or not portable?

Try running it with Jython 2.1, available from www.jython.org. I get the error

ImportError: no module named inspect

I checked on the jython site, the cvs repository, and the inspect module is only partially implemented as a wrapper around the Python inspect module. This problem you are having seems related to implementation, not technique. The technique itself can be summarized as follows: - Create a function that wraps a try/finally around the function of interest - Make class whose instances need "cleanup" automatically tell the function in which they were created (the "function of interest" above) that they should be cleaned up upon function exit. This requires three things: - A new class in which that functionality can be encapsulated so the user doesn't have to replicate it every time; just need to derive from it - Someone (the new class, or the wrapper) to create new function attribute - When an object is instantiated, add ref to self to the new, special function attribute, like Python allows. This itself requires: - Find function in which self was instantiated so that the function attribute can be accessed The inspect module is used in the very last task. The new function attribute is only used as a convenient place to store information which is closely tied to the function. But there are other possibilities. E.g. instead of creating a function attribute, the ScopeGuardian could use a global stack of list of objects. An instance of NeedsFinalization would use the top list of the stack to add a reference to istelf there. It looks like Jython has some parts of python lib implemented natively, some borrowed, and some not implemented, based on need and manpower, so the fact that the inspect module is not available hardly qualifies the technique as "not working". If Jython doesn't use stack, and implementing in terms of global var not feasible, *and* there is no other implementation possible in Jython, then yeah, I agree, scope.py is not portable to Jython. However that doesn't mean that the technique can't be implemented different in Python interpreter and Jython interpreter. Cheers, Oliver

...

The technique itself can be summarized as follows:

- Create a function that wraps a try/finally around the function of interest - Make class whose instances need "cleanup" automatically tell the function in which they were created (the "function of interest" above) that they should be cleaned up upon function exit. This requires three things:

I claim that this might not be possible to do across different implementations of Python. It relies on the ability to look at the local variables of a caller of a function, and the Python language does not guarantee support for such a mechanism.

Actually it doesn't. The file I showed in my first post to python-dev used the stack, plus the knowledge of what name was given to a local variable, but it didn't need to "look at the local variables"; it looked at one variable, of known name. I have a second implementation, below, that requires neither stack, nor local, and is faster.

...

tied to the function. But there are other possibilities. E.g. instead of creating a function attribute, the ScopeGuardian could use a global stack of list of objects. An instance of NeedsFinalization would use the top list of the stack to add a reference to istelf there.

In that implementation, how would you know how many objects to cleanup?

See my reply to your last post where you ask same question. However, as to this new implementation, I paste it in at the end of the post.

Yes, but I have no reason to believe you an implementation is possible until I have seen an implementation.

It might be that an implementation of the feature would be very expensive even if not used, in which case the feature also would not be acceptable for inclusion in Python.

I'm doing some timing tests, I should have them tonight with a bit of luck.

However, this is probably the time to ask for a PEP that properly describes the API and the semantics of the feature.

In time... I'd like a couple more opinions if possible. I'm surprised there hasn't been more given how fundamental the problem of RAII is in Python. Oliver ----------- New implementation for scope.py -------------- # Note: comments and tests are in module, but not reproduced here for space import sys def ScopeGuarded(func): return lambda *args, **kwargs: ScopeGuardian(func, *args, **kwargs) _funcStack = [] class NeedsFinalization: def __init__(self): print '\n%s: being created' % repr(self) self.__finalized = False try: _funcStack[-1].append(self) except IndexError: raise RuntimeError, "Forgot to scope-guard function? " def finalizeMaster(self): print '%s: Finalize() being called' % repr(self) self._finalize() self.__finalized = True def __del__(self): try: problem = not self.__finalized except AttributeError: msg = '%s: NeedsFinalization.__init__ not called for %s' \ % (repr(self), self.__class__) raise RuntimeError, msg if not problem: print '%s: Finalized properly' % repr(self) else: print "Forgot to scope-guard function" def ScopeGuardian(func, *args, **kwargs): try: scopedObjs = [] _funcStack.append(scopedObjs) func(*args, **kwargs) print 'Scoped variables created during call to %s: %s' \ % (func, scopedObjs) finally: _funcStack.pop() scopedObjs.reverse() # destroy in reverse order from creation for obj in scopedObjs: obj.finalizeMaster() def testBasic(): def func1(): ok = TestFree() danger = TestDanger() # test when forget to use ScopeGuardian try: hello = func1() assert False, 'Expected exception not thrown!' except RuntimeError, msg: print 'Expected exception caught: ', msg func1 = ScopeGuarded(func1) func1() def func2(objType): dontKnow = objType() func2 = ScopeGuarded(func2) print "An RuntimeError exception will happen but be ignored: " func2(TestDangerNoInit) func2(TestDanger) def testRecursive(): def recurse(n): print 'Recurse(%s)' % n danger = TestDanger() ok = TestFree() if n>0: recurse(n-1) else: print 'recurse: Raising exception' raise RuntimeError, "pretend exception thrown during recursion" print '\nTesting that recursive does not work' recurse = ScopeGuarded(recurse) try: recurse(3) assert False, 'Expected exception not thrown!' except RuntimeError, msg: print 'Expected exception caught: ', msg if __name__ == '__main__': class TestFree: def _finalize(): raise RuntimeError, 'finalize() not supposed to be called' class TestDanger(NeedsFinalization): def __init__(self): NeedsFinalization.__init__(self) def _finalize(self): """Override this. If you class inherits from a class derived from NeedsFinalization, make sure to call parent.finalize().""" pass class TestDangerNoInit(NeedsFinalization): def __init__(self): pass def _finalize(self): pass testBasic() testRecursive()

"Martin v. Löwis" <martin@v.loewis.de> writes:

Oliver Schoenborn wrote:

...

I can accept that only if there is something inherent to the technique which exploits a non-specified aspect of the language. I can't think of any. Could you be more specific about what aspect is not portable?

The idea that you can execute code at the end of a function to invoke teardown operations, even in the presence of exceptions. That cannot be done for Jython, AFAIK. Jython generates Java byte code from Python source code, so there is no Python virtual machine anywhere.

I'm sure I missed something important along the way here, but the fact that you can generate Java bytecode for Python's "finally" in Jython, it seems to me, _proves_ that you can also generate code to that will be executed at the end of a function even in the presence of exceptions. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

W liście z sob, 12-06-2004, godz. 14:36 +0200, "Martin v. Löwis" napisał:

If it involves looking at all local variables, it might be implementable, but might cause a significant slowdown even if the feature is not used.

Not just inefficient but wrong: not all such objects should be finalized. They can have references stored elsewhere. -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

From: "Martin v. Löwis" <martin@v.loewis.de>

Marcin 'Qrczak' Kowalczyk wrote:

...

...

If it involves looking at all local variables, it might be implementable, but might cause a significant slowdown even if the feature is not used.

Not just inefficient but wrong: not all such objects should be finalized. They can have references stored elsewhere.

I understand that is the whole purpose of the requested feature: to invoke a method on the object when a function completes, regardless of whether there are still references to the object.

Actually, what I am bringing forward does not address that case. The implicit assumption is that the coder is sure that zero references to the object will be left upon return from function. This is usually the case when opening files, acquiring locks etc: if the coder wants proper cleanup, they *have* to assume there are no references left. E.g., def func(): ff = file('foo.txt', 'w') # do calls to various other functions ff.close() The implicit assumption is that the close() is safe to do, i.e. that none of the functions called between the open and the close stored references. If there are refs stored and try to write, exception will occur, in section of program totally unrelated to func(), certainly not expected behavior by the coder. I think it is safe to say that in cases where deterministic "finalization" is desired, the coder doesn't want the object to be shared, and makes the assumption that refs aren't being stored without their knowledge. If they are, even current idioms, like the simplistic example above, are flawed. Oliver

"Martin v. Löwis" <martin@v.loewis.de> wrote in message news:40CAF8E0.1030909@v.loewis.de...

Certainly. However, I'm uncertain whether you find out, in this finally block, what objects require this finally-ization.

I brought up the same point on the parallel thread in c.l.p and got no response. As I remember, one way to create dangling pointers in C is for a function to pass out of its scope a pointer to a local about to be freed. I presume the same is true in C++. To me, a main purpose of ref counting is to know what *not* to release at function exit. People who want more agressive cleanup on function exit could perhaps develop a version of Python that calls a cyclic-junk finalizer *after* locals are decrefed, as part of the execution of explicit or implicit 'return'. The cyclic-junk finalizer could either be cyclic gc improved to work with __del__ methods or a separate process that used the current cycle detection apparatus to find candidates for processing. Since most people neither need nor want the extra return overhead, this should be a third-party add-on, a compilation option, or possibly a built-in decoration like staticmethod. My naive guess is that this would take a least a person-month. I leave it to *RAII-in-Python advocates to investigate further. Terry J. Reedy *Resource Aquisition Is Initialization, in C++ -- because of auto destruction of locals.

Oliver Schoenborn wrote:

Those objects that have "registered" themselves to the "cleaner" require finally-ization. Hence, if no objects registered themselves, there is only the overhead of an empty finally. The implementation details can worry about how registration takes place, and who does the cleanup. In scope.py, "registration" is handled by derivation from class NeedsFinalization -- which I can eventually rename so please let's not get stuck on terminology at this stage. The cleaner is a function called ScopeGuardian.

I'm still uncertain, though, as to what objects precisely should get the cleanup calls: *All* objects that have registered with the cleaner, or only some of them? If all objects: what if nested function calls each have to-clean objects, and only the inner function returns? Why is it then useful to also clean objects that where created in an outer function? If only some objects: which ones? Regards, Martin

Oliver Schoenborn wrote:

...

I'm still uncertain, though, as to what objects precisely should get the cleanup calls: *All* objects that have registered with the cleaner, or only some of them?

I don't see why it wouldn't be all objects. You'll have to show me an example for when that might be the case. Each object that has registered is saying, "I need cleanup at exit of scope, because I should be the only one left". If you don't clean it up you are not upholding your side of the contract.

Consider this example, where X is a class that needs finally-ization, and f and g are functions that participate in the protocol: def f(): a = X() g() a.do_something() def g(): b = X() Now, when f starts, an X instance is registered with the cleaner, and this is assigned to the local variable a. Then g is invoked. This creates another X instance that is also registered with the cleaner. Then g returns, and all objects registered with the cleaner are finally-ized. At the moment, there are two objects registered with the cleaner, so they are both finally-ized. Then, g returns, and f tries to use the objects stored in a. However, that object already has been finally-ized, so the object is no longer functional. I doubt this is the intended semantics, but I don't know what the intended semantics is instead. Regards, Martin

Oliver Schoenborn wrote:

Oh no, absolutely not! The objects are finaly-izated only for the frame where created. This is ensured because the cleaner uses a stack (unrelated to the frame stack), one item in the stack per function call. You will see this clearly if you look at the latest detscope.py. b is cleaned up upon return from g, and a will be cleaned up only upon return from f.

While I understand your implementation, I'm uncertain how much of it you consider normative, and how much is implementation detail. For example, is the order in which the finally-izers are called significant or not? Or, coming back to my original question: It appears that *not* all objects registered with the cleaner are destroyed when the function returns. This brings me back to my original question: As not all objects are finally-ized at the end of the function, then *which* objects precisely are finally-ized? Let me try guess an answer: At the end of a scope-guarded function, all need-finally-ization objects are finally-ized that have been created since the function was invoked, and which haven't yet been finally-ized. Taking threads into account, this definition is probably not what you want: likely, you want to see only those objects finally-ized, that have been created in the context of the thread in which the function was invoked. Is that correct? In particular, what happens with need-finally-ization objects which whose constructor call does not directly appear in a scope-guarded function? Regards, Martin