[Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)

[Phillip J. Eby]

At 06:56 PM 5/16/2005 +1000, Nick Coghlan wrote:
>Anyway, I think it's stable enough now that I can submit it to be put up on
>www.python.org (I'll notify the PEP editors directly once I fix a couple of
>errors in the current version - like the missing 'raise' in the statement
>semantics. . .).

If you have developer checkin privileges, it's best to get a PEP number 
sooner rather than later, if the PEP shows any signs of viability at 
all.  Once you're in the PEP infrastructure people can subscribe to get 
notified when you change it, read the revision history, and so on.

Anyway, I took a look at it, and I mostly like it.  There appears to be an 
error in "Deterministic generator finalisation" (maybe you already know 
this): the _inject_exception() should be called with exc_info, not 
TerminateIteration, and it should swallow StopIteration instead of 
TerminateIteration.  IOW, I think it should look like this:

     def __exit__(self, *exc_info):
         try:
             self._inject_exception(*exc_info)
         except StopIteration:
             pass

Hm.  Oh wait, I just realized - you don't mean this at all.  You're 
describing a use of generators as non-templates.  Ugh.  I think that might 
lead to confusion about the semantics of 'with' and generators.  I'll have 
to think about it some more, but my second impression after a little bit of 
thought is that if you're going to do this, then you should be allowed to 
use 'with' with any object, using the object as VAR if there's no 
__enter__.  My reasoning here is that it then makes it possible for you to 
use arbitrary objects for 'with' without needing to know their 
implementation details.  It should be harmless to use 'with' on objects 
that don't need it.

This insight may actually be true regardless of what generators do or don't 
do; the point is that if you change from using a generator to a built-in 
iterator type, you shouldn't have to change every place you were using the 
'with' blocks to work again.

A further part of this insight: perhaps the 'with' block translation should 
include a 'del VAR' in its finally block, not to mention the equivalent of 
'del stmt_enter,stmt_exit'.  In other words, the binding of VAR should not 
escape the 'with' block.  This would mean that for existing types that use 
__del__ for cleanup (e.g. files and sockets), then 'with open("file") as f' 
would automatically ensure closing under CPython (but other implementations 
would be allowed to wait for GC).  In other words, I'm saying that this:

      with some_expr() as foo:
          # etc.

should also be short for this (in the case where some_expr() has no 
__enter__ or __exit__ methods):

      foo = some_expr()
      try:
          # etc.
      finally:
          del foo

And that could be a useful thing for many existing object types, without 
even updating them for PEP 34[0-9].  :)  It wouldn't be *as* useful for 
non-CPython implementations, but presumably by the time those 
implementations catch up, more code will be out there with 
__enter__/__exit__ methods.  Also, by allowing a default __enter__ to exist 
(that returns self), many objects need only implement an __exit__.  (For 
example, I don't see a point to closed file objects raising an error when 
used in a 'with' block; if you're actually using the file you'll already 
get an error when you use its other methods, and if you're not actually 
using it, there's no point to the error, since close() is idempotent.)

So, at the C API level, I'm thinking something like Py_EnterResource(ob), 
that returns ob if ob has no tp_resource_enter slot defined, otherwise it 
returns the result of calling the method.  Similarly, some sort of 
Py_ExitResource() that guarantees an error return after invoking the 
tp_resource_exit slot (if any).

Finally, note that this extension now makes 'with' seem more like 'with' in 
other languages, because it is now just a scoped variable definition, with 
hooks for the object being scoped to be notified about entry and exit from 
scope.  It does mean that people encountering 'with some_expr()' (without 
an "as") may wonder about whether names inside the scope are somehow 
relative to 'some_expr', but it will probably become clear from context, 
especially via appropriate names.  For example 'with self.__locked' might 
provide that extra bit of clarity beyond 'with self.__lock'.