[Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)

[Guido van Rossum]

Nick's PEP 3XX is here:
http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html

[Phillip J. Eby]
> Anyway, I took a look at it, and I mostly like it.

I like the beginning of Nick's PEP too, since its spec for the with
statement is now identical to PEP 343 apart from the keyword choice
(and I'm leaving that open for a public vote). The fundamental
difference now is that Nick proposes an exception injection and
finalization API for generators. I think that ought to be done as a
separate PEP, since it's relatively separate from the
do-or-with-statement (even though it's clear that PEP 343 would be the
first to benefit).

Maybe Nick can strip his PEP (or start a new one) to only addresses
generator exception injection and generator finalization. I think he's
on to something but it needs to be spec'ed more exactly. Continuing
with Phillip...:

> There appears to be an
> error in "Deterministic generator finalisation" (maybe you already know
> this): the _inject_exception() should be called with exc_info, not
> TerminateIteration, and it should swallow StopIteration instead of
> TerminateIteration.  IOW, I think it should look like this:
> 
>      def __exit__(self, *exc_info):
>          try:
>              self._inject_exception(*exc_info)
>          except StopIteration:
>              pass
> 
> Hm.  Oh wait, I just realized - you don't mean this at all.  You're
> describing a use of generators as non-templates.  Ugh.  I think that might
> lead to confusion about the semantics of 'with' and generators.  I'll have
> to think about it some more, but my second impression after a little bit of
> thought is that if you're going to do this, then you should be allowed to
> use 'with' with any object, using the object as VAR if there's no
> __enter__.  My reasoning here is that it then makes it possible for you to
> use arbitrary objects for 'with' without needing to know their
> implementation details.  It should be harmless to use 'with' on objects
> that don't need it.

I think the issue here is not implementation details but whether it
follows a certain protocol. IMO it's totally acceptable to require
that the expression used in a with-statement support an appropriate
protocol, just like we require the expression used in a for-loop to be
an iterable.

> This insight may actually be true regardless of what generators do or don't
> do; the point is that if you change from using a generator to a built-in
> iterator type, you shouldn't have to change every place you were using the
> 'with' blocks to work again.

Huh? The with-statement doesn't loop, and its use of generators is
such that I don't see how you could ever replace it with a built-in
iterator.

> A further part of this insight: perhaps the 'with' block translation should
> include a 'del VAR' in its finally block, not to mention the equivalent of
> 'del stmt_enter,stmt_exit'.  In other words, the binding of VAR should not
> escape the 'with' block.

I've seen Nick's stmt_enter and stmt_exit as implementation details
that probably would be done differently by a real implementation;
personally I don't mind getting the TypeError about a missing __exit__
only when __exit__ is called (after all if this happens the program is
too buggy to care much about the precise semantics) so this is what
PEP 343 does.

About deleting VAR I have mixed feelings. I appreciate the observation
that it's most likely dead after the with-statement is over, but I'm
not sure that explicit deletion is correct. Remember that VAR can be
an arbitrary assignment target, which means it could be a global, or a
tuple, or an indexed list or dict item, or some object's attribute (I
hesitate to write "etc." after such an exhaustive list :-). Example 8
in PEP 340 shows a use case where at least one of the variables (the
error value) could meaningfully survive the block. I think that, given
that we let the for-loop variable survive, we should treat the
with-statement variable the same way. A good compiler can see that
it's dead if it's unused further and delete it earlier, but I don't
think we need more -- especially since in a GC'ed implementation like
Jython or IronPython, deleting it doesn't do us much good.

> This would mean that for existing types that use
> __del__ for cleanup (e.g. files and sockets), then 'with open("file") as f'
> would automatically ensure closing under CPython (but other implementations
> would be allowed to wait for GC).  In other words, I'm saying that this:
> 
>       with some_expr() as foo:
>           # etc.
> 
> should also be short for this (in the case where some_expr() has no
> __enter__ or __exit__ methods):
> 
>       foo = some_expr()
>       try:
>           # etc.
>       finally:
>           del foo

-1 on this. You're trying to pack too much into a single statement.

> And that could be a useful thing for many existing object types, without
> even updating them for PEP 34[0-9].  :)  It wouldn't be *as* useful for
> non-CPython implementations, but presumably by the time those
> implementations catch up, more code will be out there with
> __enter__/__exit__ methods.

Most likely it would just give an implementation-dependent false sense
of security.

Also note that if you really need this, you can usually get the effect
with a del statement *without* a try/finally clause -- if an exception
is raised that you don't explicitly catch, the variable will go out of
scope anyway, so you only need the del upon normal completion of the
block.

> Also, by allowing a default __enter__ to exist
> (that returns self), many objects need only implement an __exit__.  (For
> example, I don't see a point to closed file objects raising an error when
> used in a 'with' block; if you're actually using the file you'll already
> get an error when you use its other methods, and if you're not actually
> using it, there's no point to the error, since close() is idempotent.)
> 
> So, at the C API level, I'm thinking something like Py_EnterResource(ob),
> that returns ob if ob has no tp_resource_enter slot defined, otherwise it
> returns the result of calling the method.  Similarly, some sort of
> Py_ExitResource() that guarantees an error return after invoking the
> tp_resource_exit slot (if any).
> 
> Finally, note that this extension now makes 'with' seem more like 'with' in
> other languages, because it is now just a scoped variable definition, with
> hooks for the object being scoped to be notified about entry and exit from
> scope.  It does mean that people encountering 'with some_expr()' (without
> an "as") may wonder about whether names inside the scope are somehow
> relative to 'some_expr', but it will probably become clear from context,
> especially via appropriate names.  For example 'with self.__locked' might
> provide that extra bit of clarity beyond 'with self.__lock'.

-1 on the whole thing (and noting that this would be an easy extension
if at some point in the future we change our mind).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)