[Python-Dev] PEP 343 - Abstract Block Redux

[Guido van Rossum]

[Guido van Rossum]
> > I'm in favor of the general idea, but would like to separate the error
> > injection and finalization API for generators into a separate PEP,
> > which would then compete with PEP 288 and PEP 325.

[Nick Coghlan]
> Without that it pretty much devolves into the current version of PEP 343, though
> (as far as I can tell, the two PEP's now agree on the semantics of with statements)

But that's okay, right? We can just accept both PEPs and we're done.
(And PEP 342 at the same time. :-) Discussing one PEP at a time is
sometimes easier, even if they are meant to be used together.

> > I think the API
> > *should* be made public if it is available internally; I don't see any
> > implementation reasons why it would simplify the implementation if it
> > was only available internally.
> 
> If it's internal, we don't need to name it immediately - working out the public
> API can then be decoupled from the ability to use generators to write resource
> managers.

That's a minor point -- it's not like generators have a namespace for
the user that we don't want to pollute, so we can pick any name we
like.

> > Here are some issues to resolve in such a PEP.
> >
> > - What does _inject_exception() return? It seems that it should raise
> > the exception that was passed into it, but I can't find this written
> > out explicitly.
> 
> That's a good point. The intent was for it to be equivalent to the exception
> being reraised at the point of the last yield. At that point, control flows like
> it would for a call to next() with code inside the generator that looked like:
> 
>    yield
>    exc_type, value, tb = _passed_in_exception()
>    raise exc_type, value, tb

So, also, _inject_exception() will appear to raise the exception that
was passed to it, unless the generator catches it.

> > - What should happen if a generator, in a finally or except clause
> > reached from _inject_exception(), executes another yield? I'm tempted
> > to make this a legitimate outcome (from the generator's perspective)
> > and reserve it for some future statement that implements the looping
> > semantics of PEP 340; the *_template decorator's wrapper class however
> > should consider it an error, just like other protocol mismatches like
> > not yielding the first time or yielding more than once in response to
> > next(). Nick's code in fact does all this right, Ijust think it should
> > be made explicit.
> 
> Yep, that was the intent - you're correct that describing it in the text as well
> would make it clear that this is deliberate.

OK. Please update the PEP then!

> > - TerminateIteration is a lousy name, since "terminate" means about
> > the same as "stop", so there could be legitimate confusion with
> > StopIteration. In PEP 340 I used StopIteration for this purpose, but
> > someone explained this was a poor choice since existing generators may
> > contain code that traps StopIteration for other purposes. Perhaps we
> > could use SystemExit for this purpose? Pretty much everybody is
> > supposed to let this one pass through since its purpose is only to
> > allow cleanup upon program (or thread) exit.
> 
> Wouldn't that mean we run the risk of suppressing a *real* SystemExit if it
> occurs while a generator is being finalised?

D'oh. Yes.

> Perhaps a new exception IteratorExit, which is a subclass of SystemExit. Then
> well-behaved code wouldn't trap [SystemExit] accidentally,
> and the finalisation code wouldn't?

Nah, I think it needs to be a brand spanking new exception. It just
can't be called TerminateIteration. How about GeneratorFinalization to
be utterly clear?

> > - I really don't like reusing __del__ as the method name for any kind
> > of destructor; __del__ has all sorts of special semantics (the GC
> > treats objects with a __del__ method specially).
> 
> I guess if file objects can live without a __del__ method to automatically
> close, generators that require finalisation can survive without it.

Right. At the C level there is of course finalization (the tp_dealloc
slot in the PyTypeObject struct) but it doesn't have a Python entry
point to call it. And that's intentional, since the typical code
executed by that slot *really* destroys the object and must only be
called when the VM is absolutely sure that the object can't be reached
in any other way. if it were callable from Python, that guarantee
would be void.

> > - The all_lines() example jars me. Somehow it bugs me that its caller
> > has to remember to care about finalizing it.
> 
> The alternative is to have some form of automatic finalisation in for loops, or
> else follow up on PJE's idea of making with statements allow pretty much
> *anything* to be used as VAR1.
> 
> Automatic finalisation (like that now described in the Rejected Options section
> of my PEP) makes iterators/generators that manage resources 'just work' (nice)
> but complicates the semantics of generators (bad) and for loops (bad).
> 
> The current version of my PEP means you need to know that a particular
> generator/iterator needs finalisation, and rearrange code to cope with that
> (bad), but keeps for loops simple (nice).

I agree with that so far.

> PJE's idea about a permissive with statement may actually give the best of both
> worlds - you can *always* use the with statement, and if the supplied object
> doesn't need cleaning up, there's no more overhead than checking for the
> existence of methods in a couple of slots. (See my answer to Phillip on that
> topic for the gory details)

But I don't like that solution -- and it's up to you to explain why
(see previous exchange between Phillip & me :-).

> > Maybe it's just not a
> > good example;
> 
> I was having trouble thinking of a case where it made sense for the generator to
> manage its own resources. I agree all_lines isn't a great example, but its a
> safe bet that things like it will be written once the restriction on yielding
> inside try/finally is lifted.

I'd rather drop the example than make one up that's questionable.

It's better to look for potential use cases in existing code than just
look at the proposed construct and ponder "how could I use this..." --
existing code showing a particular idiom/pattern being used repeatedly
shows that there's an actual need.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)