[Python-Dev] Re: anonymous blocks

[Michael Hudson]

Whew!  This is a bit long...

On 25 Apr 2005, at 00:57, Guido van Rossum wrote:

> After reading a lot of contributions (though perhaps not all -- this
> thread seems to bifurcate every time someone has a new idea :-)

I haven't read all the posts around the subject, I'll have to admit.  
I've read the one I'm replying and its followups to pretty carefully, 
though.

> I'm back to liking yield for the PEP 310 use case. I think maybe it was
> Doug Landauer's post mentioning Beta, plus scanning some more examples
> of using yield in Ruby. Jim Jewett's post on defmacro also helped, as
> did Nick Coghlan's post explaining why he prefers 'with' for PEP 310
> and a bare expression for the 'with' feature from Pascal (and other
> languages :-).

The history of iterators and generators could be summarized by saying 
that an API was invented, then it turned out that in practice one way 
of implementing them -- generators -- was almost universally useful.

This proposal seems a bit like an effort to make generators good at 
doing something that they aren't really intended -- or dare I say 
suited? -- for.  The tail wagging the dog so to speak.

> It seems that the same argument that explains why generators are so
> good for defining iterators, also applies to the PEP 310 use case:
> it's just much more natural to write
>
>     def with_file(filename):
>         f = open(filename)
>         try:
>             yield f
>         finally:
>             f.close()

This is a syntax error today, of course.  When does the finally: clause 
execute  with your proposal? [I work this one out below :)]

> than having to write a class with __entry__ and __exit__ and
> __except__ methods (I've lost track of the exact proposal at this
> point).

> At the same time, having to use it as follows:
>
>     for f in with_file(filename):
>         for line in f:
>             print process(line)
>
> is really ugly,

This is a non-starter, I hope.  I really meant what I said in PEP 310 
about loops being loops.

> so we need new syntax, which also helps with keeping
> 'for' semantically backwards compatible. So let's use 'with', and then
> the using code becomes again this:
>
>     with f = with_file(filename):
>         for line in f:
>             print process(line)
>
> Now let me propose a strawman for the translation of the latter into
> existing semantics. Let's take the generic case:
>
>     with VAR = EXPR:
>         BODY
>
> This would translate to the following code:
>
>     it = EXPR
>     err = None
>     while True:
>         try:
>             if err is None:
>                 VAR = it.next()
>             else:
>                 VAR = it.next_ex(err)
>         except StopIteration:
>             break
>         try:
>             err = None
>             BODY
>         except Exception, err: # Pretend "except Exception:" == 
> "except:"
>             if not hasattr(it, "next_ex"):
>                 raise
>
> (The variables 'it' and 'err' are not user-visible variables, they are
> internal to the translation.)
>
> This looks slightly awkward because of backward compatibility; what I
> really want is just this:
>
>     it = EXPR
>     err = None
>     while True:
>         try:
>             VAR = it.next(err)
>         except StopIteration:
>             break
>         try:
>             err = None
>             BODY
>         except Exception, err: # Pretend "except Exception:" == 
> "except:"
>             pass
>
> but for backwards compatibility with the existing argument-less next()
> API

More than that: if I'm implementing an iterator for, uh, iterating, why 
would one dream of needing to handle an 'err' argument in the next() 
method?

> I'm introducing a new iterator API next_ex() which takes an
> exception argument.  If that argument is None, it should behave just
> like next().  Otherwise, if the iterator is a generator, this will
> raised that exception in the generator's frame (at the point of the
> suspended yield).  If the iterator is something else, the something
> else is free to do whatever it likes; if it doesn't want to do
> anything, it can just re-raise the exception.

Ah, this answers my 'when does finally' execute question above.

> Finally, I think it would be cool if the generator could trap
> occurrences of break, continue and return occurring in BODY.  We could
> introduce a new class of exceptions for these, named ControlFlow, and
> (only in the body of a with statement), break would raise BreakFlow,
> continue would raise ContinueFlow, and return EXPR would raise
> ReturnFlow(EXPR) (EXPR defaulting to None of course).

Well, this is quite a big thing.

> So a block could return a value to the generator using a return
> statement; the generator can catch this by catching ReturnFlow.
> (Syntactic sugar could be "VAR = yield ..." like in Ruby.)
>
> With a little extra magic we could also get the behavior that if the
> generator doesn't handle ControlFlow exceptions but re-raises them,
> they would affect the code containing the with statement; this means
> that the generator can decide whether return, break and continue are
> handled locally or passed through to the containing block.
>
> Note that EXPR doesn't have to return a generator; it could be any
> object that implements next() and next_ex().  (We could also require
> next_ex() or even next() with an argument; perhaps this is better.)

My main objection to all this is that it conflates iteration and a more 
general kind of execution control (I guess iteration is a kind of 
execution control, but I contend that it's a sufficiently common case 
to get special treatment and also that names like 'for' and 'next' are 
only applicable to iteration).

So, here's a counterproposal!

with expr as var:
    ... code ...

is roughly:

def _(var):
     ... code ...
__private = expr
__private(_)

(var optional as in other proposals).

so one might write:

def open_file(f):
     def inner(block):
         try:
             block(f)
         finally:
             f.close()
     return inner

and have

with auto_closing(open("/tmp/foo")) as f:
     f.write('bob')

The need for approximation in the above translation is necessary 
because you'd want to make assignments in '...code...' affect the scope 
their written in, and also one might want to allow breaks and continues 
to be handled as in the end of your proposal.  And grudgingly, I guess 
you'd need to make returns behave like that anyway.

Has something like this been argued out somewhere in this thread?

As another example, here's how you'd implement something very like a 
for loop:

def as_for_loop(thing):
     it = iter(thing)
     def inner(thunk):
         while 1:
             try:
                 v = it.next()
             except StopIteration:
                 break
             try:
                 thunk(v)
             except Continue:
                 continue
             except Break:
                 break

so

for x in s:

and

with as_for_loop(s) as x:

are now equivalent (I hope :).

Cheers,
mwh