Abstracting try..finally with generators (instead of macros)

[Beni Cherniavsky]

Hi,

I believe I've just discovered a new cool use for generators, one that
people thought they need macros for [1].  The problem at hand is creating
things like with-output-to-file, save-excursions, etc. like many lisps
proudly have through macros.  These are one of the more readable of the
common uses for macros -- they don't modify the control flow in unobvious
ways but just abstract away particular recurring combinations of pre/post
code (including exception safety).  In other words, tasty stuff ;-)

For the purpose of the example, let's have a file-like object convenient
for debugging:

class MyOut:
    def write(self, s):
        sys.__stdout__.write(s.replace('\n', ' @@@\n'))

myout = MyOut()

Useful code pieces will be represented by bare print statements.  Without
abstractions one has to write:

print 1
savedout = sys.stdout
sys.stdout = myout
try:
    print 2
finally:
    sys.stdout = savedout
print 3

To get the intended:

1
2 @@@
3

I was looking for statements which could be modified to support
abstractions of the above when I realised that -- Eurika! -- the for loop
can do it already, combined with a simple generator:

def withmyout():
    saved = sys.stdout
    sys.stdout = myout
    yield None
    sys.stdout = saved

print 1
for dummy in withmyout():
    print 2
print 3

We get a quite elegant idiom (IMHO).  Too bad, since this has a bug: if we
exit the loop by break or an exception, the generator isn't exhausted and
`sys.stdout` remains garbled.  The Right Fix it would be to put the yield
inside a try..finally.  Alas, that is currently undefined in Python.

Luckily, there is a chance to return control to the iterator: when it's
deleted (which is right away, -- the only references to it was held by the
for loop).  The iterator must now be spelled out as a class to define the
custom `.__del__()`: [2]

class withmyout_safe:
    def __iter__(self):
        self.saved = None
        return self
    def next(self):
        if self.saved:
            raise StopIteration
        self.saved = sys.stdout
        sys.stdout = myout
    def __del__(self):
        sys.stdout = self.saved

Now the following:

print 1
try:
    for dummy in withmyout_safe2():
        print 2
        raise Exception
except:
    print 3
print 4

correctly prints:

1
2 @@@
3
4

The refcounting seems to work reliably!  Too bad since the code is ugly
:-(.  Thinking some more, there is a way to abstract just this ugly
have-a-__del__ part:

class safe_gen:
    def __init__(self, generator):
        self.gen = generator
    def __call__(self):
        return safe_iter(self.gen())

class safe_iter:
    def __init__(self, iterator):
        self.iter = iterator
    def __iter__(self):
        return self
    def next(self):
        return self.iter.next()
    def __del__(self):
        for dummy in self.iter:
            pass

This implements wrappers for iterators and generators that make sure the
iterator is exhausted to completion.  Now one can just write the original
unsafe but consice generator code and say:

withmyout_safe2 = safe_gen(withmyout)

Then the loop (with ``withmyout_safe2()``) works safely!

Conclusions:

- The for loop, can be used to abstract some simple control structures
  with generators that have side side effects (instead of returning
  meaningful info) and yield at the point where the body should be
  executed.  A possible term to for this: "control iterators".

  To make things simpler, maybe it would makes sense to allow a
  variable-less version of for (``for iterator: body``)?

- Generators with side effects would greatly benefit from a defining
  try:..finally around yield.  It is possible to define sensible semantics
  for it - to be executed when the last reference to the iterator is
  deleted (if `.next()` wasn't called).  I strongly vote for it.  Any
  chance for Python 2.3?

  Possible implementation: on `.__del__()`, resume the generator with an
  exception (`KillGenerator`?) and silently catch it if the generator
  doesn't.

  Wart to watch out: explain that try..except in generators doesn't catch
  exceptions in the consumer while try..finally effectively does (would).

  - This can be currently emulated (with somewhat different semantics)
    with something like the above wrappers.  That's slower and less
    powerful.

----

[1] See also:
http://groups.google.com/groups?selm=slrn9dv5st.68t.neelk%40alum.mit.edu&rnum=10
Seems that his prediction turns out at least partially true...

[2] Idea (probably too hard to implement): turn each generator into a
class/type implementing `.__init__()` (`.__new__()`?) and `.next()`.
Note that the class describes the iterator that's created by calling the
factory generator object - like a class should behave.  Then one could
write the main logic as a generator and add methods by subclassing...
This would be top cool if the variables of the generator were attributes
of the iterators but that could be slooow.  Compare PEP 288.

-- 
Beni Cherniavsky <cben at tx.technion.ac.il>