Abstracting try..finally with generators (instead of macros)

[Beni Cherniavsky]

On 2002-12-16, Oren Tirosh wrote:

> On Mon, Dec 16, 2002 at 02:03:47AM +0200, Beni Cherniavsky wrote:
> > Well, iterators that require finalization should be documented as such.
> > After some reading I see that this is Python's general approach.Implicit
> > finalization can't be trusted.
>
> But "no finalization" can...
>
Yes, probably.  It's more predicatable.  Better yet would be if CPython
gave warnings when generators with pending ``finally`` clauses are
deallocated without their `finalize` methods having been called.

> > The for loop seems to be the only construct which holds an iterator
> > without exposing it to user code so it must have this finalization
> > support.Besides, a for loop is the most common way to use iterators.
>
> map(f,x), filter(f,x), dict(x), list(x) all iterate over x.
> i in x also iterates over x if it does not have a __contains__ method.
>
Thanks a lot for the correction.  Forgot them entirely.  I think they too
should be expected to finilize the iterators.  But wait -- can they abort
in the middle (not due to the iterator)?  `map` and `filter` can
(exception in function), `dict` can (unhashable), `list` can't (except for
memory errors).  ``i in x`` can (exception in `__contains__`).  OK, they
should.  Rationale: the need to finalize has also arosen for holding a
lock while iterating.  This has nothing in common with contrived conrol
structure abstraction and is a real need.  So iterators should be
encouraged to be explicictly finalized by all generic language facilities
using them.  I need to study the C iterators API...

But then it's also good to expect this from user code.  That's annoying
but worthy, since otherwise iterators with side effects are very
restricted.  Actually, if the language builtins do this, only the
infrequent code using ``.next()`` manually has to do this.

Backward compatibility with custom iterators demands that ``.finalize()``
be called only if the iterator has this attribute.  This might also be a
performance consideration -- we don't want the for loop to always behave
like a try..finally, it's probably cheaper to detect this during the loop
"setup".  Builtin generators are not entirely predictible but the presence
of a yield inside try..finally can trigger having a `finalize` method.

But then the distinction between iterators with and without `finalize`
is perpetuated and user code needs to test that:

def trivial(iterable):
    i = iter(iterable)
    try:
        while 1:
            yield i.next()
    finally:
            getitem(i, 'finalize', lambda:None)()

or (like the builtins maybe should do for speed, if it's better indeed):

def trivial(iterable):
    i = iter(iterable)
    finalize = getattr(i, 'finalize', None):
    if finalize:
        try:
            while 1:
                yield i.next()
        finally:
            finalize()
    else:
        while 1:
            yield i.next

Nobody is gonna write that...  The first is cumbersome enough too.  Ideas?


Anyway now I will be able to:

map(some_func, withmyout())

instead of:

for dummy in withmyout():
    somefuct()

(Not that it's readable.)

> There are several other cases of implicit iteration.
>
Additions welcome.  I want to collect them all :)

- list comprehension fors.

- `enumerate` from PEP 279 (when it's implemented.  when, BTW ?-)

-- 
Beni Cherniavsky <cben at tx.technion.ac.il>