[Python-ideas] With clauses for generator expressions

[Masklinn]

On 2012-11-15, at 04:44 , Andrew Barnert wrote:

> First, I realize that people regularly propose with expressions. This is not the 
> same thing.
> 
> The problem with the with statement is not that it can't be postfixed 
> perl-style, or used in expressions. The problem is that it can't be used with 
> generator expressions.
> 
> Here's the suggestion:
> 
>    upperlines = (lines.upper() for line in file with open('foo', 'r') as file)
> 
> This would be equivalent to:
> 
>    def foo():
>        with open('foo', 'r') as file:
>            for line in file:
>                yield line.upper()
>    upperlines = foo()
> 
> The motivation is that there is no way to write this properly using a with 
> statement and a generator expression—in fact, the only way to get this right is 
> with the generator function above.

Actually, it's extremely debatable that the generator function is
correct: if the generator is not fully consumed (terminating iteration
on the file) I'm pretty sure the file will *not* get closed save by the
GC doing a pass on all dead objects maybe. This means this function is
*not safe* as a lazy source to an arbitrary client, as that client may
very well use itertools.slice or itertools.takewhile and only partially
consume the generator.

Here's an example:

--
import itertools

class Manager(object):
    def __enter__(self):
        return self

    def __exit__(self, *args):
        print("Exited")

    def __iter__(self):
        for i in range(5):
            yield i

def foo():
    with Manager() as ms:
        for m in ms:
            yield m

def bar():
    print("1")
    f = foo()
    print("2")
    # Only consume part of the iterable
    list(itertools.islice(f, None, 2))
    print("3")

bar()
print("4")
--

CPython output, I'm impressed that the refcounting GC actually bothers
unwinding the stack and running the __exit__ handler *once bar has
finished executing*:

> python3 withgen.py 
1
2
3
Exited
4

But here's the (just as correct, as far as I can tell) output from pypy:

> pypy-c withgen.py 
1
2
3
4

If the program was long running, it is possible that pypy would run
__exit__ when the containing generator is released (though by no means
certain, I don't know if this is specified at all).

This is in fact one of the huge issues with faking dynamic scopes via
threadlocals and context managers (as e.g. Flask might do, I'm not sure
what actual strategy it uses), they interact rather weirdly with
generators (it's also why I think Python should support actually
dynamically scoped variables, it would also fix the thread-broken
behavior of e.g. warnings.catch_warnings)