[Python-ideas] With clauses for generator expressions

[Mathias Panzenböck]

I think this syntax would still make sense for list comprehensions:

upperlines = [lines.upper() for line in file with open('foo', 'r') as file]

On 11/15/2012 10:29 AM, Masklinn wrote:
>
> On 2012-11-15, at 04:44 , Andrew Barnert wrote:
>
>> First, I realize that people regularly propose with expressions. This is not the
>> same thing.
>>
>> The problem with the with statement is not that it can't be postfixed
>> perl-style, or used in expressions. The problem is that it can't be used with
>> generator expressions.
>>
>> Here's the suggestion:
>>
>>     upperlines = (lines.upper() for line in file with open('foo', 'r') as file)
>>
>> This would be equivalent to:
>>
>>     def foo():
>>         with open('foo', 'r') as file:
>>             for line in file:
>>                 yield line.upper()
>>     upperlines = foo()
>>
>> The motivation is that there is no way to write this properly using a with
>> statement and a generator expression—in fact, the only way to get this right is
>> with the generator function above.
>
> Actually, it's extremely debatable that the generator function is
> correct: if the generator is not fully consumed (terminating iteration
> on the file) I'm pretty sure the file will *not* get closed save by the
> GC doing a pass on all dead objects maybe. This means this function is
> *not safe* as a lazy source to an arbitrary client, as that client may
> very well use itertools.slice or itertools.takewhile and only partially
> consume the generator.
>
> Here's an example:
>
> --
> import itertools
>
> class Manager(object):
>      def __enter__(self):
>          return self
>
>      def __exit__(self, *args):
>          print("Exited")
>
>      def __iter__(self):
>          for i in range(5):
>              yield i
>
> def foo():
>      with Manager() as ms:
>          for m in ms:
>              yield m
>
> def bar():
>      print("1")
>      f = foo()
>      print("2")
>      # Only consume part of the iterable
>      list(itertools.islice(f, None, 2))
>      print("3")
>
> bar()
> print("4")
> --
>
> CPython output, I'm impressed that the refcounting GC actually bothers
> unwinding the stack and running the __exit__ handler *once bar has
> finished executing*:
>
>> python3 withgen.py
> 1
> 2
> 3
> Exited
> 4
>
> But here's the (just as correct, as far as I can tell) output from pypy:
>
>> pypy-c withgen.py
> 1
> 2
> 3
> 4
>
> If the program was long running, it is possible that pypy would run
> __exit__ when the containing generator is released (though by no means
> certain, I don't know if this is specified at all).
>
> This is in fact one of the huge issues with faking dynamic scopes via
> threadlocals and context managers (as e.g. Flask might do, I'm not sure
> what actual strategy it uses), they interact rather weirdly with
> generators (it's also why I think Python should support actually
> dynamically scoped variables, it would also fix the thread-broken
> behavior of e.g. warnings.catch_warnings)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>