[Python-ideas] With clauses for generator expressions

[Mathias Panzenböck]

Oh yes, you're right. Didn't think of this. Maybe I should go to bed/not write comments here at 5:38am.

On 11/16/2012 05:27 AM, Guido van Rossum wrote:
> On Thu, Nov 15, 2012 at 8:12 PM, Mathias Panzenböck
> <grosser.meister.morti at gmx.net> wrote:
>> I think this syntax would still make sense for list comprehensions:
>>
>> upperlines = [lines.upper() for line in file with open('foo', 'r') as file]
>
> -1000. There is no discernible advantage over
>
> with open(...) as file:
>    upperlines = [lines.upper() for line in file]
>
> Also you've got the order backwards -- when there's a sequence of
> 'for' and 'if' clauses in a comprehension, they are to be read from
> left to right, but here you're tacking something onto the end that's
> supposed to go first.
>
> Please don't destroy my beautiful language.
>
> --Guido
>
>> On 11/15/2012 10:29 AM, Masklinn wrote:
>>>
>>>
>>> On 2012-11-15, at 04:44 , Andrew Barnert wrote:
>>>
>>>> First, I realize that people regularly propose with expressions. This is
>>>> not the
>>>> same thing.
>>>>
>>>> The problem with the with statement is not that it can't be postfixed
>>>> perl-style, or used in expressions. The problem is that it can't be used
>>>> with
>>>> generator expressions.
>>>>
>>>> Here's the suggestion:
>>>>
>>>>      upperlines = (lines.upper() for line in file with open('foo', 'r') as
>>>> file)
>>>>
>>>> This would be equivalent to:
>>>>
>>>>      def foo():
>>>>          with open('foo', 'r') as file:
>>>>              for line in file:
>>>>                  yield line.upper()
>>>>      upperlines = foo()
>>>>
>>>> The motivation is that there is no way to write this properly using a
>>>> with
>>>> statement and a generator expression—in fact, the only way to get this
>>>> right is
>>>> with the generator function above.
>>>
>>>
>>> Actually, it's extremely debatable that the generator function is
>>> correct: if the generator is not fully consumed (terminating iteration
>>> on the file) I'm pretty sure the file will *not* get closed save by the
>>> GC doing a pass on all dead objects maybe. This means this function is
>>> *not safe* as a lazy source to an arbitrary client, as that client may
>>> very well use itertools.slice or itertools.takewhile and only partially
>>> consume the generator.
>>>
>>> Here's an example:
>>>
>>> --
>>> import itertools
>>>
>>> class Manager(object):
>>>       def __enter__(self):
>>>           return self
>>>
>>>       def __exit__(self, *args):
>>>           print("Exited")
>>>
>>>       def __iter__(self):
>>>           for i in range(5):
>>>               yield i
>>>
>>> def foo():
>>>       with Manager() as ms:
>>>           for m in ms:
>>>               yield m
>>>
>>> def bar():
>>>       print("1")
>>>       f = foo()
>>>       print("2")
>>>       # Only consume part of the iterable
>>>       list(itertools.islice(f, None, 2))
>>>       print("3")
>>>
>>> bar()
>>> print("4")
>>> --
>>>
>>> CPython output, I'm impressed that the refcounting GC actually bothers
>>> unwinding the stack and running the __exit__ handler *once bar has
>>> finished executing*:
>>>
>>>> python3 withgen.py
>>>
>>> 1
>>> 2
>>> 3
>>> Exited
>>> 4
>>>
>>> But here's the (just as correct, as far as I can tell) output from pypy:
>>>
>>>> pypy-c withgen.py
>>>
>>> 1
>>> 2
>>> 3
>>> 4
>>>
>>> If the program was long running, it is possible that pypy would run
>>> __exit__ when the containing generator is released (though by no means
>>> certain, I don't know if this is specified at all).
>>>
>>> This is in fact one of the huge issues with faking dynamic scopes via
>>> threadlocals and context managers (as e.g. Flask might do, I'm not sure
>>> what actual strategy it uses), they interact rather weirdly with
>>> generators (it's also why I think Python should support actually
>>> dynamically scoped variables, it would also fix the thread-broken
>>> behavior of e.g. warnings.catch_warnings)