[Python-ideas] With clauses for generator expressions

Guido van Rossum guido at python.org
Fri Nov 16 05:27:49 CET 2012


On Thu, Nov 15, 2012 at 8:12 PM, Mathias Panzenböck
<grosser.meister.morti at gmx.net> wrote:
> I think this syntax would still make sense for list comprehensions:
>
> upperlines = [lines.upper() for line in file with open('foo', 'r') as file]

-1000. There is no discernible advantage over

with open(...) as file:
  upperlines = [lines.upper() for line in file]

Also you've got the order backwards -- when there's a sequence of
'for' and 'if' clauses in a comprehension, they are to be read from
left to right, but here you're tacking something onto the end that's
supposed to go first.

Please don't destroy my beautiful language.

--Guido

> On 11/15/2012 10:29 AM, Masklinn wrote:
>>
>>
>> On 2012-11-15, at 04:44 , Andrew Barnert wrote:
>>
>>> First, I realize that people regularly propose with expressions. This is
>>> not the
>>> same thing.
>>>
>>> The problem with the with statement is not that it can't be postfixed
>>> perl-style, or used in expressions. The problem is that it can't be used
>>> with
>>> generator expressions.
>>>
>>> Here's the suggestion:
>>>
>>>     upperlines = (lines.upper() for line in file with open('foo', 'r') as
>>> file)
>>>
>>> This would be equivalent to:
>>>
>>>     def foo():
>>>         with open('foo', 'r') as file:
>>>             for line in file:
>>>                 yield line.upper()
>>>     upperlines = foo()
>>>
>>> The motivation is that there is no way to write this properly using a
>>> with
>>> statement and a generator expression—in fact, the only way to get this
>>> right is
>>> with the generator function above.
>>
>>
>> Actually, it's extremely debatable that the generator function is
>> correct: if the generator is not fully consumed (terminating iteration
>> on the file) I'm pretty sure the file will *not* get closed save by the
>> GC doing a pass on all dead objects maybe. This means this function is
>> *not safe* as a lazy source to an arbitrary client, as that client may
>> very well use itertools.slice or itertools.takewhile and only partially
>> consume the generator.
>>
>> Here's an example:
>>
>> --
>> import itertools
>>
>> class Manager(object):
>>      def __enter__(self):
>>          return self
>>
>>      def __exit__(self, *args):
>>          print("Exited")
>>
>>      def __iter__(self):
>>          for i in range(5):
>>              yield i
>>
>> def foo():
>>      with Manager() as ms:
>>          for m in ms:
>>              yield m
>>
>> def bar():
>>      print("1")
>>      f = foo()
>>      print("2")
>>      # Only consume part of the iterable
>>      list(itertools.islice(f, None, 2))
>>      print("3")
>>
>> bar()
>> print("4")
>> --
>>
>> CPython output, I'm impressed that the refcounting GC actually bothers
>> unwinding the stack and running the __exit__ handler *once bar has
>> finished executing*:
>>
>>> python3 withgen.py
>>
>> 1
>> 2
>> 3
>> Exited
>> 4
>>
>> But here's the (just as correct, as far as I can tell) output from pypy:
>>
>>> pypy-c withgen.py
>>
>> 1
>> 2
>> 3
>> 4
>>
>> If the program was long running, it is possible that pypy would run
>> __exit__ when the containing generator is released (though by no means
>> certain, I don't know if this is specified at all).
>>
>> This is in fact one of the huge issues with faking dynamic scopes via
>> threadlocals and context managers (as e.g. Flask might do, I'm not sure
>> what actual strategy it uses), they interact rather weirdly with
>> generators (it's also why I think Python should support actually
>> dynamically scoped variables, it would also fix the thread-broken
>> behavior of e.g. warnings.catch_warnings)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
--Guido van Rossum (python.org/~guido)



More information about the Python-ideas mailing list