[Python-ideas] With clauses for generator expressions

[Andrew Barnert]

I'm pretty sure both my original message and the blog post linked from there 
explained why this is not particularly useful for list comprehensions. (If 
you're guaranteed to exhaust the iteration in the current block—which you 
obviously always are for comprehensions—just make the with a statement with its 
own block.)

The only reason I suggested it for comprehensions as well as generator 
expressions is that someone convinced me that it would be slightly easier to 
implement, and to teach to users, than if it were only available for generator 
expressions.


From: Mathias Panzenböck <grosser.meister.morti at gmx.net>
Sent: Thu, November 15, 2012 8:39:34 PM
> 
> Oh yes, you're right. Didn't think of this. Maybe I should go to bed/not write  
>comments here at 5:38am.
> 
> On 11/16/2012 05:27 AM, Guido van Rossum  wrote:
> > On Thu, Nov 15, 2012 at 8:12 PM, Mathias Panzenböck
> >  <grosser.meister.morti at gmx.net>  wrote:
> >> I think this syntax would still make sense for list  comprehensions:
> >>
> >> upperlines = [lines.upper() for line in  file with open('foo', 'r') as 
file]
> >
> > -1000. There is no  discernible advantage over
> >
> > with open(...) as file:
> >     upperlines = [lines.upper() for line in file]
> >
> > Also you've  got the order backwards -- when there's a sequence of
> > 'for' and 'if'  clauses in a comprehension, they are to be read from
> > left to right, but  here you're tacking something onto the end that's
> > supposed to go  first.
> >
> > Please don't destroy my beautiful  language.
> >
> > --Guido
> >
> >> On 11/15/2012 10:29 AM,  Masklinn wrote:
> >>>
> >>>
> >>> On 2012-11-15,  at 04:44 , Andrew Barnert wrote:
> >>>
> >>>> First, I  realize that people regularly propose with expressions. This  
is
> >>>> not the
> >>>> same  thing.
> >>>>
> >>>> The problem with the with  statement is not that it can't be postfixed
> >>>> perl-style, or  used in expressions. The problem is that it can't be used
> >>>>  with
> >>>> generator  expressions.
> >>>>
> >>>> Here's the  suggestion:
> >>>>
> >>>>       upperlines = (lines.upper() for line in file with open('foo', 'r')  
>as
> >>>> file)
> >>>>
> >>>> This would  be equivalent to:
> >>>>
> >>>>       def foo():
> >>>>          with  open('foo', 'r') as file:
> >>>>               for line in file:
> >>>>                   yield line.upper()
> >>>>       upperlines = foo()
> >>>>
> >>>> The  motivation is that there is no way to write this properly using  a
> >>>> with
> >>>> statement and a generator  expression—in fact, the only way to get this
> >>>> right  is
> >>>> with the generator function  above.
> >>>
> >>>
> >>> Actually, it's extremely  debatable that the generator function is
> >>> correct: if the  generator is not fully consumed (terminating iteration
> >>> on the  file) I'm pretty sure the file will *not* get closed save by the
> >>>  GC doing a pass on all dead objects maybe. This means this function  is
> >>> *not safe* as a lazy source to an arbitrary client, as that  client may
> >>> very well use itertools.slice or itertools.takewhile  and only partially
> >>> consume the  generator.
> >>>
> >>> Here's an  example:
> >>>
> >>> --
> >>> import  itertools
> >>>
> >>> class  Manager(object):
> >>>       def  __enter__(self):
> >>>           return  self
> >>>
> >>>       def __exit__(self,  *args):
> >>>            print("Exited")
> >>>
> >>>       def  __iter__(self):
> >>>           for i in  range(5):
> >>>               yield  i
> >>>
> >>> def foo():
> >>>        with Manager() as ms:
> >>>           for m  in ms:
> >>>               yield  m
> >>>
> >>> def bar():
> >>>        print("1")
> >>>       f = foo()
> >>>        print("2")
> >>>       # Only consume  part of the iterable
> >>>        list(itertools.islice(f, None, 2))
> >>>        print("3")
> >>>
> >>> bar()
> >>>  print("4")
> >>> --
> >>>
> >>> CPython output,  I'm impressed that the refcounting GC actually bothers
> >>> unwinding  the stack and running the __exit__ handler *once bar has
> >>>  finished executing*:
> >>>
> >>>> python3 withgen.py
> >>>
> >>>  1
> >>> 2
> >>> 3
> >>> Exited
> >>>  4
> >>>
> >>> But here's the (just as correct, as far as I  can tell) output from pypy:
> >>>
> >>>> pypy-c  withgen.py
> >>>
> >>> 1
> >>> 2
> >>>  3
> >>> 4
> >>>
> >>> If the program was long  running, it is possible that pypy would run
> >>> __exit__ when the  containing generator is released (though by no means
> >>> certain, I  don't know if this is specified at all).
> >>>
> >>> This is  in fact one of the huge issues with faking dynamic scopes via
> >>>  threadlocals and context managers (as e.g. Flask might do, I'm not  sure
> >>> what actual strategy it uses), they interact rather weirdly  with
> >>> generators (it's also why I think Python should support  actually
> >>> dynamically scoped variables, it would also fix the  thread-broken
> >>> behavior of e.g.  warnings.catch_warnings)
> 
> 
> _______________________________________________
> Python-ideas  mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>