[Python-ideas] With clauses for generator expressions

[Mathias Panzenböck]

Just throwing random syntax variations on the wall to see what/if anything sticks (because I think 
the "as file"-assignment serves no purpose here):

     upperlines = (lines.upper() for line in with open('foo', 'r'))
     upperlines = (lines.upper() for line with open('foo', 'r'))
     upperlines = (lines.upper() with for line in open('foo', 'r'))

Or should the for loop check if there are __enter__ and __exit__ methods and call them? Guess not, 
but I thought I just mention it as an alternative.

For now one can do this, which is functional equivalent but adds the overhead of another generator:

     def managed(sequence):
         with sequence:
             for item in sequence:
                 yield item

     upperlines = (lines.upper() for line in managed(open('foo', 'r')))

You could even call this helper function "with_", if you like.

Or write a helper like this:

     def iterlines(filename,*args,**kwargs):
         with open(filename,*args,**kwargs) as f:
             for line in f:
                 yield line

     upperlines = (lines.upper() for line in iterlines('foo', 'r'))

Maybe there should be a way to let a file be automatically closed when EOF is encountered? Maybe an 
"autoclose" wrapper object that passes through every method call to the file object but when EOF is 
encountered during a read it closes the file object? Then one could write:

     upperlines = (lines.upper() for line in autoclose(open('foo', 'r')))


On 11/15/2012 04:44 AM, Andrew Barnert wrote:
> First, I realize that people regularly propose with expressions. This is not the
> same thing.
>
> The problem with the with statement is not that it can't be postfixed
> perl-style, or used in expressions. The problem is that it can't be used with
> generator expressions.
>
> Here's the suggestion:
>
>      upperlines = (lines.upper() for line in file with open('foo', 'r') as file)
>
> This would be equivalent to:
>
>      def foo():
>          with open('foo', 'r') as file:
>              for line in file:
>                  yield line.upper()
>      upperlines = foo()
>
> The motivation is that there is no way to write this properly using a with
> statement and a generator expression—in fact, the only way to get this right is
> with the generator function above. And almost nobody ever gets it right, even
> when you push them in the right direction (although occasionally they write a
> complex class that has the same effect).
>
> That's why we still have tons of code like this lying around:
>
>      upperlines = (lines.upper() for line in open('foo', 'r'))
>
> Everyone knows that this only works with CPython, and isn't even quite right
> there, and yet people write it anyway, because there's no good alternative.
>
> The with clause is inherently part of the generator expression, because the
> scope has to be dynamic. The file has to be closed when iteration finishes, not
> when creating the generator finishes (or when the generator is cleaned up—which
> is closer, but still wrong).
>
> That's why a general-purpose "with expression" wouldn't actually help here; in
> fact, it would just make generator expressions with with clauses harder to
> parse. A with expression would have to be statically scoped to be general.
>
> For more details, see this:
>
> http://stupidpythonideas.blogspot.com/2012/11/with-clauses-for-generator-expressions.html